Transformers

PART 7 — Transformers

The Transformer is the specific neural network architecture underlying almost all modern AI (GPT, BERT, Llama).

• Encoder: Reads the entire input text at once and builds a deep understanding of it (good for classification/search).
• Decoder: Takes a sequence and predicts the very next token, one at a time (good for generation).
• Encoder-Decoder: Uses both (e.g., Translation: encode English, decode French).
• Decoder-Only: Just the decoder. Almost all modern LLMs (GPT-4, Claude) are Decoder-Only.

High-Level Architecture of a Transformer Block:

Tokens
 ↓
Embeddings (Convert to numbers)
 ↓
Positional Information (Injecting order, since vectors have no inherent order)
 ↓
Transformer Block
 ├── Attention (Tokens talk to each other)
 ├── Normalization & Residual (Keep numbers stable)
 ├── Feed-Forward Network / MLP (Tokens think individually)
 └── Normalization & Residual
 ↓
Output

Why does an HLD engineer care?

Knowing if a model is an Encoder (like BERT for embeddings) or a Decoder (like Llama for chat) dictates the infrastructure you build. Encoders are fast and parallelizable. Decoders are sequential, slow, and memory-hungry.

---