Transformers

PART 7 — Transformers

The Transformer is the DAG (Directed Acyclic Graph) of operations that houses the Attention mechanism.

• Encoder: Processes the entire input array in parallel. Used for Embeddings (e.g., BERT).
• Decoder: Processes token i to predict token i+1. Used for Text Generation (e.g., GPT).

SDE Takeaway: Encoder models are highly parallelizable and fast. Decoder models are strictly sequential and suffer from the autoregressive bottleneck (you need the output of step N to start step N+1).

---