Attention

PART 5 — Attention

Attention solves the problem of "context". It allows token #50 to dynamically look back at tokens 1-49 to understand what it means.

The Database Analogy: * Query (Q): What I'm looking for (e.g., SELECT *). * Key (K): The index I match against (e.g., WHERE key = ...). * Value (V): The actual data payload returned. * Attention Score: Soft-matching. Instead of returning exactly 1 row, it returns a weighted sum of all rows based on similarity score.

SDE Takeaway: Attention has an O(N^2) time and space complexity with respect to sequence length N. If you double the context window, you quadruple the compute and memory required. This is the primary scaling bottleneck for long-context models.

---