Attention

PART 5 — Attention

Attention lets tokens "look" at other tokens to understand context.

Example Trace: Sentence: "Bank of the river" (Tokens: [Bank, of, the, river])

We want to understand what "Bank" means here (Financial vs. River). 1. "Bank" generates a Query (Q) vector: [I need context to know what type of bank I am] 2. "river" generates a Key (K) vector: [I am water, nature] 3. We multiply Q and K. The result is a high Attention Score (e.g., 0.95). 4. Because the score is high, "Bank" absorbs the Value (V) vector of "river".

Now, the vector representing "Bank" has been mathematically altered to include the concept of water. It now means "River Bank".

---