Skip to content

Attention

PART 5 — Attention

Attention is the breakthrough that made modern LLMs possible.

Consider the sentence: "The animal didn't cross the road because it was tired." What does "it" refer to? The animal or the road?

  • Attention: A mechanism that allows the model to look at all other tokens in a sequence and decide which ones are most relevant to the current token. When processing "it", the model pays heavy attention to "animal".
  • Query (Q): What information am I looking for? (e.g., "I am 'it', I need a noun that can be tired").
  • Key (K): What information do I contain? (e.g., "I am 'animal', I am a living noun").
  • Value (V): What information should I provide if matched? (e.g., the actual semantic meaning of animal).
  • Attention Score: How well the Query and Key match. High score = high relevance.

Why does an HLD engineer care?

Attention requires the model to compare every token to every other token. If you double the context size, the computation historically increases by 4x (\(O(N^2)\) complexity). This is why processing huge documents in AI requires vast amounts of memory and compute.