Multi Head Attention

PART 6 — Multi-Head Attention

One attention mechanism (one "head") might only look for one type of relationship, like grammar.

• Multi-Head Attention (MHA): Running multiple attention mechanisms in parallel.
• Why? So the model can understand multiple complex relationships simultaneously.

Intuition:

Sentence: "The bank of the river"
   ↓
       Head 1 → focuses on syntax (noun, preposition)
       Head 2 → focuses on semantic context (water, nature)
       Head 3 → focuses on positional distance (words nearby)
   ↓
Combined rich representation

Why does an HLD engineer care?

Multi-Head Attention dictates memory usage during inference. The states of these Keys and Values for every head must be temporarily stored in GPU memory to generate text efficiently. This leads directly to the concept of the KV Cache (covered in Part 10).

---