Head 1 (Dimensions 0-127): Focuses on Grammar
PART 6 — Multi-Head Attention¶
Instead of one generic Attention calculation, we split the vector and run multiple in parallel.
Example: Our embedding has 1024 dimensions. We split it into 8 "Heads" of 128 dimensions each.
# Head 1 (Dimensions 0-127): Focuses on Grammar
head_1_output = grammar_attention(Q[0:127], K[0:127], V[0:127])
# Head 2 (Dimensions 128-255): Focuses on Subject-Verb relationships
head_2_output = relation_attention(Q[128:255], K[128:255], V[128:255])
# Final output is concatenated back together
final_output = concatenate([head_1_output, head_2_output, ...])