Kv Cache

PART 10 — KV Cache

The Problem: In autoregressive generation, to guess token #100, the model processes tokens 1-99. To guess token #101, it processes tokens 1-100. Recalculating the attention scores for tokens 1-99 every single time is incredibly wasteful.

• KV Cache: Caching the "Key" and "Value" vectors of previously processed tokens in GPU memory.
• Result: To generate token #101, the GPU only calculates the Q, K, V for token #100, and pulls the rest from the KV Cache.

Without KV Cache:
Token 1 → Compute 1
Token 2 → Compute 1, 2
Token 3 → Compute 1, 2, 3 (Massive wasted compute)

With KV Cache:
Token 1 → Cache K/V of 1
Token 2 → Compute 2, read 1 from cache
Token 3 → Compute 3, read 1,2 from cache

Why does an HLD engineer care?

The KV Cache consumes a massive amount of VRAM (GPU memory). If 100 users are sending 10k token prompts, the KV cache alone might exceed the memory of an $80,000 GPU server. Memory limits concurrency. Managing KV cache efficiently (e.g., PagedAttention) is the core feature of modern inference servers like vLLM.

---