Block Table for User A's Request
PART 10 — KV Cache & PagedAttention¶
In the while loop above, calculating iteration #5 requires recalculating Attention for tokens 1, 2, 3, and 4. To avoid this, we cache the K and V matrices.
Example Problem: If user A has a max context of 32k, we historically allocated 32k slots of VRAM immediately. If they only generated 10 tokens, 99% of that VRAM was wasted (Fragmentation).
PagedAttention (vLLM) Example: Just like OS Virtual Memory, we allocate memory in "Blocks" (e.g., 16 tokens per block).
# Block Table for User A's Request
request_A_blocks = [
PhysicalBlock(id=102, size=16_tokens), # Stores tokens 0-15
PhysicalBlock(id=45, size=16_tokens), # Stores tokens 16-31
]
# When token 32 is generated, we allocate a NEW physical block dynamically:
request_A_blocks.append(PhysicalBlock(id=89, size=16_tokens))