Ai System Trade Offs Sde Context

PART 19 — AI System Trade-offs (SDE Context)

Example: Context vs Concurrency tradeoff calculation Let's calculate the VRAM used by the KV Cache for a Llama-3 70B model. * Formula: 2 (K&V) * 2 bytes (FP16) * 80 (layers) * 8 (KV heads) * 128 (head dim) * context_length * Cache per token ≈ 320 KB.

Scenario A (Short Chat): * User context: 1,000 tokens. * KV Cache size: 320 MB. * A server with 80GB free VRAM can handle ~250 concurrent users.

Scenario B (Book Summarization): * User context: 100,000 tokens. * KV Cache size: 32,000 MB (32 GB!). * That same server can handle only 2 concurrent users.

HLD Decision: If your product involves long documents, you must implement aggressive chunking (RAG), or purchase 100x more GPU hardware to support the same traffic volume.