Llm Inference

PART 9 — LLM Inference

This is the most critical section for HLD. Inference (generating text) happens in two distinct phases:

1. Prefill Phase: The user submits the prompt. The GPU processes all the input tokens in parallel. This is highly compute-bound.
2. Decode Phase: The GPU generates new output tokens one-by-one. This is highly memory-bandwidth bound.

Important Metrics: * Time to First Token (TTFT): How long the Prefill takes. Critical for user experience (nobody likes a loading spinner). * Inter-token latency (Time Between Tokens): How fast the Decode phase is. Critical for reading speed. * Throughput: Total tokens processed per second across all concurrent users. * Latency: Time for a single request to complete.

Prompt (1000 tokens)
  ↓
Prefill (Fast, processes all 1000 tokens at once)
  ↓
First token generated (TTFT measured here)
  ↓
Decode Phase (Slow, generates 1 token at a time)
  ↓
Token... Token... Token...

Why does an HLD engineer care?

Prefill and Decode compete for GPU resources. If you have 10 users decoding, and 1 user sends a massive prefill request, the prefill will starve the decode users, causing their text generation to stutter. HLD engineers must design inference servers that balance these phases (e.g., using chunked prefill).

---