Skip to content

Llm Inference Prefill Vs Decode

PART 9 — LLM Inference (Prefill vs Decode)

This is the most critical concept for AI Serving Architecture.

  1. Prefill Phase: The first step. The LLM processes the entire user prompt (e.g., 1000 tokens) in one giant parallel matrix multiplication.

    • Bottleneck: Compute-bound (Limited by GPU TFLOPS).
    • Metric: Time to First Token (TTFT).

  2. Decode Phase: The while loop. The LLM generates 1 token, appends it, and repeats.

    • Bottleneck: Memory-bandwidth bound. To generate 1 token, the GPU must load all 70 Billion parameters from VRAM into the compute cores. It does this over and over.
    • Metric: Inter-Token Latency (Time between tokens).

SDE Takeaway: Serving LLMs means juggling requests in these two phases. A large prefill request will hog the GPU compute cores and pause all the decode requests, causing stuttering. Modern inference servers isolate or chunk prefill to prevent this.