Llm Inference Prefill Vs Decode

PART 9 — LLM Inference (Prefill vs Decode)

This is crucial for HLD. The while loop above has two distinct performance phases.

Example Trace (Prompt: 1000 tokens. Output: 50 tokens):

Phase 1: Prefill (Iteration 0) * Input: Array of 1000 tokens. * Math: Massive Matrix * Matrix multiplication. * GPU Status: 100% Compute Utilization. * Time taken: 200ms. (This is the Time-To-First-Token).

Phase 2: Decode (Iterations 1 to 50) * Input: Array of 1 token (the newest token). * Math: Vector * Matrix multiplication. * GPU Status: 10% Compute Utilization, 100% Memory Bandwidth Utilization (Waiting for weights to load from VRAM). * Time taken: 20ms per token. (Total 1000ms for 50 tokens).

---