Speculative Decoding

PART 14 — Speculative Decoding

Because loading the 70B model into cores is slow (Memory Bound), we cheat using a tiny 1B model.

Example Trace: 1. Draft Phase: The tiny 1B model quickly predicts the next 3 tokens: "The", "quick", "brown". (Takes 5ms). 2. Verify Phase: We feed ["The", "quick", "brown"] to the massive 70B model all at once as a mini-prefill batch. (Takes 15ms). 3. The 70B model processes them in parallel and outputs its own probabilities. 4. It agrees with "The", agrees with "quick", but rejects "brown" (it wanted "red"). 5. We accept "The quick red". We generated 3 tokens in 20ms instead of 45ms.

---