Skip to content

Speculative Decoding

PART 14 — Speculative Decoding

Because generating one token at a time (decode) is slow, we can cheat.

  • Idea: Run a tiny, super-fast model to quickly guess the next 5 tokens. Then, pass those 5 tokens to the massive, slow model in one big batch to "verify" if they are correct.
  • Because the large model processes batches quickly, verifying 5 tokens takes the same time as generating 1 token. 
Small / fast model → Generates "is a good day"
       ↓
Large model → Verifies: "is a good" (Accepts), "day" (Rejects, meant "time")

Why does an HLD engineer care?

Speculative decoding reduces latency and increases text generation speed, especially when you have spare compute capacity. It is an architectural optimization pattern.