Speculative Decoding

PART 14 — Speculative Decoding

Because Decode is memory-bandwidth bound (slow), we use a smaller, faster "draft" model to predict the next 5 tokens. We then pass those 5 tokens to the large, slow model in a single batch to verify them.

SDE Analogy: Optimistic Concurrency Control or CPU Branch Prediction. We guess the future state. If we guess right, we commit the transaction (accept the tokens) and skip 5 slow loop iterations. If we guess wrong, we rollback to the first incorrect token and resume normal generation.

---