Continuous Batching

PART 11 — Continuous Batching

Standard batching waits for 4 requests, groups them, and runs them. But in LLMs, if Request A finishes in 10 tokens and Request B takes 100 tokens, the GPU wastes cycles waiting for B to finish.

Continuous Batching (Iteration-Level Scheduling): * SDE Analogy: OS Process Scheduling (Round Robin / Preemption). * At every single step of the decode while loop, the inference server (the scheduler) checks the queue. It evicts finished requests and dynamically injects new requests into the active GPU batch without waiting for the whole batch to finish.

---