Continuous Batching
PART 11 — Continuous Batching¶
Standard batching waits for a group of requests to finish together. Continuous batching injects them on the fly.
Example Trace:
Time T0:
- Batch starts.
- Req 1 (Prefill), Req 2 (Prefill)
Time T1 (Req 1 finished early!):
- Req 1 (Done. Evicted from GPU memory).
- Req 3 (NEW! Injected dynamically. Runs Prefill)
- Req 2 (Runs Decode token #2)
Time T2:
- Req 3 (Runs Decode token #1)
- Req 2 (Runs Decode token #3)