Skip to content

Batching

PART 11 — Batching

GPUs are massive parallel processing factories. If you send a GPU one request, 95% of its cores sit idle.

  • Batching: Grouping multiple user requests together and processing them at the exact same time on the GPU.
  • Static Batching: Waiting for 4 requests to arrive, processing them together. But if one request generates 10 tokens and another generates 100, the GPU sits idle waiting for the longest one to finish.
  • Continuous (In-flight) Batching: Dynamically injecting new requests into the batch the exact millisecond another request finishes. 
Request A ─┐
Request B ─┼→ GPU (Batched together)
Request C ─┘

Why does an HLD engineer care?

Batching dramatically increases Throughput (tokens/sec) but slightly hurts individual Latency. Continuous batching is a mandatory requirement for any production LLM serving architecture to achieve cost-efficiency.