Skip to content

Sync vs Async LLM Inference

1. Decision

Choose Synchronous Streaming LLM Inference (Server-Sent Events / WebSockets) for interactive user-facing chatbots and real-time conversational UIs where initial Time-To-First-Token (TTFT) latency is critical; choose Asynchronous Batch LLM Inference (Kafka / SQS / AWS Bedrock Batch) for high-throughput, non-blocking background tasks (document summarization, offline embeddings generation, dataset processing, batch evaluations) to achieve 50% lower cost and higher GPU utilization.

2. 30-Second Answer

LLM inference execution patterns differ significantly from traditional lightweight web APIs. Large language model token generation takes seconds to complete. Synchronous Streaming Inference uses HTTP Server-Sent Events (SSE) or WebSockets to stream output tokens back to the user interface incrementally as they are generated by the model. This optimizes perceived latency: while total generation may take 5 seconds, the user sees the first token in \(<300\text{ms}\) (TTFT - Time-To-First-Token). Asynchronous Batch Inference decouples request submission from processing using background message queues (Kafka/SQS) or managed batch APIs (OpenAI Batch API, AWS Bedrock Batch). Clients submit thousands of prompts asynchronously, receiving job completion webhooks. Batch inference maximizes GPU throughput via continuous batching (vLLM / TensorRT-LLM) and cuts API/compute costs by 50%.

3. Decision Matrix

Option Best For Avoid When Latency Metric GPU Throughput Cost Efficiency Operational Complexity
Synchronous Streaming (SSE / WebSockets) Interactive chatbots, AI copilots, live coding assistants, real-time customer support UIs Offline document processing, bulk dataset classification, background cron jobs Low Perceived Latency (TTFT \(<300\text{ms}\)); Total time bounded by token length Medium (Requires reserving active GPU worker slots) Standard Managed API / GPU Cost Medium (Managing long-lived HTTP SSE / WebSocket connections)
Asynchronous Batch (Queues / Batch APIs) Bulk PDF document processing, offline data extraction, offline embedding generation, dataset labeling Real-time interactive user chatbots requiring immediate conversational turns High Total Latency (Minutes to Hours); Zero impact on user UI Maximum (Continuous Batching saturates GPU VRAM) Extremely High (50% discount on OpenAI / Bedrock Batch APIs) Low to Medium

4. Selection Rules

Requirement Best Choice Rationale / Example
Interactive AI Chatbot (ChatGPT / Claude Style) Sync SSE Streaming Stream tokens token-by-token over Server-Sent Events (text/event-stream) for instant TTFT
Nightly Summarization of 100,000 Customer Support Transcripts Async Batch API Submit bulk JSONL prompt batch to OpenAI Batch API at 50% cost discount
Self-Hosted High-Throughput LLM Serving Cluster vLLM + Continuous Batching Deploy vLLM with PagedAttention; processes concurrent streaming and batch requests
Long-Running Multimodal Document Extraction (>30s) Async Webhook / Queue Push PDF processing request to SQS; worker processes via LLM and fires completion webhook

5. Tradeoffs

  1. Low Perceived Latency (TTFT) vs High Total GPU Throughput
  2. Gain (Sync Streaming): User reads the first generated sentence in \(<300\text{ms}\), creating a responsive UI feel.
  3. Lose (Sync Streaming): Reserving GPU memory per active streaming connection reduces total system throughput.
  4. 50% Cost Discount (Batch) vs Delayed Processing Time
  5. Gain (Async Batch): Managed providers (OpenAI / Anthropic / AWS) offer 50% price cuts for 24-hour SLA batch jobs.
  6. Lose (Async Batch): Requests wait in queues; un-suited for interactive user workflows.
  7. HTTP Server-Sent Events (SSE) vs WebSockets
  8. Gain (SSE): Simple lightweight unidirectional HTTP streaming (text/event-stream) over standard HTTP/2.
  9. Lose (SSE): Unidirectional only (Client cannot send mid-generation cancel messages without closing TCP socket).
  10. GPU PagedAttention Memory Utilization (vLLM)
  11. Gain (vLLM): Dynamic KV-cache allocation eliminates VRAM fragmentation, enabling \(10\times\) higher request concurrency.
  12. Lose (vLLM): Requires operating dedicated GPU cluster infrastructure (T4/A10G/A100/H100).
  13. Connection Failure Resilience
  14. Gain (Async): Network disconnects do not abort execution; results persist in database.
  15. Lose (Sync): Browser connection drop cancels in-flight LLM generation unless decoupled via background buffers.

6. Interview Counter Questions

  1. What is TTFT (Time-To-First-Token) and why is it the primary SLA metric for interactive LLM applications?
  2. How does Server-Sent Events (SSE) differ from WebSockets for streaming LLM responses?
  3. What is Continuous Batching (Iteration-Level Batching) in vLLM / TensorRT-LLM compared to Traditional Batching?
  4. How does PagedAttention solve KV-Cache memory fragmentation in GPU LLM serving engines?
  5. How does OpenAI Batch API / AWS Bedrock Batch deliver 50% cost discounts?
  6. What happens to GPU KV-Cache memory when 100 users stream long-context LLM generation concurrently?
  7. How do you handle connection drops during long synchronous SSE LLM streams?
  8. What is Speculative Decoding and how does it accelerate synchronous LLM token generation latency?
  9. When would you combine Async Queue processing with SSE streaming (Redis Pub/Sub bridge)?
  10. How do you implement load shedding and rate limiting on LLM inference servers?

7. Gold Standard Answers

  1. TTFT: Time between sending prompt and receiving the first generated token. It dictates user-perceived responsiveness. Users accept a 5s total generation time if TTFT is \(<300\text{ms}\).
  2. SSE: Simple unidirectional HTTP/2 protocol (text/event-stream). Lightweight, built-in browser reconnection, works through standard proxies. WebSockets: Bi-directional full-duplex TCP socket (overkill for text streaming).
  3. Traditional Batching waits for \(N\) requests, processes them together, and blocks until the longest request finishes. Continuous Batching: Operates at the token-iteration level, adding new requests to the active batch as completed requests finish.
  4. KV-Cache stores attention keys/values per token in GPU VRAM. PagedAttention (vLLM) partitions KV-cache into virtual memory pages (like OS virtual memory), eliminating 96% of VRAM fragmentation and doubling concurrency.
  5. Managed providers run batch API jobs on idle off-peak GPU capacity across their datacenters, guaranteeing 24-hour completion windows in exchange for 50% lower API pricing.
  6. KV-cache memory scales linearly with Context Length × Batch Size. Under concurrent long-context requests, GPU VRAM exhausts quickly, forcing the engine to preempt/swap KV-caches to RAM (latency spike).
  7. Decouple: Client calls API \(\rightarrow\) App launches background job generating tokens into Redis Pub/Sub \(\rightarrow\) SSE endpoint streams from Redis. If browser reconnects, it fetches missed tokens from Redis.
  8. Speculative Decoding: A tiny, fast draft model (Llama-8B) generates candidate tokens rapidly; a large target model (Llama-70B) validates the batch in a single forward pass, accelerating speed \(2\text{x}-3\text{x}\).
  9. Client sends job to SQS queue \(\rightarrow\) GPU Worker processes job \(\rightarrow\) Worker publishes tokens to Redis Pub/Sub \(\rightarrow\) Gateway streams tokens to user browser via SSE bridge.
  10. Measure active KV-cache VRAM usage and queue depth. When VRAM \(>90\%\), reject incoming requests immediately with HTTP 429 / 503 (or fall back to smaller models) to prevent GPU OOM crashes.

8. Common Mistakes

  • Returning a blocking, non-streaming synchronous HTTP 200 OK JSON response for a 1,000-token chatbot output (forces user to wait 6 seconds staring at a blank screen).
  • Using expensive synchronous API endpoints for bulk offline document processing instead of using 50%-discounted Batch APIs.
  • Using heavy bi-directional WebSockets when simple unidirectional HTTP Server-Sent Events (SSE) is completely sufficient.
  • Self-hosting LLMs using naive HuggingFace Transformers pipelines without continuous batching engines (vLLM / TGI), wasting 80% of GPU capacity.
  • Failing to handle client network disconnects during long SSE streams, leaving background GPU inference running unnecessarily.

9. Keywords

Sync LLM Inference, Async LLM Inference, TTFT (Time-To-First-Token), Server-Sent Events (SSE), WebSockets, Continuous Batching, vLLM, PagedAttention, KV-Cache, Speculative Decoding, OpenAI Batch API, AWS Bedrock Batch, TensorRT-LLM, Redis Pub/Sub.

10. Cheat Sheet

  • Decision Rule: Interactive Chatbot / Copilot \(\rightarrow\) Synchronous SSE Token Streaming; Bulk Offline Jobs / Embeddings \(\rightarrow\) Asynchronous Batch API (50% cost discount).
  • Core SLA Metric: TTFT (Time-To-First-Token) \(<300\text{ms}\) is critical for user-perceived speed.
  • Streaming Standard: Use Server-Sent Events (SSE) (text/event-stream) over HTTP/2. Simpler & lighter than WebSockets.
  • Self-Hosted Engine: Always serve LLMs using vLLM or TensorRT-LLM for Continuous Batching and PagedAttention VRAM optimization.
  • Latency Booster: Use Speculative Decoding (small draft model + large validation model) to double token generation speed.
  • Cost Reduction: Submit background jobs to OpenAI Batch API / AWS Bedrock Batch for 50% pricing discounts.

11. Related Decisions