Skip to content

22 Ai System Trade Offs Hld Perspective

PART 21 & 22 — AI System Trade-offs (HLD Perspective)

Designing AI systems requires balancing directly competing constraints:

  • Latency vs Throughput: To get max throughput (tokens/sec), you must batch many requests. Bigger batches slow down the time-to-first-token (latency) for individual users.
  • Quality vs Cost: Using a massive 70B parameter model gives high quality but requires expensive multi-GPU servers. Using a smaller 8B model reduces cost by 10x but increases hallucinations.
  • Context Length vs Memory: Passing a massive document into the prompt is easy, but it exponentially explodes the KV Cache memory usage on the GPU, preventing you from serving concurrent users.
  • Model Size vs Inference Cost: Larger models generate better answers, but cost significantly more per token to run because of the larger VRAM footprint.
  • Precision vs Memory: FP16 gives perfect model quality. INT4 quantization cuts memory usage by 75% (allowing cheaper GPUs) but degrades reasoning capabilities slightly.
  • Retrieval Recall vs Latency (Vector DBs): Perfect search requires scanning every document (high latency). ANN indexes give low latency but might miss the best document.
  • Number of Retrieved Chunks vs Context Size: Retrieving 50 chunks in RAG gives the LLM more context, but eats up the context window, slowing down inference, increasing token cost, and confusing the model (Lost in the Middle syndrome).
  • Larger Model vs Smaller Model: Larger models take longer to serve, are more expensive, but generalize better to complex tasks. Smaller models can be fine-tuned to beat larger models on specific, narrow tasks.
  • GPU Utilization vs Request Latency: Keeping GPU utilization at 100% means you are queuing requests, which spikes latency. To keep latency low, you must overprovision GPUs (lower utilization).
  • Synchronous vs Asynchronous Inference: Simple LLM chats are synchronous web requests. Agents or massive batch jobs (like offline evaluation) require asynchronous queue-based designs.
  • Centralized vs Distributed Serving: Centralized (API endpoints) is easy to manage. Distributed (edge inference / SLMs on device) reduces cloud costs and privacy concerns but is incredibly hard to orchestrate.