Ai System Trade Offs For Sdes

PART 19 — AI System Trade-offs for SDEs

When designing AI architectures, you manipulate these core engineering levers:

1. Time to First Token (TTFT) vs Inter-Token Latency: High batch sizes increase overall throughput (tokens/sec) but delay the TTFT for individual users due to queueing.
2. PagedAttention Block Size: Smaller blocks reduce memory fragmentation (good) but increase the pointer-chasing overhead in the block table (bad).
3. Context Size vs Concurrency: Expanding the context window to 128k means the KV Cache grows quadratically. 1 user with a 128k context might consume the VRAM of 100 users with a 1k context.
4. Recall vs Latency (Vector DBs): Tuning the HNSW ef_search parameter. Higher values search more graph nodes (higher accuracy) but increase latency.

---