Skip to content

Ai System Architecture Vocabulary Cheat Sheet

PART 19 — AI System Architecture Vocabulary (Cheat Sheet)

  • LLM: A massive model generating text. The brain of the system.
  • SLM (Small Language Model): A smaller, cheaper, faster model for simple tasks.
  • Embedding model: A specialized model that converts text to number arrays.
  • Reranker: A specialized model used after vector search to re-sort results based on high-accuracy relevance.
  • Tokenizer: Software that chops strings into model-readable token IDs.
  • Transformer: The neural network architecture standard for AI.
  • Attention / MHA: The mathematical mechanism letting tokens relate to each other.
  • KV cache: GPU memory cache of past token states to speed up text generation.
  • Context window: The max input + output size allowed per request.
  • Prefill: The fast, compute-heavy phase of reading the user's prompt.
  • Decode: The slow, memory-heavy phase of generating words one by one.
  • TTFT (Time To First Token): Latency metric measuring prefill speed.
  • Throughput: How many tokens the server processes per second globally.
  • Continuous batching: Dynamically grouping requests on the GPU to maximize throughput.
  • Quantization: Compressing a model by using smaller data types (INT4) to save VRAM.
  • Tensor parallelism: Splitting model math across multiple GPUs in the same server.
  • Pipeline parallelism: Splitting model layers across different servers.
  • RAG: Architecture to inject external database knowledge into an LLM prompt.
  • Vector DB: Database designed for fast similarity search of embeddings.
  • ANN: Approximate Nearest Neighbor. Algorithm used by Vector DBs for fast search.
  • Agent: An orchestration loop where an LLM repeatedly plans, calls tools, and observes.
  • Tool calling: Exposing functions/APIs for an LLM to use.
  • Guardrails: Input/output filtering systems to catch toxicity, PII, or hallucinations.
  • Evaluation: Frameworks (like LLM-as-a-judge) to measure output quality over time.
  • Model Router / Gateway: Infrastructure layer that routes requests to the cheapest/fastest model suitable for the task.
  • Inference Server: The software (like vLLM or TGI) running on the GPU that handles batching, KV caching, and execution.