Skip to content

Ai Evaluation

PART 18 — AI Evaluation

You cannot run traditional unit tests on an LLM because the output is non-deterministic (different every time).

  • Offline evaluation: Running scripts against datasets to test metrics before deploying.
  • Online evaluation: Tracking metrics in production (e.g., user thumbs up/down).
  • Hallucination: The model confidently stating false information.
  • LLM-as-a-judge: Using a massive, smart model (like GPT-4) to grade the output of a smaller, cheaper model.

Why does an HLD engineer care?

System changes (swapping a model, changing a prompt) will break things silently. Without an evaluation pipeline in your CI/CD architecture, you are flying blind.