Ai Evaluation

PART 18 — AI Evaluation

Unit tests (assert output == expected) fail because LLMs are non-deterministic and output natural language.

SDE Takeaway: You must treat AI Evaluation like CI/CD for Prompts. * You use LLM-as-a-judge: You write a test where GPT-4 grades the output of your smaller production model on a scale of 1-5 for relevance, hallucination, and toxicity. * You run these eval suites offline over a golden dataset before merging any prompt changes.

---