Evals: measuring an LLM system
Why every LLM system needs an evaluation set, how to build one, code-based checks, LLM-as-judge with a rubric, and how to use evals to compare prompts, models and settings.
AI: production
AI series 4 of 5. What it takes to ship an LLM system. Evals, observability, cost, security, guardrails, a decision map, and one app that puts it all together.
7 parts
- Part 1
- Part 2
Observability: seeing what an LLM system does
What to log for every model call and agent step, traces and spans, a tiny tracer in Python, the metrics worth watching, privacy, and turning production failures into evals.
- Part 3
Cost and latency: prompt caching, batches and model choice
Where the money and time go in an LLM system, and the levers that cut them: the right model and effort, shorter outputs, prompt caching, the Batch API and streaming.
- Part 4
Security: prompt injection and the OWASP LLM Top 10
The main risks of LLM apps, from prompt injection to excessive agency, and the defenses that work: keeping data apart from instructions, least privilege and screening.
- Part 5
Guardrails: checks around the model
Layered guardrails for LLM applications: screening input with a small model, validating and redacting output, checking actions, limits, and what to show the user when a check fails.
- Part 6
Decision map: prompt, RAG, tools, agent or fine-tune?
The whole series in one place: a decision map for choosing the right technique, what each one changes and costs, a worked example, and a checklist before going to production.
- Part 7
Putting it together: a support assistant from start to finish
One small app built with the whole series: a prompt, RAG over help articles, a tool for live order data, input screening, evals, caching and logging. Each step links to the post that explains it.