Cost and latency: prompt caching, batches and model choice
Where the money and time go in an LLM system, and the levers that cut them: the right model and effort, shorter outputs, prompt caching, the Batch API and streaming.
Tag: LLMOps
7 entries
Decision map: prompt, RAG, tools, agent or fine-tune?
The whole series in one place: a decision map for choosing the right technique, what each one changes and costs, a worked example, and a checklist before going to production.
Putting it together: a support assistant from start to finish
One small app built with the whole series: a prompt, RAG over help articles, a tool for live order data, input screening, evals, caching and logging. Each step links to the post that explains it.
Evals: measuring an LLM system
Why every LLM system needs an evaluation set, how to build one, code-based checks, LLM-as-judge with a rubric, and how to use evals to compare prompts, models and settings.
Guardrails: checks around the model
Layered guardrails for LLM applications: screening input with a small model, validating and redacting output, checking actions, limits, and what to show the user when a check fails.
Observability: seeing what an LLM system does
What to log for every model call and agent step, traces and spans, a tiny tracer in Python, the metrics worth watching, privacy, and turning production failures into evals.
Security: prompt injection and the OWASP LLM Top 10
The main risks of LLM apps, from prompt injection to excessive agency, and the defenses that work: keeping data apart from instructions, least privilege and screening.