Why Your App Is Already Broken
(And Demos Won't Tell You)
LLM Evals From Zero: build an eval suite that blocks bad releases. This chapter sets up the problem the whole course solves: a change that looks better in a demo and is measurably worse in production.
Learning objectives
- Explain why "looks good" fails as a quality signal for LLM systems
- Define an evaluation formally: dataset, system, scorers, gate
- Distinguish evals from unit tests, observability, and guardrails
- Read an eval report and make a merge or block decision from it
How this lecture runs
- 20 minutes, 20 slides: one core concept every 45 to 60 seconds
- One running example throughout: a support RAG assistant
- Every claim is backed by an artifact: a diff, a dataset, a report, a pipeline
- No tooling tour. Plain concepts first, tools later in the course
Why "looks good" is not a metric
A metric is a number you can compare across two versions of a system and act on. A vibe check produces none of the properties a release decision needs.
| Property a release decision needs | Vibe check | Measured evaluation |
|---|---|---|
| Reproducible | Different reviewer, different verdict. Mood, order, and recency all leak in. | Same dataset, same scorers, same result. Anyone can rerun it. |
| Comparable | No baseline. "Better than last week" is unrecorded memory. | Candidate is diffed against a saved baseline run, per category. |
| Coverage | Whoever demos picks the examples, so the sample is biased by construction. | Dataset is stratified on purpose: easy, hard, adversarial, out of scope. |
| Actionable | Output is an opinion. Nothing in CI can consume an opinion. | Output is a score against a threshold. A gate can merge or block on it. |
| Attributable | "Quality feels off" names no component and no fix. | Failures cluster by cause: retrieval, generation, refusal, citation. |
Engineering question: what exactly am I measuring, and could a script act on the answer?
The system under test: Support RAG
One system, used for the entire course. It answers support questions from product docs, cites its sources, and must refuse when the answer is not in the docs.
Contract
- Answer only from retrieved docs
- Every claim carries a citation
- No source in docs means refuse, do not guess
Change surfaces
- Prompt revisions
- Chunking and embedding model swaps
- Model upgrades and provider changes
Failure surfaces
- Retriever returns the wrong chunk
- Model answers beyond the context
- Model refuses a question it could answer
A prompt change that appears to improve quality
Prompt v13 asks for warmer, more complete answers. In the demo the answers read better: fuller, friendlier, more confident. Nothing about the demo tells you what it cost.
What the change really does
- Removes the grounding constraint. "Fill small gaps" is a license to hallucinate.
- Weakens refusal. A warm, complete answer is the opposite of "I do not know."
- Makes citations optional. "Where they fit" means unverifiable claims ship.
- Improves surface quality. Longer, warmer text reads better in a spot check, which is exactly why the demo passes.
Engineering question: which behavior did this diff trade away, and who measured it?
Manual spot-check: 5 examples, 5 passes
The team tries v13 on five familiar questions before merging. Every one looks good. This is the exact moment most bad releases get approved.
| # | Query | v13 behavior observed | Verdict | Hidden property |
|---|---|---|---|---|
| 1 | How do I reset my password? | Correct steps, warm tone, cited | PASS | Easy, in docs, high frequency |
| 2 | Where is my invoice? | Correct steps, cited | PASS | Easy, in docs |
| 3 | How do I add a team member? | Correct, fuller than v12 | PASS | Easy, in docs |
| 4 | Can I export to CSV? | Correct, cited | PASS | Medium, in docs |
| 5 | What is the refund window? | Correct number, cited | PASS | Medium, in docs |
What the sample contains
Five answerable, in-scope, previously seen questions. Zero refusals, zero adversarial phrasing, zero missing-doc cases.
What a pass here proves
Almost nothing. With n = 5 drawn from the easiest slice of traffic, even a badly regressed system passes most of the time.
500 production queries expose the regression
Run the same v13 against a 500 query dataset drawn from real support traffic, scored the same way as the v12 baseline. The demo result does not survive contact with the distribution.
Engineering question: how do I detect regressions before users do? Run the candidate on the full distribution and diff it against a baseline.
Failure taxonomy: name it to measure it
A single accuracy number hides everything. Split failures by cause and each one gets its own scorer, its own owner, and its own fix.
Hallucination
Answer asserts facts absent from the retrieved context.
Signal: claims with no supporting chunk.
Scorer: faithfulness judge plus citation check.
Retrieval failure
The right chunk never reaches the model, so no prompt can save the answer.
Signal: expected source missing from top-k.
Scorer: recall@k, deterministic.
Refusal error
Answers what it should refuse, or refuses what it could answer.
Signal: behavior mismatch on should-refuse set.
Scorer: confusion matrix, next visual.
Citation error
Citation present but wrong, stale, or not supporting the claim.
Signal: cited chunk does not entail the sentence.
Scorer: citation accuracy check.
Refusal confusion matrix, example run (refusal slice, n = 200 of 500)
Refusal correctness = (143 + 34) / 200 = 88.5%, matching the report on slide 6.
Failure clusters among failed cases, example run
Read the clusters, not the score. Retrieval is the top fix here, not prompt wording.
Why demos systematically miss these failures
Demo failure is not bad luck. Four structural biases guarantee the demo sample looks nothing like production traffic.
1 · Selection bias
The presenter picks examples the system is known to handle. The sample is filtered by the outcome before it is shown.
2 · Distribution bias
Demo queries are short, clean, and in scope. Production adds typos, ambiguity, multi-part asks, and out-of-scope requests, which is where refusals and retrieval break.
3 · Sample size
Five examples cannot reveal a failure mode that fires on 5 to 10 percent of traffic. Rare per query, certain per day at production volume.
4 · No baseline, no memory
A demo compares the new version against a feeling about the old one. Regressions on categories nobody re-tests are invisible by definition.
| Demo sample | Production distribution | |
|---|---|---|
| Query source | Chosen by the team | Real user traffic, logged |
| Should-refuse cases | Almost never shown | A fixed slice of the golden set |
| Verdict | "Looked good live" | Scored, diffed, gated |
What is an evaluation? A formal definition
An evaluation is a repeatable procedure that runs a system over a fixed dataset, scores the outputs against defined criteria, and compares the result to a decision rule.
D · Dataset
Fixed, versioned inputs with expected behavior per item.
S · System
The exact candidate: model, prompt, retriever, config, all pinned.
M · Scorers
Deterministic checks, reference scoring, and judges, per category.
G · Gate
Pre-agreed thresholds that turn a report into a merge or block call.
Offline vs. online evaluation
Offline evals answer "should this ship?" before release. Online evals answer "is it still true?" after release. You need both; they catch different things.
| Dimension | Offline evaluation | Online evaluation |
|---|---|---|
| When | Pre-release, on every candidate change | Post-release, continuously on live traffic |
| Data | Golden dataset: fixed, labeled, versioned | Sampled production traffic, often weakly labeled |
| Ground truth | Expected answers, sources, and refusal labels | User signals, judge scores, human spot review |
| Decision it drives | Merge or block the release | Rollback, canary promotion, dataset updates |
| Blind spot | Cannot see queries nobody anticipated | Cannot run a controlled A/B against a baseline build |
| Failure it catches | The v13 regression on slide 6 | Drift: new products, new phrasing, stale docs |
The loop between them: every real failure found online becomes a permanent case in the offline golden set. The dataset is a growing memory of everything production has taught you.
Evals vs. unit tests
Unit tests verify code behaves as specified. Evals measure how a probabilistic system behaves across a distribution. Related discipline, different object.
| Dimension | Unit tests | Evaluations |
|---|---|---|
| Unit under test | A function or component | The whole system: prompt, retrieval, model, config |
| Output | Deterministic: same input, same output | Probabilistic: same input, a distribution of outputs |
| Oracle | Exact expected value | Criteria, references, rubrics, and judges |
| Pass condition | Binary: pass or fail per test | Aggregate scores vs. thresholds, plus per-category gates |
| Flakiness | A bug: tests must never flake | Managed: fixed seeds where possible, noise bands, rerun rules |
| Cost per run | Milliseconds, free | Real API calls: budget it, cache it, subset it in CI |
Rule of thumb: if the expected output can be written exactly, write a unit test. If quality is a matter of degree across many inputs, write an eval. Most LLM systems need deterministic checks inside the eval suite too, which is Chapter 4 of this course.
Evals vs. observability
Observability tells you what the system did in production. Evaluation tells you whether what it did was good. Traces without scores are surveillance, not measurement.
| Dimension | Observability | Evaluation |
|---|---|---|
| Question | What happened? Latency, tokens, errors, traces | Was the output correct, grounded, and safe? |
| When | Live traffic, after release | Before release on datasets, and sampled live |
| Data | Logs, spans, metrics from real requests | Inputs paired with expected behavior and scores |
| Comparison | Against time: yesterday vs. today | Against a baseline build on identical inputs |
| Action | Alert, debug, roll back | Merge or block, promote or hold a canary |
Worked example
Observability shows latency up and zero errors after v13 ships. Everything looks healthy. Only the eval report shows faithfulness and refusal correctness down, because wrong answers return HTTP 200.
How they compose
Traces feed evals: sampled production traces become scored cases and, once labeled, golden dataset entries. Observability is the supply chain; evaluation is the judgment.
Evals vs. guardrails
Guardrails intervene on a single live request. Evals decide whether the system as a whole is fit to ship. One is a runtime control; the other is a release control.
| Dimension | Guardrails | Evaluations |
|---|---|---|
| Scope | One request or response at a time | The full system across a dataset |
| Timing | Inline, in the request path, adds latency | Offline in CI, plus sampled checks online |
| Action | Block, redact, rewrite, or refuse that response | Block or approve the release itself |
| What it cannot do | Detect a slow quality regression across thousands of allowed responses | Stop a single harmful response in real time |
| Example | PII filter on outputs; a refusal classifier in the path | Refusal correctness scored over the should-refuse set before merge |
And guardrails themselves need evals. A guardrail is a model or ruleset with its own false positives and false negatives. You measure those with the same machinery: a labeled dataset, a confusion matrix, and thresholds, exactly like slide 7.
Engineering question: where does this fit into my deployment pipeline? Next three slides.
The evaluation pipeline, end to end
The PR is the trigger, the pipeline is the judge, and the gate is the decision. This is the architecture the course builds, piece by piece.
- Golden Dataset
- Retrieval Metrics
- LLM Judge
- Deterministic Checks
Dataset → Model → Scorers → Report → Gate
What each stage guarantees
- Dataset: the questions never change mid-comparison. Version it like code.
- Model: one pinned candidate per run. No silent provider drift inside a run.
- Scorers: cheap deterministic checks first, judge only where judgment is needed.
- Report: a diff, not a score. Baseline vs. candidate, per category, with failing case IDs.
- Gate: thresholds agreed before the run, so the decision is not negotiated after it.
Where evaluation fits into CI/CD
Two speeds keep the gate affordable: a fixed subset on every PR for a fast signal, and the full suite nightly for the complete picture.
Gate policy used in this course
- Block on any critical category regression: refusal, faithfulness, safety.
- Block on an overall drop beyond the suite noise band.
- Warn, do not block, on cost and latency drift. Page a human, not the pipeline.
- When the gate fires, read the failures, not the score. The report links every failing case.
Engineering question: how do I know whether to block a release? The gate policy answers it before the PR exists.
What this course will build
One project, built in order: a versioned eval suite in a repo that runs on every PR, compares candidates against a baseline, and fails the build when quality regresses.
The artifacts you will own
- Golden dataset: stratified JSONL with expected answers, sources, and refusal labels
- Scorer library: deterministic checks, retrieval metrics, citation checks
- Calibrated LLM judge: a rubric validated against your own hand labels
- Eval report: baseline vs. candidate diff, per category, with failing cases
- CI gate: GitHub Actions workflow that merges or blocks on the report
The skills behind the artifacts
- Failure analysis: cluster failures into causes before touching a prompt
- Judge calibration: measure judge vs. human agreement and fix the rubric, not the labels
- Threshold setting: separate real regressions from run to run noise
- Suite maintenance: fold production failures back into the dataset, re-baseline deliberately
Tooling stays boring on purpose: plain Python, pytest, JSONL, GitHub Actions. Tracing and dashboards appear as an optional layer, never as the core idea.
Repository structure: one suite, one home
Everything the pipeline needs lives in the repo and is versioned with the code it judges: dataset, scorers, judge prompts, baselines, and the workflow.
Versioning rules that keep it honest
- Dataset changes are PRs too. Editing the golden set to make a failing build pass is the one unforgivable move.
- Judge prompts are code. Changing the judge changes the measurement, so judge changes re-baseline the suite.
- Baselines are saved runs, not remembered numbers. Every report names the baseline it diffed against.
- Broken variants ship in the repo on purpose, so each chapter can break the system and watch the suite catch it.
Course roadmap: nine chapters, one suite
Part 1 · Foundations
- 1. Why your app is already broken YOU ARE HERE
- 2. Anatomy of an eval suite: the four scorer layers
- 3. Build the golden dataset
Part 2 · Scoring
- 4. Score what you can without an LLM first
- 5. LLM-as-judge, done properly: rubrics and judge bias
- 6. Calibrate the judge against humans
Part 3 · Shipping
- 7. RAG and agent evals, where averages lie
- 8. Put it in CI: the suite that blocks bad releases
- 9. Keep it alive: online evals, drift, maintenance
How the chapters compound
Each chapter ends with a failure the suite catches that it could not catch before. Chapter 3's dataset makes Chapter 4's scorers meaningful; Chapter 6's calibration is what makes Chapter 8's gate trustworthy; Chapter 9 keeps the whole thing true after launch. Downloads for the course: repo template with the sample RAG app and broken variants, golden dataset template, judge rubric template, failure analysis worksheet, and the CI workflow file.
Key takeaways, and what comes next
Five things to keep
- "Looks good" is not a metric. If a script cannot act on it, it cannot gate a release.
- Demos are biased samples. Small, cherry picked, and missing exactly the cases that fail in production.
- An eval is dataset, system, scorers, report, gate. Drop any one and you have an opinion again.
- Measure by category. Faithfulness, retrieval, refusal, and citations fail for different reasons and need different fixes.
- The gate is the point. Thresholds agreed in advance turn measurement into a deployment decision.
Next video · Chapter 2
Anatomy of an Eval Suite. The four scorer layers: deterministic checks, reference based scoring, LLM-as-judge, and human review, and how to evaluate by capability instead of chasing a single average score.
Bring: your running RAG app and one change you are afraid to ship. Chapter 2 gives it a measurement.