After the Merge
Online evaluation, drift, and keeping the eval system honest after the gate. Chapter 8 made the release decision trustworthy at merge time. This chapter covers everything that happens after: sampling production traffic for scoring, telling input drift apart from provider drift apart from a changed ruler, and the maintenance operations that keep the dataset, the judge, and the gate from quietly expiring.
By the end, you can answer four questions
- Which production traces may estimate a rate, and which may only be mined for cases?
- When a production score moves, how do you decide whether the inputs, the system, or the ruler moved?
- What does a label that arrives seven days late do to this week's online score?
- Which maintenance artifacts prove the eval system is still healthy, and who owns each one?
How this lecture runs
- 20 to 25 minutes, 22 slides: one loop, from merged release back to new golden cases
- Section 1: online evaluation, slides 4 to 8
- Section 2: drift, three kinds, slides 9 to 14
- Section 3: maintenance and eval debt, slides 15 to 21
- Every invented number is labeled example data or an example threshold on the slide where it appears
The gate answers one question. Production answers the other.
Offline · Chapters 3 to 8 · before the merge
The gate answers: should this ship?
Online · this chapter · after the merge
Production answers: is it still true?
The return path is the point of the chapter. Production is not a place the system goes to be forgotten; it is the dataset's editor, continuously proposing new cases, new failure modes, and evidence that the frozen set's assumptions have expired.
Two questions, not two tools
| Dimension | Offline evaluation | Online evaluation |
|---|---|---|
| Distribution | Frozen golden set. Fixed cases, fixed difficulty mix, fixed corpus snapshot. | Live traffic. Whatever users actually ask this week, in whatever proportion they ask it. |
| Labels | Prewritten. Expected behavior, sources, and must-not-contain are authored before the run. | Delayed and partial. Outcomes resolve hours to days later, slide 8, and most traces never get a human label at all. |
| Repeatability | Exact. Same case ids, same pairing, rerunnable on demand, Chapter 8. | Never. The traffic of last Tuesday cannot be replayed; only its logged traces can be rescored. |
| Primary job | Block regressions. Compare a candidate against a baseline on identical inputs. | Detect drift and new failure modes. Find what the frozen set never contained. |
Why online cannot replace the gate
Production traffic has no counterfactual. You cannot observe what the rejected candidate would have done on the same users, and the users themselves change when the system changes. Regression blocking needs identical inputs on both sides of a comparison, and only the frozen set provides them.
Why the gate cannot replace online
The frozen set is a photograph of what users asked when it was built. New topics, new vocabulary, new document versions, and provider side changes all arrive in production first. A gate that passes forever on a stale set is certifying a system that no longer exists.
Log everything. Sample in three streams, on purpose.
The log comes first
Every production trace is logged in full: query, retrieved chunk ids, prompt hash, model and index versions, output, tool calls, latency, trace id. PII is stripped at the log boundary using the Chapter 3 rules, before any sampling happens. Sampling decides what gets scored; it never decides what gets kept, because a failure you did not log cannot become a case later.
The bias rule, stated once and enforced in the report code: enriched samples never enter a rate. An escalation heavy sample reporting an escalation rate is measuring the sampler, not the system.
Online scoring reuses the offline scorers. It does not fork them.
Deterministic scorers · synchronous
Schema validity, citation target existence, refusal classification, latency and token accounting run on the sampled trace at log time. They are cheap, exact, and need no label, so their online numbers are available within minutes and cover the full log, not just the sample.
Judge · asynchronous batch
The calibrated judge from Chapters 5 and 6 runs in batch on the core sample only, with verdict caching keyed exactly as in Chapter 8. Judging the enriched stream would spend the budget where rates cannot be computed anyway.
Human audit · weekly queue
A fixed size queue drawn each week: part core sample, part enriched stream, part judge disagreement cases. Auditors label blind, as in Chapter 6. The queue is the calibration feedstock and the case intake's front door, slide 14.
Because both runners emit the same report schema, an online faithfulness number and an offline faithfulness number are the same measurement taken in two places, and a divergence between them is itself a finding: same scorer, different distribution. Fork the scorers and that comparison dies; every online number becomes its own dialect.
User behavior is a hint. Every hint has a confound.
| Signal | What it might mean | The confound that stops you trusting it alone |
|---|---|---|
| Thumbs down | The answer dissatisfied the user | Accidental taps and brigaded clicks. Only a small, self selected fraction of users rate anything, and they are not a random sample of the dissatisfied. |
| Immediate rephrase | The answer missed, so the user asked again differently | The user was refining a vague first question. Rephrase rate rises when users learn the system rewards detail, which is a success pattern wearing a failure costume. |
| Abandonment | The session failed and the user gave up | The user got the answer and left. For a support assistant, a short session with no follow up is the goal state as often as it is the failure state. |
| Escalation to human | The system could not resolve the issue | Policy required the escalation: account changes, refunds above a limit, regulated topics. Those escalations are correct behavior counted as failure. |
| Copy of the answer | Success proxy: the answer was worth keeping | The user is copying it into a complaint, a ticket, or a message to a colleague showing the error. Copy records the act, not the intent. |
The rule for implicit signals
Implicit signals trigger investigation; they never gate and never headline a report alone. A spike in rephrases buys a human audit draw oversampled from rephrased sessions. If the auditors confirm real failures, the failures enter case intake with labels attached. The signal opened the door; labeled evidence walks through it. Any dashboard that turns a confounded proxy into a quality percentage has built a random number generator with good branding.
Two ways to meet real traffic before real users depend on you
Shadow mode · zero user exposure
Shadow answers: how would the candidate behave on today's real distribution? It cannot measure user outcomes, because no user ever saw its answers. That limit is the price of zero exposure, and it is worth paying before any canary.
Canary · 5 percent of traffic, example thresholds
| Guardrail metric | Rollback if |
|---|---|
| Refusal rate vs control | moves over 3 points |
| Error rate (5xx + empty responses) | exceeds 1 percent |
| Escalation rate vs control | rises over 2 points |
| Latency p95 vs control | rises over 25 percent |
Thresholds are written into the rollout config before the canary starts, and breaching one triggers automatic rollback to the previous version, no meeting required. A threshold negotiated after the metric moves is not a threshold; it is a press release. Canary metrics are computed on the canary's own full log, never on the core sample, because at 5 percent of traffic the sample is too thin to catch a fast regression.
The label for Tuesday's answer may arrive next week
The join that makes online labels possible
Label delay · example distribution, support outcomes
The unresolved remainder is not random. Long running, multi touch, and disputed tickets resolve last, and they are disproportionately the hard cases. Scoring this week on resolved cases only grades the easy 40 percent and calls it the week.
The reporting consequence
Every online score carries its unresolved fraction beside it, computed at report time: faithfulness on the core sample is final, but outcome based success is reported as, for example, success 91.0% · unresolved 23%, and the number is restated when the window matures. A score whose denominator is still arriving is a preliminary result, and the report says so in the same font size as the score.
A score moved. Three different things could have moved it.
Input drift
Owner: product and eval team jointly. Fix: dataset intake, slide 14. Measured on slide 10.
System drift
Owner: platform team. Fix: changelogs, fingerprints, revalidation. Slides 11 and 12.
Measurement drift
Owner: eval team. Fix: recalibration protocol, Chapter 6, and version pinning everywhere.
The drift report separates the three by construction: input mix computed from the log, system changes read from changelogs and fingerprints, ruler changes read from scorer version fields. A report that merges them into one "quality went down" line sends three owners hunting for one cause that does not exist.
Measure the mix before blaming the model
Topic mix · share of core sample queries, example weeks
| Topic | Baseline week | Current week | Move |
|---|---|---|---|
| Billing | 18% | 27% | +9 pts |
| Setup | 22% | 20% | -2 pts |
| Refunds | 15% | 14% | -1 pt |
| API | 12% | 12% | 0 pts |
| Other | 33% | 27% | -6 pts |
| Total | 100% | 100% |
Topic labels come from the same classifier run on both weeks, version pinned; relabeling history with a new classifier manufactures drift that never happened.
Two companion signals
Why mix explains scores mechanically
Billing questions in this system carry the oldest policy documents and the highest refusal rate. If billing grows from 18 to 27 percent of the mix, the aggregate faithfulness and refusal numbers move even if per topic quality is completely unchanged. That is why the drift report always shows per segment scores beside the aggregate: the aggregate is a weighted average, and the weights just changed.
Retired evidence invalidates cases, not just answers
Corpus changelog · per week, example week
| Change | Count | Why the eval system cares |
|---|---|---|
| Docs added | 41 | New answerable territory with zero golden coverage until intake runs |
| Docs changed | 63 | Expected answers and must-contain strings may now be wrong |
| Docs retired | 7 | Cases pointing at them are unanswerable as written |
| Index rebuilds | 2 | Recorded in the run manifest diff: embedding model, chunking config, rebuild timestamp |
The changelog is a diff of document ids and content hashes between corpus snapshots, generated by the indexing job, not assembled by hand for the report.
The stale pointer check · Chapter 3 lint, run against production
The Chapter 3 dataset lint gains a production mode: for every golden case, resolve its expected_sources against the current corpus, not the snapshot the case was written against.
- Retired source: the case's evidence no longer exists. The case is stale, and any score it produces this week measures a ghost.
- Materially edited source: content hash changed beyond the lint's edit threshold. Expected answer fields are queued for human re-verification.
- Counted, not vibes: the check emits a number per dataset version, the stale case count, which feeds the health dashboard on slide 15.
A RAG system is evaluated against its corpus. When the corpus moves and the cases do not, the eval stops describing the deployed system while continuing to produce confident looking numbers.
A frozen probe set catches the change nobody announced
The fingerprint protocol
- Freeze a probe set: 200 prompts, fixed text, fixed parameters, temperature 0, spanning the system's capability slices. The probe set is not the golden set; it never gates and is never tuned against.
- Run it daily against the production endpoint, from a scheduled job with its own small budget line.
- Hash every output and diff against the previous day's outputs, case by case. The daily output diff rate is the fraction of probe outputs whose hash changed.
- Store the raw outputs. When the rate steps, the diff itself is the evidence: which slices changed, and how the text moved.
Daily output diff rate · example, 20 days
Baseline noise sits at 1 to 2 percent: serving nondeterminism flips a few outputs daily even at temperature 0. On day 14 the rate steps to 11 percent and stays there. Deploy log for day 14: no deploy on our side.
That step is a provider change until proven otherwise. The response is mechanical: trigger a full offline gate run against the current endpoint, compare capability scores to the stored baseline, and open the incident path on slide 19 if gate capabilities moved. Not an argument with the provider; a measurement.
Four signatures, four different first actions
| What the evidence shows | Diagnosis to test first | First action |
|---|---|---|
| Score moved and input mix moved | Mix shift, slide 10 | Investigate mix first. Compare per segment scores across the two weeks. If per segment scores are flat, there is no quality event yet; route new topics to case intake. |
| Score moved, mix flat, fingerprint stepped | Provider drift, slide 12 | Provider drift protocol. Full offline gate run on the frozen set against the current endpoint. The frozen set is what makes the provider's change measurable at all. |
| Score moved, mix flat, our deploy yesterday | Regression | Chapter 8 regression protocol. The deploy diff is the suspect list; work the new failures by first failing stage, and the gate thresholds decide whether rollback or fix forward applies. |
| Score moved and the judge or rubric version changed | Measurement drift | Chapter 6 recalibration protocol. Rescore a fixed output set under both judge versions. If verdicts differ, the ruler moved; recalibrate before any product conclusion is drawn. |
The cost of skipping this table is symmetric: treat drift as regression and you roll back a healthy deploy while the real cause keeps moving; treat regression as drift and a defect you shipped gets a monitoring dashboard instead of a fix. Diagnosis order is an engineering control, not a discussion topic.
A drift report is done when its cases land
Why the PR, not a script
Auto generated cases would import production's noise along with its signal, including mislabeled outcomes and one off user errors. The PR forces the Chapter 3 disciplines onto production material: an authored expectation, a named capability, a review. Volume is deliberately capped by reviewer time; the intake queue prioritizes new failure modes over more instances of known ones, because the gate already tests what it knows.
This is the loop from slide 2 closing. Every case that lands here is production teaching the frozen set something the frozen set could not have known on its own.
The eval system gets a dashboard too. One owner per row.
| Health metric | Definition | Alert, example | Owner |
|---|---|---|---|
| Stale case rate | Cases untouched across two corpus versions / all cases, from the slide 11 lint | over 8% | Dataset owner |
| Quarantined case count | Cases parked in Chapter 7 quarantine: disputed expectation or unstable verdict | over 15 open | Dataset owner |
| Calibration days remaining | Days until the Chapter 6 judge calibration expires | under 14 days | Eval owner |
| Coverage gaps | Shipped product features and tools with zero golden cases | any high risk gap | Feature team |
| Judge error rate trend | judge_error share of online and CI judge calls, from Chapter 8 accounting | over 2% or rising 3 weeks | Eval owner |
| Gate false block rate | Blocks reviewers judged incorrect / all blocks, Chapter 8 health metric | over 10% | Eval owner |
Each row is a number the earlier chapters already produce; this slide's work is putting them on one page with a threshold and a name. Eval debt is invisible precisely because its symptoms appear in other systems' dashboards: the stale case shows up as a confusing gate failure, the expired calibration as a judge argument. The health dashboard is where those symptoms get their own line.
Three exits only. Everything else is a backlog that lies.
Weekly failure intake review · inputs
- Production failures confirmed through outcome joins and audits, slide 8 and slide 14
- Audit disagreements between human labels and judge verdicts from the weekly queue, slide 5
- Drift cases nominated by the drift report, slides 10 to 12
- Stale pointer results from the slide 11 lint: retired or edited expected sources
The three exits, applied to every item
- Add a case. Authored and reviewed per Chapter 3, scheduled into the next dataset release.
- Retire a case, with a reason. Evidence retired, behavior spec changed, duplicate of a stronger case. The reason is recorded in the dataset changelog; silent deletion is not an exit.
- Open a scorer bug. The case is right and the scorer is wrong: a lint gap, a judge rubric hole, a deterministic check misfiring. The fix lands in the scorer package, with the case as its regression test.
Dataset releases · versioned, on a cadence
No cases, no gate entry, no launch
The checklist is deliberately ordered: cases define the behavior, predicates make it checkable, calibration decides who may judge the rest, and the policy entry wires the result into the gate. Skipping to step 4 produces the failure mode this course opened with: a number that exists, attached to nothing, blocking nothing.
Honesty has a budget line. Report it next to the escape rate.
Judge tokens per month · example accounting
Total example: about 49.7M judge tokens per month, plus candidate generation tokens accounted separately in the same report. Human time, example: 24 hours per month: 12 audit and labeling, 8 intake review and case authoring, 4 calibration maintenance.
Levers when the budget grows
- Subset sizes: shrink the PR tier subset where slice counts exceed what the thresholds need to resolve, Chapter 8 math, never below min_cases
- Verdict caching: Chapter 5 cache keys make reruns of unchanged PR heads nearly free
- Judge tiering: capabilities with the highest calibrated agreement graduate to a cheaper judge model, revalidated by a Chapter 6 spot calibration
- Audit sample rate: the weekly queue size is a dial, turned down only with the disagreement rate in view
Why cost sits beside escape rate
The Chapter 8 escape rate, regressions production found that the gate passed, is the price of too little evaluation. The budget is the price of enough. Reported together, monthly, in the same review, a proposal to cut eval spend has to state which escapes it is buying. Reported separately, the cut looks free for exactly one quarter, until the escapes arrive.
Five days of drift protocol, in order
Nothing in this timeline required a hero. Each step was a protocol written down in an earlier chapter, executed in order. That is the entire argument of the course, demonstrated once at production scale.
Nine chapters, one loop
The arrow that closes the system
Case intake, Chapter 9: confirmed production failures and drift findings return to the golden dataset as reviewed case PRs, where they become part of the specification the gate enforces. Remove this arrow and the diagram is a pipeline that ages. With it, every component downstream of the dataset is also a sensor feeding the dataset, and the system's knowledge compounds instead of decaying.
Every video left an artifact in the repo
| Video | Title | Artifact |
|---|---|---|
| 1 | Why Your App Is Already Broken | failure taxonomy |
| 2 | Anatomy of an Eval Suite | suite architecture |
| 3 | Building the Golden Dataset | golden_v1.jsonl, schema.json, DATASHEET.md |
| 4 | Scoring Without an LLM | deterministic + reference scorers |
| 5 | LLM-as-Judge, Implemented | judge prompts + verdict cache |
| 6 | Calibrating the Judge | calibration report + trust table |
| 7 | RAG and Agent Evals | stage attribution |
| 8 | The Gate | gate.yaml + CI workflow |
| 9 | After the Merge | drift dashboard + intake process |
LLM Evals From Zero
- Evaluation is a system, not a metric. Dataset, scorers, judge, gate, and drift monitors, each with an owner and a failure mode of its own.
- The dataset is the specification and production is its editor. The frozen set defines correct behavior; production traffic continuously proposes what the specification is missing.
- Trust is measured per capability and it expires. Judge agreement, calibration windows, and gate authority are all numbers with dates on them.
- Gates work when policy is code and thresholds come from noise. A threshold inside the noise band builds a gate nobody believes; a policy in a wiki blocks nothing.
- Every escape becomes a case. The system learns from what it missed, permanently, in the dataset, or it repeats the miss on schedule.
Where to start on Monday
Build the smallest version of this loop first: one dataset, one scorer, one gate. Then grow it in the order these nine videos did, because each step's output is the next step's input, and skipping ahead produces artifacts with nothing underneath them. The course repo is the template; your system, your corpus, and your users supply everything that makes it true.
Bring to your first build session: 50 real queries from your own logs, and one release you regret.