The Gate
Wiring the eval suite into CI so a bad release blocks itself. Chapters 4 to 7 built scores you can trust: deterministic checks, a calibrated judge, stage attribution for RAG and agents. None of that changes a single release decision until a policy file, a baseline, and a required status check stand between a pull request and the merge button. This chapter builds that machinery, with its decision arithmetic exposed.
By the end, you can answer four questions
- Where do gate thresholds come from, if not from someone picking a round number?
- What exactly happens, step by step, between PR opened and merge blocked?
- Why is an infrastructure error a different outcome from a quality failure, and who acts on each?
- How do you roll a gate out so the team keeps it switched on?
How this lecture runs
- 20 to 25 minutes, 22 slides: one PR, followed from open to blocked merge
- Section 1: policy as code, slides 4 to 7
- Section 2: pipeline engineering, slides 8 to 14
- Section 3: operating the gate, slides 15 to 21
- Every threshold on screen is either derived from measured noise or labeled an example policy value
The v13 prompt PR, reopened. This time it cannot merge.
The deliverable, precisely
- gate.yaml: the release policy as a reviewed file in the repo. Slide 4.
- A two tier pipeline: a fast PR subset and a full nightly run that owns the baseline. Slides 8 and 14.
- A PR comment report: paired deltas plus the new failures list a reviewer actually reads. Slide 11.
Recap: what Chapters 4 to 7 produced
- Chapter 4 scored single answers deterministically, with paired diffs against a baseline.
- Chapters 5 and 6 built the judge, then calibrated it and marked each capability GATE, WARN ONLY, or HUMAN ALWAYS.
- Chapter 7 attributed every RAG and agent failure to its first failing stage, and set stage thresholds: Recall@10, tool arguments, pass@1, zero tolerance policy violations.
The gap this chapter closes
A score nobody enforces is a dashboard. Dashboards get read after incidents. The gate moves the same numbers into the one place a release cannot route around: a required status check on the protected branch. Everything in this chapter is plumbing in service of that single fact.
Eight components between a pull request and a decision
Two outputs leave every run, win or lose: the status check that branch protection reads, and the report artifact plus PR comment that humans read. The first decides. The second explains.
The release policy is a file, reviewed like code
Field by field
Measure the noise first. Then place the threshold outside it.
The protocol
- Freeze everything: same model, same prompt, same index, same dataset version. The candidate is the baseline.
- Run it 5 times. Any spread across identical runs is pure measurement noise: judge variance, sampling, serving nondeterminism.
- Record the spread per capability, max minus min across the 5 runs.
- Set max_drop_points above the spread and below the smallest regression worth blocking. If no such gap exists for a capability, that capability cannot gate yet. It stays warn while you grow the slice or cut judge variance.
Example noise run · Faithfulness, 5 identical runs
| Run | Faithfulness | Note |
|---|---|---|
| 1 | 95.2 | |
| 2 | 94.8 | Lowest of the five |
| 3 | 95.6 | Highest of the five |
| 4 | 95.0 | |
| 5 | 95.4 |
gate.yaml threshold = drop over 2.0 points blocks
The 2.0 threshold sits outside the 0.8 point noise spread and inside what the slice can resolve: at the gate subset slice sizes on slide 9, a 2 point paired shift is detectable, per the Chapter 3 confidence band logic, because paired comparison removes per case difficulty from the variance.
A threshold placed inside the noise band fires on unchanged code roughly as often as the noise crosses it. The team learns the gate cries wolf, starts rerunning CI until green, and the gate is dead while still technically enabled. Noise first is not rigor for its own sake; it is what keeps the gate believed.
Three outcomes, decided by two numbers per capability
| Capability | Baseline | Candidate | Delta | Paired 95% interval | Threshold | Outcome |
|---|---|---|---|---|---|---|
| Faithfulness | 95.6 | 95.1 | -0.5 | [-1.1, +0.1] | block over 2.0 | PASS |
| Refusal correctness | 88.5 | 86.2 | -2.3 | [-4.9, +0.3] | block over 5.0 | WARN |
| Citation accuracy | 96.8 | 91.5 | -5.3 | [-7.0, -3.6] | block over 2.0 | BLOCK |
PASS
Drop is within the noise band. Faithfulness moved 0.5 points against a measured spread of 0.8. Nothing to review; the row renders green and the run moves on.
WARN · human review
Drop sits between noise and threshold, or the paired interval includes zero. Refusal fell 2.3 points, but the interval [-4.9, +0.3] cannot exclude no change. A human reads the flipped cases before merge.
BLOCK
Drop is beyond the threshold and the interval excludes zero. Citation accuracy fell 5.3 points against a 2.0 threshold, worst case end of the interval still a 3.6 point loss. The status check fails.
The interval does the epistemic work: the point delta says how much, the interval says how sure. Blocking on a point delta alone blocks on noise one run in twenty by construction.
Calibration decided who may block. The gate only wires it up.
Why the wiring is one directional
Promotion flows from calibration evidence, never from convenience. A capability moves from WARN ONLY to GATE only through a new Chapter 6 calibration run with the agreement numbers attached, landed as a gate.yaml change with eval owner review. A team under release pressure cannot promote a metric inside the PR that metric is failing. That is slide 17's standing rule, enforced here by file ownership.
A fast tier guards the PR. A full tier owns the truth.
PR tier · target under 10 minutes
| Stage | Minutes |
|---|---|
| Checkout, setup, config extraction | 0.8 |
| Generate: 150 subset cases | 3.4 |
| Deterministic + reference scorers | 0.3 |
| Judge pass, gate capabilities only | 3.2 |
| Paired compare + policy engine | 0.4 |
| Upload artifacts, post PR comment | 0.3 |
| Total | 8.4 |
Fixed stratified 150 case subset, slide 9. Deterministic checks on everything, judge only where Chapter 6 granted gate trust. 1.6 minutes of headroom absorbs runner variance without breaching the target.
Nightly tier · completeness over speed
- Full 500 cases from golden_v1.2, every capability, every scorer.
- Agent tasks at 3 repeated runs, per the Chapter 7 protocol, so pass@1 and pass-all-3 both trend.
- Full judge pass, including WARN ONLY capabilities, whose trend lines are the evidence base for future promotion.
- Report trended, not gated: nightly output feeds the baseline on merged code, slide 14, and the gate health metrics on slide 20.
- Failure mode it catches: slow drift inside the PR subset's blind spots, and subset level noise that only the full slice can resolve.
Same 150 cases on every PR, or no two PRs are comparable
Stratification · proportional, with floors
| Capability slice | Full set | Subset |
|---|---|---|
| Faithfulness | 200 | 60 |
| Refusal | 120 | 36 |
| Citation accuracy | 100 | 30 |
| Safety | 80 | 24 |
| Total | 500 | 150 |
Proportional at 30 percent per slice, then checked against gate.yaml: every gate slice must clear its min_cases floor at subset scale for paired comparison, and difficulty strata from Chapter 3 are preserved inside each slice, so the subset is not 150 easy cases.
Why fixed beats random
Random per run sampling makes each PR fight a different exam. A delta between two PRs then mixes the code change with the sample change, and the paired interval on slide 6 loses its meaning, because pairing requires identical case ids on both sides. Fixed ids restore the pairing and make the subset itself a versioned artifact: gate_subset_v1, changed only by its own reviewed PR.
Why fixed beats hand picked
A hand picked subset drifts toward cases the current system passes; that is how demos were built in Chapter 1. Stratified selection from the frozen set, with floors per slice, is mechanical and auditable. Anyone can regenerate the subset from the rule and get the same 150 ids.
One workflow. One step owns the exit code.
Three properties to notice
The report a reviewer reads leads with new failures
Eval gate report · PR #214 · candidate prompt v13 vs baseline
| Capability | Baseline | Candidate | Delta | Status |
|---|---|---|---|---|
| Faithfulness | 95.6 | 90.8 | -4.8 | BLOCK |
| Citation accuracy | 96.8 | 91.5 | -5.3 | BLOCK |
| Refusal correctness | 88.5 | 86.2 | -2.3 | WARN |
| Safety violations | 0 | 0 | 0 | PASS |
| Latency p95 | 1.8 s | 2.0 s | +11% | WARN |
New failures · passed at baseline, fail now · 14 cases
faith-0031 · faith-0107 · cite-0019 · cite-0044
cite-0082 · ref-0056 · + 8 more in failures.jsonl
Each id links to the case, the baseline output, the candidate output, and its Chapter 7 first failing stage. This list is the review surface. The author works it case by case; nobody debugs an aggregate.
Fixed · failed at baseline, pass now · 3 cases
faith-0060 · ref-0012 · cite-0101
Improvements are reported with the same precision as regressions. A change that fixes 3 and breaks 14 is a net decision a human can now make with both lists open, instead of a single delta argued over in comments. Full records: failures.jsonl in the run artifact.
Cache the judge. Never cache the candidate.
Cache keys, from Chapter 5
The token math · example run
150 cases x 2,070 tokens = 310,500 tokens / PR
Nightly judge pass
500 cases x 2,070 tokens = 1,035,000 tokens
plus agent tasks at 3 repeated runs, costed per Chapter 7 accounting
The 2,070 tokens per case is the Chapter 5 measured judge call size: rubric, evidence, and answer. Both totals land in the report as a cost field on every run, trended on slide 20, so cost drift is a reviewed artifact, not an invoice surprise at month end.
An infra error is never a quality verdict
| Outcome | Trigger | Exit code | What the author does next |
|---|---|---|---|
| PASS | All gate capabilities within policy; warn rows allowed | 0 | Merge, subject to normal code review. Warn rows are read, not actioned by rule. |
| QUALITY FAIL | At least one BLOCK row from the policy engine | 1 | Work the new failures list, slide 17 playbook. The verdict is about the change, and it stands until the change does. |
| INFRA ERROR | judge_error rate above its budget, runner timeouts, API failures mid run | 2 | Nothing about the change. The run reruns once automatically; a second infra error pages the eval owner, and the PR waits on infrastructure, not on quality. |
Why the judge error budget exists
If 8 percent of judge calls errored and returned no verdict, a report computed over the surviving cases is a report about a different, easier subset. The run declares itself infra error when the judge error rate crosses its budget, set in gate.yaml beside the thresholds, rather than quietly scoring what survived.
Why the distinction protects trust
Merge the two failure kinds into one red check and authors learn that red sometimes means the API was down, so red stops meaning anything. Three outcomes, three owners, three next actions: that separation is what lets a quality fail keep its authority.
The baseline is produced on main. It is never produced by the PR it judges.
The baseline lifecycle
- Produced nightly on merged code: the nightly tier runs on main, and its full report becomes the candidate baseline.
- Stored as a versioned artifact: baseline report carries dataset version, judge version, model and index versions, and the run id that produced it.
- Promoted by its own PR: adopting a new baseline is a small dedicated pull request with a changelog entry stating what changed and why scores moved.
- Reviewed like a dataset change, because it is one: every future delta is measured against this file.
The anti-pattern: the self ratifying baseline
Individually acceptable drops compound into a large quality loss that no single report ever shows, because each report compares against yesterday. A baseline that updates itself out of a failure is a ratchet pointed down.
The gate binds because the branch requires it
Settings · branch protection rule · main
The emergency override, priced in advance
Admins retain an override for genuine emergencies, and it is deliberately expensive. Using it requires, at the moment of use:
- A linked incident the override is relieving
- A postmortem filed on the incident
- A new golden case within one week covering the failure mode that forced the bypass, so the escape route closes behind it
Every override is logged with author, PR, and reason, and counted in the gate health metrics on slide 20. An override nobody counts is an unmarked second merge path.
A gate that blocks on day one gets disabled by Friday
Promotion between phases is earned by measurement, not scheduled by calendar: the phase advances when the measured false block rate sits at or below the level the team agreed in Phase 1. A gate that skips the measurement arrives at Phase 4 with unknown credibility and spends it in the first week.
Read the failures. Not the score.
The procedure, in order
- Open the new failures list from the PR comment, slide 11. Cases that passed at baseline and fail now are the entire initial scope.
- Classify each by first failing stage, using the Chapter 7 triage: retrieval, assembly, generation, attribution, tool selection, tool arguments, recovery.
- Fix the change or revert it. If the cluster points at one prompt hunk or one config line, that is the suspect; verify by rerunning the gate on the fix.
- Add a case for any new failure mode the failures exposed, through the Chapter 3 dataset process, so this exact regression can never pass silently again.
The standing rule
Thresholds and the dataset are never edited in the same PR as the failing product change.
A threshold change gets its own PR, its own noise evidence, and eval owner review under slide 18. The mechanism matters more than the intention: if the PR that fails the gate can also edit the gate, then every block is negotiable at exactly the moment negotiation pressure peaks, and the gate's verdicts stop being evidence about the product.
Whoever can edit the gate can pass anything
The rules
Minute by minute: the Chapter 1 failure, replayed against the gate
The paired report · Chapter 1 example, candidate v13 vs baseline v12
Policy engine verdict on these rows: BLOCK BLOCK BLOCK three BLOCK rows. Status check fails. The merge button is disabled by branch protection. In Chapter 1, this exact report was produced after the release. Here it is produced before it.
The timeline
A gate with no measured false block rate is an opinion with an exit code
| Metric | Definition | What a bad value triggers |
|---|---|---|
| False block rate | Blocks that reviewers, reading the flipped cases, judge incorrect / all blocks | Threshold and noise review. This is the Phase 1 measurement from slide 16, kept permanently, not a rollout one off. |
| Escape rate | Regressions found in production that the gate passed / releases in the period | Every escape becomes a golden case through Chapter 3, and its stage attribution names the scorer or slice that missed it. |
| Eval duration p95 | 95th percentile wall clock of the PR tier run | Subset or judge pass redesign before authors start routing around a slow gate. |
| Cost per PR | The run cost field from slide 12, aggregated per PR | Cache hit review and tier rebalance. Cost is a design constraint with a number, reviewed like latency. |
| Override count | Emergency overrides used, from the slide 15 log | Any rising trend is a governance review, whatever the quality metrics say. |
The gate is a product with users, failure modes, and an operating cost. These five numbers are its eval suite. A team that measures its model this carefully and its gate not at all has the rigor exactly backwards, because the gate is the component every other measurement flows through.
The gate will be gamed, drift, and erode. Plan for all three.
| Threat | Mechanism | Countermeasure |
|---|---|---|
| Goodhart pressure | Prompts get tuned against the frozen gate set until the set measures prompt fitting, not quality | Chapter 3 holdout rotation: a rotated holdout slice is scored nightly; a widening gap between gate set and holdout scores is the detection signal. |
| Subset memorization | The fixed 150 are known, so changes can be aimed at exactly those cases | The nightly full 500 owns the baseline, and subset membership is regenerated by rule when the dataset version bumps, never by the teams being gated. |
| Judge drift | The judge model changes or decays, and its verdicts quietly stop matching human labels | Chapter 6 recalibration has an expiry. Past it, judge backed capabilities lose gate trust automatically and drop to warn until recalibrated. |
| Override strangulation | Each override is defensible; the count compounds until the exception is the process | Overrides are counted on slide 20 and the count is reviewed. The countermeasure is visibility with a named reviewer, not a harder lock. |
None of these threats arrives as an attack. They arrive as reasonable local decisions by busy people, which is why every countermeasure above is a measurement or a rotation, not an exhortation to be careful.
The Gate
- Policy is a file, and thresholds come from measured noise. gate.yaml is reviewed like code; a threshold inside the noise band builds a gate nobody believes.
- Three outcomes, kept distinct. Pass, quality fail, infra error: different exit codes, different owners, different next actions. An infra error is never a quality verdict.
- The baseline never updates itself out of a failure. It is produced on main, promoted by its own PR, or it becomes a ratchet pointed down.
- Roll out in phases and measure the gate like a product. False block rate is the promotion criterion, and gate health metrics run for as long as the gate does.
Next video · Chapter 9
After the merge. The gate guards the release; it says nothing about next month. Video 9 covers online evals on sampled production traffic, production drift against the golden set's assumptions, and keeping the whole system, dataset, judge, and gate, honest over time.
Bring: your gate health table from slide 20, and the date your judge calibration expires.