Trust, Measured

By the end, you can answer four questions

• What does agreement mean once chance and the base rate are removed?
• How do you sample and label so the agreement number means anything at all?
• Which error does your gate actually pay for: the false pass or the false alarm?
• What happens to trust the day the judge model changes?

How this lecture runs

• 20 to 25 minutes, 22 slides: one judge, validated, trusted in pieces
• Section 1: build the validation set, slides 4 to 6
• Section 2: measure agreement per capability, slides 7 to 14
• Section 3: keep it calibrated, slides 15 to 21
• Every agreement claim shows its matrix, its formula, and its arithmetic

Why identical items are non-negotiable

Agreement is defined item by item. Humans and the judge must see the same output, same retrieved context, same rubric text. Compare a judge score on one sample against human labels on another and you have measured sampling noise, not the judge. One item set, two independent labelings, one comparison.

Why the pipeline ends in a policy, not a score

The output of calibration is decisions: which capabilities this judge may gate, which it may only report, which stay human. Collapsing that into one trust number repeats the Chapter 2 mistake one level up, a single average hiding exactly the capability that should never have been trusted.

Stratify on three axes

• Capability: faithfulness, instruction following, helpfulness, safety flagging. Sampled and reported separately, always.
• Difficulty: the easy, medium, and hard tags already on the golden set from Chapter 3.
• Predicted outcome: judge pass and judge fail items both, so every cell of the confusion matrix fills with real counts.

Then oversample failures and borderline cases. A random production sample is mostly easy passes: at a 95 percent pass rate, 100 random items yield about 5 fail items, and fail recall is unmeasurable on 5 items. Borderline means deterministic scores and the judge disagree, or the judge claim table splits. That is where calibration information lives.

The precision note, with the arithmetic

Agreement is a proportion, and its 95 percent band is roughly 1.96 × sqrt( p × (1 - p) / n ). At observed agreement p = 0.85:

The band is per capability, not overall. Two hundred items spread across four capabilities is 50 each, and a plus or minus 10 point band cannot certify a gate. Budget items per capability first, then total them.

The protocol

1. Blind, both ways. The labeler sees the query, the retrieved context, and the candidate output. Never the judge verdict, and never which model or prompt produced the output.
2. The guide is the rubric, verbatim. The labeling guide is the Chapter 5 rubric text, unchanged. If humans apply a different standard, you measure rubric drift, not judge error.
3. Evidence with every verdict. Labelers record the evidence span behind each verdict, the same requirement the judge works under. A verdict without a span cannot be adjudicated later.
4. Independence first. No conferring during labeling. Discussion happens only in adjudication, and the original labels are kept alongside the outcome.

Human versus human agreement · example calibration

Example calibration: 60 of the 200 items per capability carried two independent labels. These four numbers are the ceilings for slide 12.

The arithmetic

The degenerate judge test

Replace the judge with a rubber stamp that passes everything. On this item set it agrees with humans on 78 of 100 items: raw agreement 78 percent, and it catches zero failures. Its kappa: Po = 0.78, Pe = (0.78 × 1.00) + (0.22 × 0.00) = 0.78, so kappa = (0.78 - 0.78) / (1 - 0.78) = 0.00. Exactly what it deserves. With a 78 percent pass base rate, raw agreement hands a useless judge a passing grade. Kappa does not.

Reporting discipline

Kappa never travels alone. Report it beside the matrix and the raw agreement: kappa compresses both error types into one number, and slide 9 shows the two errors have wildly different prices. A capability can post a respectable kappa while its false passes, the expensive error, stay too high to gate on.

Consequence: bias the judge toward failing

• Require a quoted evidence span for every claim the judge marks supported. No span, no pass.
• Any unsupported claim fails the item. Not a deduction, not a partial credit: a fail.
• Borderline resolves to fail. A false alarm costs a review. A false pass costs a user. Spend the cheap error.

The headline number changes

Report the false pass rate as the headline calibration number, with agreement and kappa as supporting statistics. The gate exists to stop bad releases, so the statistic that prices a bad release slipping through leads the report. In the example calibration that number is 4 / 22 = 18.2%, and it is why faithfulness does not gate yet at slide 11's first pass.

Delta after the fix loop · example calibration

Same items, same human labels, judge prompt v3 to v4 with rubric_v4. Faithfulness now reads 0.78 against a human ceiling of 91 percent raw agreement. Helpfulness improved and still sits far below its 82 percent ceiling. Both facts go into the trust decisions next, unedited.

The verdict set is pass, fail, abstain

• Evidence missing: the criterion needs retrieved context the item does not contain. The judge cannot verify what it cannot see.
• Criteria conflict: an answer is faithful to the context and violates the format instruction on the same claim, and the rubric gives no precedence.
• Uncovered claims: claim decomposition produces a claim type the rubric does not address.

Every abstention routes to human review. An abstention is never silently counted as a pass, and never dropped from the denominator without being reported.

Coverage, reported beside agreement

The protocol

1. Swap consistency first. Rerun every pair with the order swapped, the Chapter 5 check. A pair the judge answers inconsistently across the swap is position noise, and agreement is not computed until that rate is measured and reported.
2. Same pairs, both judges. Humans and the judge choose on identical pairs: A better, B better, or tie.
3. Ties are explicit. A tie is a choice, counted in the comparison. Ties are never discarded and never split into half agreements to make the number prettier.

Example calibration, 120 pairs

judge_golden_v1, and when it reruns

• The set is frozen and versioned like the golden dataset from Chapter 3: items and human labels fixed, changes only as a new version.
• On every judge model change, including the silent provider side kind. You detect those by rerunning, not by reading changelogs.
• On every judge prompt or rubric change. A rubric edit is a new instrument; slide 11's loop is the small version of this rule.
• Monthly, even if nothing changed. Candidate output distributions drift on their own, and the judge was calibrated on the old distribution.

The revocation rule is automatic

Calibration records a band, not just a point: faithfulness agreement 89 percent with a band of plus or minus 5 points at n = 200, from slide 4's arithmetic. A scheduled rerun landing outside the recorded band revokes GATE trust automatically for that capability, until a recalibration re-earns it. No meeting, no exception path, no grace period. Warn only is the state trust decays into, and the runner enforces it from the report's expiry and band fields, slide 19.

The mechanism of the inflation

Every rubric fix in the fix loop was motivated by specific disagreements on specific items. Report agreement on those same items and the number stops measuring judge quality on new outputs. It measures how well the fixes memorized the items that inspired them. Nothing looks wrong while it happens. The matrix just gets kinder, one motivated edit at a time.

The contamination rule

If a held-out item is ever used to fix a rubric, that item moves to the dev slice permanently, and the held-out slice is rebuilt from freshly labeled items before the next trust decision. There is no unseeing an item. This is Chapter 3's contamination rule, applied to the judge's dataset instead of the candidate's.

Why cross family, and what it buys

Agreement between two judges from different model families is real evidence, because their errors are less correlated. Same family judges, even at different sizes, share training data and failure modes: their agreement is weak evidence wearing a second badge. What the ensemble buys is a smaller, better targeted human review queue, the disagreements, which calibration already identified as the informative items.

The honest price

Weekly judge versus human agreement, production sample

Example audit series: 88, 89, 90, 87, 88, 85, 83, 81 across eight weeks. Week 7 prints below the band's lower edge, two weeks before any monthly recalibration would have looked.

Why each field earns its place

What never happens

• Never average the failure away. A failing capability folded into an overall judge score is a gate decision made by arithmetic nobody reviewed.
• Never lower the threshold until it passes. That converts calibration into a search for a number the judge can already hit.
• Never relabel humans into agreement. The labels are the standard. Editing the standard to match the instrument ends the measurement and keeps the costume.

Each rung of the ladder is a legitimate operating state with a name, an owner, and a path back up: recalibrate. Teams that predefine the ladder report failures. Teams that do not, hide them.

The architecture enforces the calibration

Only trusted scores can block. The runner reads the trust table before scoring, and an expired or missing calibration report drops the judge to warn only mode by default. Trust is not a comment in a wiki page. It is a config file the pipeline obeys.

Changing trust is a reviewed act

The trust table changes only through a recalibration run, and the diff lands in version control next to the report that justifies it. Anyone can read which numbers, on which items, under which rubric version earned each capability its GATE. That audit trail is what makes the Chapter 8 gate defensible.

1. Calibrate per capability against blind human labels on identical items. A judge trusted in general is trusted nowhere in particular.
2. Report the matrix, kappa, and false passes, not a vibe. Raw agreement flatters at high base rates; the false pass rate is the number the gate pays for.
3. Humans set the ceiling, disagreements set the worklist. Measure human versus human agreement first, then bucket every disagreement and fix what owns it.
4. Trust expires. Freeze a judge validation set, rerun it on every judge change and monthly, and let an out of band result revoke the gate automatically.