LLM-as-a-Judge, Implemented
A scorer made of a model, run like an instrument, not an oracle. Layer 3 from Chapter 2, built for the capabilities Chapter 4 cannot measure: faithfulness, helpfulness, and instruction following quality.
By the end, you can answer four questions
- Which judging mode fits which decision: gating, selection, or coverage?
- What exactly goes into a judge prompt, in what order, and why?
- How large are position, verbosity, and self preference bias on your own suite?
- What does one full judge pass cost, in tokens you can compute?
How this lecture runs
- 20 to 25 minutes, 22 slides: one scorer, designed, attacked, and costed
- Section 1: design the judge, slides 4 to 11
- Section 2: biases, measured not assumed, slides 12 to 16
- Section 3: run it like infrastructure, slides 17 to 22
- Every bias claim in this chapter is an experiment with a protocol, not a warning label
One more scorer, same contract, same report
The deliverable, precisely
- A judge scorer that implements the Chapter 4 contract: score(case, output) returns value, passed, reason, metadata. Slide 19.
- Structured verdicts per case per criterion, validated in code before they reach report.json. Slides 8 and 16.
Recap: what Chapter 4 established
- Deterministic checks are free and exact. Structure, citations that resolve, refusal behavior, tool call validity, latency, cost.
- Reference scoring compares against ground truth. Exact match, token F1, retrieval recall at k, with known failure modes.
- Every score is a diff against a frozen baseline, per capability, with n.
And what it is blind to
A citation can resolve to a real document and still not support the claim it is attached to. An answer can contain every required token and still invent a detail between them. Faithfulness, helpfulness, and whether instructions were followed in spirit are semantic judgments. No string operation computes them. That gap, and only that gap, is what the judge is for.
The judge is one scorer. Never the pipeline.
What did not change
- Generation is still separate from scoring. The judge reads cached outputs, so rejudging never regenerates candidate answers.
- The Score contract is unchanged. The aggregator cannot tell a judge scorer from any other scorer, and that is the point.
- Deterministic scorers still run first in the report. If a fact can be checked in code, the judge never gets a vote on it.
What is new, and what it costs you
One scorer in this pipeline calls a model, costs money, and can disagree with itself. Everything else in this chapter follows from those three facts: the prompt is versioned like code (slide 10), the output is validated like untrusted input (slides 8 and 16), variance and bias are measured like instrument error (slides 11 to 14), and the spend is accounted like infrastructure (slide 17).
Three modes, three different decisions
| Mode | Judge sees | Verdict shape | Use it for |
|---|---|---|---|
| Pointwise rubric | One output, plus context and rubric | Per criterion pass or fail, absolute | Release gating. The question is whether this output meets a fixed standard, with no second output involved |
| Pairwise | Two outputs for the same case, labeled A and B | A wins, B wins, or tie | Model or prompt selection. The question is which of two candidates is better, not whether either is good enough |
| Reference based | One output, plus the expected answer from the dataset | Coverage and contradiction vs. reference | Factual coverage. The question is whether the output covers what the reference requires and adds nothing that contradicts it |
This course defaults to pointwise
Release gates need absolute criteria that hold still while candidates change. A pointwise verdict on Tuesday means the same thing on Friday. A pairwise win rate does not: it moves when the opponent changes, even if the candidate did not.
Mixing modes in one score is a category error
Averaging a pointwise faithfulness pass rate with a pairwise win rate produces a number with no decision attached to it. It cannot gate, because it is not absolute. It cannot select, because it is not relative. Record each mode as its own report row, or do not record it at all.
Seven parts, in a fixed order
Why the order is load bearing
- Context before candidate: the judge reads the evidence before the claim, so verdicts anchor on the source, not on the answer's fluency.
- Rubric after the case material: criteria sit closest to the output schema that applies them, and cannot be diluted by the candidate text above them.
- Schema last: the final instruction the model reads is the shape of its reply. Prompts that bury the schema in the middle get prose back.
Delimiters are a security boundary, not formatting
Use markers the candidate cannot plausibly produce by accident, and treat any marker appearing inside the candidate block as content, not structure. The parser splits on the first begin marker and the last end marker only. A candidate that prints its own end marker does not get to close the block early. Slide 15 shows why this rule exists.
Binary criteria. Scales drift, pass conditions do not.
A 1 to 10 quality score asks the judge to invent a scale, hold it steady across 500 cases, and reproduce it next week. It does none of those reliably. A binary criterion asks one falsifiable question.
The vague criterion
- Unfalsifiable: a 7 and an 8 cannot be argued about, because nothing defines either
- Drifts between runs: the same output scores differently as surrounding cases shift the judge's implicit scale
- Drifts between judges: two judge models with different score habits produce incomparable reports
- Ungateable: no one can write a release policy against a mean of invented numbers
The decomposition
| Criterion | Pass condition |
|---|---|
| Answers the question asked | The response addresses the query directly, not an adjacent topic |
| Supported by context | Every factual claim traces to a quoted context span. Slide 9 |
| Complete | All parts of a multi part query are addressed, or the gap is stated |
| No invented detail | Zero numbers, dates, or policies appear that are absent from the context |
Each row can be checked, disagreed with, and fixed independently. When Chapter 6 finds a criterion the judge gets wrong, you repair that row, not the judge's taste.
The design rule: if two competent humans could read a criterion and apply it differently to the same output, the criterion is not finished. Rewrite it until the pass condition names observable evidence in the output or the context. Three point scales (pass, partial, fail) are acceptable only where partial has its own written definition. Vague middles are where drift hides.
Faithfulness, as three checkable claims
| Sub criterion | Pass example · Support RAG | Fail example · Support RAG |
|---|---|---|
| F1 · Every factual claim is supported | Answer: "Refunds are available within 30 days of purchase." Context span states the 30 day window. Every claim has a span. | Answer adds: "Enterprise plans get 45 days." No context span states a 45 day window for any plan. |
| F2 · No claim contradicts the context | Answer: "SSO is included on Enterprise plans." Context lists SSO under Enterprise. Consistent. | Answer: "SSO is included on Team plans." Context lists SSO under Enterprise only. Direct contradiction. |
| F3 · Citations support the claims they mark | The 30 day claim carries a citation to the refunds document, and that document contains the window. | The refund claim cites the invoices document, which covers payment timing and says nothing about refunds. |
How the three combine
All three are critical criteria for this capability, so the overall verdict is their conjunction. There is no partial faithfulness score to hide behind: an answer with one invented number has failed the property users care about.
Note what Chapter 4 already covers
F3 is only the semantic half of citation checking. Whether the cited document exists and whether it was in expected_sources were settled deterministically on slide 8 of Chapter 4. The judge is asked only the question code cannot answer: does this document support this claim?
A verdict is JSON, or it did not happen
Field by field
The parse failure path
- Parse and validate. JSON parse, then schema validation. Any failure captures the raw text and the validator error.
- Retry once, with the schema error appended to the prompt, so the judge can correct its own formatting.
- Still failing: record judge_error for that case, increment the run level judge_error rate, and exclude the case from judge aggregates with the exclusion counted. Never silently drop it, and never coerce prose into a pass.
Decompose the claims before scoring them
The judge prompt orders the work: split the candidate answer into atomic claims, find a context span for each, and only then apply the rubric. A verdict produced before the evidence exists is an impression. Candidate answer, worked in full:
| Claim | Atomic claim from the candidate | Quoted context span | Supported? |
|---|---|---|---|
| 1 | Refunds are available within 30 days of purchase | "you may request a refund within 30 days of purchase" · billing/refunds | YES |
| 2 | The refund returns to the original payment method | "refunds are issued to the original payment method" · billing/refunds | YES |
| 3 | SSO is included on Enterprise plans | "Enterprise plans include SAML SSO" · plans/enterprise | YES |
| 4 | Enterprise plans get a 45 day refund window | No span found. The refunds doc states one window, 30 days, with no plan specific exception | NO · INVENTED |
The verdict follows mechanically
F1 requires every claim supported. Claim 4 has no span, so F1 fails, and faithfulness fails, regardless of how good claims 1 to 3 are. Three quarters supported is not a score. It is a failed criterion with a precise location.
Why this ordering works
- The rationale is generated from this table. It can only cite claims and spans that exist, so it cannot invent a justification after the fact.
- Failures become debuggable. The record names the exact unsupported claim. Compare that with a bare score of 6 out of 10.
- Humans can audit the middle step. In Chapter 6, reviewers check claim tables, not just verdicts, which is where judge errors actually live.
Pin the judge separately from the candidate
The hard rule
Never let a model be the sole judge of its own outputs.
Self judging stacks two correlated systems: the candidate's blind spots and the judge's blind spots are the same blind spots. Slide 14 measures the leniency this creates, and Chapter 6 tests it against human labels.
| Judge choice | Cost and latency | Agreement and risk |
|---|---|---|
| Larger judge | Highest tokens per verdict, slowest verdicts, dominates the run budget on slide 17 | Example run: highest agreement with human labels on hard criteria. Still biased, slides 12 to 14. Cost buys accuracy, not immunity |
| Smaller judge | A fraction of the tokens per verdict, fast enough to judge every case on every run | Example run: agreement close to the larger judge on easy criteria, visibly lower on contradiction detection (F2). Deploy per capability, where Chapter 6 says it holds up |
Selection is an empirical question with a protocol attached, not a default. Run both judges on the calibration set in Chapter 6, compare agreement per criterion, and assign each capability the cheapest judge that clears your agreement bar.
Versioning rule: changing the judge model or the judge prompt changes the measurement, exactly like changing a scorer in Chapter 4. Reports are only comparable within one judge configuration, and the manifest is what proves which configuration produced a report. A silent judge upgrade in the middle of a release comparison invalidates the comparison.
Same input, three runs: count the flips
Temperature 0 reduces sampling variance. It does not eliminate it, for the same provider side reasons Chapter 4 covered. So do not assume stability. Measure it: judge the same 100 cached outputs three times, with an identical prompt and model snapshot, and cross tabulate the verdicts.
| Outcome across 3 runs | Outputs | Bar |
|---|---|---|
| Pass, all three runs | 58 | |
| Fail, all three runs | 33 | |
| Stable verdicts, total | 91 | |
| Flipped at least once | 9 |
Arithmetic check: 58 plus 33 is 91 stable, plus 9 unstable, is the full 100. Self agreement in this example run is 91 percent. That number belongs in the run metadata, next to the judge scores it qualifies.
Flips are a routing signal, not noise
- Do not average the flips away. Majority voting over three runs hides exactly the cases where the rubric meets a genuinely borderline output.
- Route them to humans. The 9 unstable cases join the Chapter 6 review queue. They are the highest information cases you own, because the judge itself has marked them ambiguous.
- Read them for rubric defects. Flips cluster where a pass condition is underspecified. A cluster of flips on one criterion is a rewrite request for that criterion.
The stability budget
Report self agreement per capability, per run. If a capability's self agreement falls below the floor you set in Chapter 6, its judge scores for that run are flagged unstable in the report and cannot support a gate decision, however good they look.
Swap the order. Keep only the wins that survive.
The protocol
- Take 50 case pairs: candidate A and candidate B outputs for the same 50 golden cases.
- Judge each pair twice: once presented as A then B, once presented as B then A. Same prompt, same judge, only the order changes.
- Compare win rates by position. Any gap between the two orders is position bias, measured on your own suite with your own judge.
Nothing about candidate A changed between the two conditions except where it sat on the page. A 17 point swing means a single order pairwise run can manufacture a winner. Reported rates are the example run values for this experiment.
The counting rule this imposes: a pairwise win counts only when it survives the swap. Win in both orders: a win. Win in one order only: recorded as a tie, with the position sensitivity logged. Yes, this doubles pairwise judging cost. The alternative is publishing selection decisions that a coin flip on ordering could reverse.
Add words, add no information, watch the score
The protocol
- Start from passing answers: outputs that already pass the helpfulness rubric, so any movement is pure bias signal.
- Pad each one: append a sentence that restates a fact already in the answer, in different words. Information content is unchanged by construction.
- Rejudge with the identical rubric and compare pass rates, padded against original, case by case.
Original ends: "...refunds are issued to the original payment method."
Added: "In other words, the money goes back to the same payment method you used to pay."
New facts added: zero. New claims to verify: zero.
Example run: the pass rate rose 12 points for answers that say the same thing at greater length. The judge rewarded the appearance of thoroughness. Left uncorrected, this bias pays candidates to be verbose, and your token budget pays for it twice.
Two mitigations, both mandatory
- Rubric text: every quality criterion states explicitly that length is not evidence. Restating content adds no credit, and the rubric says so in the pass condition.
- Per run audit: compute the correlation between answer length and judge score on every run and print it in the report metadata. A rising correlation is a drift alarm, even when scores look healthy.
Judges grade their own family leniently
A judge model rates outputs in its own style more favorably: familiar phrasing reads as correct, and shared failure modes look reasonable to the system that shares them. This is a structural property of the setup, not a defect in one model.
Blind the identity
The judge prompt contains no model names, no provider names, no version strings. Outputs arrive labeled only as the candidate answer. Identity metadata lives in the manifest, where the judge cannot read it.
Cross the families
Use a judge from a different model family than the candidate, per the hard rule on slide 10. When the candidate changes family, recheck the pairing. It is a property of the pair, not of the judge alone.
Disclose the pairing
The manifest records candidate model, judge model, and the family_differs flag from slide 10. Any report whose pairing later turns out to be same family is flagged on sight in review, because its judge scores carry a known, directional bias.
How you would know it is happening on your suite
| Test | Procedure | Signal |
|---|---|---|
| Style swap test | Rewrite a set of outputs into the judge family's typical phrasing without changing content, rejudge both versions | A systematic score gap between content identical versions is family preference, quantified |
| Cross judge check | Judge the same outputs with a second judge from another family, compare per criterion pass rates | Criteria where the home judge passes its own family and the foreign judge does not are the leniency surface |
Both tests run on cached outputs, so they cost judge calls only. Chapter 6 folds the cross judge check into calibration, where human labels arbitrate which judge is right.
The text being graded can talk back
The payload rides inside a mostly correct answer, which is exactly why it is dangerous: the surrounding content deserves a pass, so a judge that follows the instruction produces a plausible looking verdict with a corrupted cause. In production traffic this text can arrive from a poisoned retrieved document, not only from the candidate model itself.
Four defenses, in layers
- Delimiters with strict parsing, from slide 5. The candidate block is extracted as data. Instruction like text inside it has no structural path to the judge's instructions.
- Instruction hierarchy stated in the prompt: the judge's system text declares that only the rubric defines passing, and that commands inside scored content are themselves scorable content, to be flagged, never obeyed.
- Strip or escape instruction like lines before judging: lines matching directive patterns are neutralized in a preprocessing pass, and the alteration is logged in the verdict metadata so the transformation is auditable.
- Adversarial judge cases in the golden set: injection payloads become permanent cases in Chapter 3's dataset, capability tagged, so every judge or prompt change is regression tested against the attacks you already know.
Success criterion for the worked example above: the verdict scores the refund and SSO claims on their evidence, flags the injected instruction as a finding, and the flag itself lands in failures.jsonl.
Check the judge with code before counting it
| Check | Rule, in code | What it catches |
|---|---|---|
| Evidence spans are real | span text is an exact substring of the cited context | A judge quoting evidence that does not exist. The single most dangerous judge failure, because it looks rigorous |
| Verdict is consistent | overall_pass == AND of all critical criteria | A verdict that fails a critical criterion and passes overall, or the reverse. Arithmetic the judge got wrong |
| Evidence is present | every passed criterion cites at least one evidence index | Verdicts asserted with no evidence at all. Rejected outright, however confident the rationale sounds |
The principle: everything about a verdict that can be checked deterministically is checked deterministically. The judge's irreducible job shrinks to the semantic residue: does this real span support this claim. Validation does not make the judge right. It makes a whole class of judge wrongness impossible to ship silently.
Count the tokens before you run the judge
| Component | Arithmetic | Total |
|---|---|---|
| Input tokens: prompt template, context, candidate, rubric | 500 x 1,850 | 925,000 |
| Output tokens: claim table, verdict JSON, rationale | 500 x 220 | 110,000 |
| Full pass total | 925,000 + 110,000 | 1,035,000, about 1.04M |
| Run cost | 0.925M x price_in + 0.110M x price_out (per 1M rates) | compute it per judge model, per run |
Note where the tokens sit: 89 percent of the spend is input, and most of that input is the retrieved context you already paid to fetch once. Judge cost is mostly re reading. That fact drives every lever below.
Lever 1 · Judge only what needs it
Faithfulness and helpfulness slices get a judge. Format and refusal slices are already settled deterministically. Judging all 500 cases when 260 need it is a 48 percent saving with zero information lost: 260 x 2,070 is about 0.54M tokens.
Lever 2 · Cache verdicts
An unchanged case, output, and judge configuration reuses its verdict. Slide 18 defines the key. In a typical prompt iteration, most outputs do not change, so most verdicts should not be repurchased.
Lever 3 · Downsize where proven
Where Chapter 6 shows the smaller judge agreeing with humans as well as the larger one, route that capability to the smaller judge. Savings here are earned by measurement, not assumed by price list.
A cache key that invalidates correctly
Why output text, not output ID
Two runs can share a case ID and a run label while containing different answer text after a regeneration. Hashing the text itself means the cache can never serve a verdict for words it never saw. Hashing identifiers alone is how stale verdicts survive a model change.
Execution discipline
| Control | Rule |
|---|---|
| Batching | Submit verdict requests in batches sized to the provider limit, with per case records preserved. A batch failure retries per case, not per batch |
| Concurrency cap | A fixed maximum of in flight judge calls, set in the run config. Uncapped parallelism converts a rate limit into a failed run |
| Backoff | Exponential backoff with jitter on rate limit and transient errors: wait, double, cap, and log every retry in the run metadata |
| judge_error budget | A run level ceiling, set in advance (for example, 2 percent of judged cases). Exceed it and the run is marked invalid for gating, however complete it looks |
Failure accounting: judge errors are a property of the run, reported next to the scores on slide 20. A run that quietly judged 470 of 500 cases and averaged the rest has not produced a slightly smaller result. It has produced a biased one, because parse failures correlate with unusual outputs.
Fifteen lines, same contract as Chapter 4
Line by line, what is doing the work
Compare with Chapter 4: the deterministic scorer differences are exactly two lines, call_judge and the error path. Everything else is the same machinery, which is why the judge composes with the rest of the suite instead of replacing it.
Beside the deterministic scores. Never merged into them.
| Capability | Scorer | n | Rate | Reading |
|---|---|---|---|---|
| Faithfulness | judge, F1 to F3 | 240 judged | 88.3% | Semantic support, claim by claim. The Chapter 4 suite has no row like this at all |
| Citation accuracy | deterministic | 500 | 96.8% | Citations parse, resolve, and match expected sources. Existence only, by design |
| Citation support | judge, F3 only | 240 judged | 91.7% | The semantic half of the row above. The 5.1 point gap between resolving and supporting is a real, fixable defect class |
| Helpfulness | judge, pointwise | 260 judged | 79.6% | Reported with the verbosity audit attached: length correlation printed in run metadata, slide 13 |
| Judge health | run metadata | 500 cases | judge_error 1.2% | 6 of 500 cases produced no valid verdict after retry: 6 divided by 500 is 1.2 percent, inside the 2 percent budget from slide 18 |
Why there is no combined row
Averaging a deterministic citation rate over 500 cases with a judge faithfulness rate over 240 judged cases yields a number whose denominator does not exist. It also lets a strong mechanical score subsidize a weak semantic one, which is precisely the failure shape Chapter 2 slide 4 warned about, one level down.
Diffing judge scores across runs
- Valid only within one judge model plus prompt version. The report header carries both, slide 10
- Read judge diffs per criterion first. An F2 drop with stable F1 is a contradiction problem, not a faithfulness problem
- Check judge_error and self agreement before interpreting any delta. Instrument drift mimics candidate drift, slide 11
Implemented and instrumented. Not yet allowed to gate.
What is true at the end of this chapter
- The judge runs: prompt, rubric, schema, validator, cache, and cost accounting are all in place and exercised on the full golden set
- Its failure modes are measured: variance on slide 11, position bias on slide 12, verbosity bias on slide 13, injection handling on slide 15
- Its outputs are auditable: every verdict carries evidence spans, a per criterion breakdown, and the judge configuration that produced it
What is not yet known
An uncalibrated judge is a hypothesis about human judgment.
Nothing in this chapter compared a single verdict against a human label. Self agreement proves the judge is consistent. Consistency is not correctness: an instrument can be stably, measurably, confidently wrong, and the bias experiments show three specific directions in which this one leans.
The trust rule, and what would change it
| Until Chapter 6 delivers | The judge may | The judge may not |
|---|---|---|
| Per criterion agreement with human labels, measured on hand labeled outputs, with every disagreement inspected | Rank candidates for human review, surface failures, fill report rows flagged provisional, and prioritize the review queue, flips first | Block or pass a release. No gate threshold is set on judge scores, in CI or anywhere else, until per capability trust is earned with data |
This is not caution for its own sake. A gate built on an uncalibrated judge trains the whole team to optimize for the judge, biases included. Calibration first is what makes the gate in Video 8 worth obeying.
The judge, in five lines
Key takeaways
- A judge is a scorer with a schema, a version, and a validator. If any of the three is missing, it is an oracle, and oracles do not ship.
- Binary rubrics, evidence first, verdict last. Decompose vague quality into falsifiable criteria, and force the claim table to exist before the verdict does.
- Biases are experiments you run, not risks you list. Swap the order, pad the answer, cross the family, inject the payload, and publish the numbers on your own suite.
- Cache and cost it like infrastructure. About 1.04M tokens per full pass in the example run, with a cache key that invalidates on any real change and an error budget that can invalidate a run.
- Provisional until calibrated. Implemented today, trusted only after Chapter 6 measures it against humans.
Next video · Chapter 6
Calibrating and validating the judge. Hand label 50 to 100 outputs before looking at any judge score, measure agreement per criterion, read every disagreement, and fix the rubric where the judge diverges. Where agreement holds, the judge earns automation. Where it does not, that capability stays human reviewed or deterministic.
Bring: the judge verdicts from today, the 9 flipped cases from slide 11, and your own labels, written before you peek.