Scoring Without an LLM
Deterministic and reference-based evaluation, implemented. Layers 1 and 2 from Chapter 2, run against the golden dataset from Chapter 3. No judge in this chapter: every score here is computed by code you can read, test, and rerun for free.
By the end, you can answer four questions
- What can be scored exactly, with no model in the loop?
- What does each metric actually compute: formula, inputs, failure mode?
- How do per case scores become a report you can diff?
- How do you know the scorers themselves are correct?
How this lecture runs
- 20 to 25 minutes, 22 slides: the first scoring code of the course
- Section 1: Layer 1, deterministic checks, slides 5 to 11
- Section 2: Layer 2, reference-based scoring, slides 12 to 17
- Section 3: the baseline run, paired diffs, and trusting the scorers
- Every metric slide shows its formula and a worked numeric example
One command in, two artifacts out
The two artifacts that matter
- report.json: aggregated scores per capability, with n per slice. The diffable summary, slide 20.
- failures.jsonl: one record per failed case, with the scorer name and a reason string. The debugging surface, slide 20.
Where we are in the course
- Chapter 2: the architecture exists. Four layers, capabilities, policies.
- Chapter 3: the dataset exists. golden_v1.jsonl, 500 cases, linted and frozen.
- Today: we run it. Generation, scoring, aggregation, and the first baseline.
The scope line for this chapter
Layers 1 and 2 answer questions with objective or reference based answers. Faithfulness and helpfulness are not in scope yet, because no string comparison can measure them. That boundary is the whole reason Video 5 exists, and slide 8 draws it precisely.
Seven components, one cache boundary
The key design decision
Separate generation from scoring. Cache everything in between.
Generation is the expensive, slow, nondeterministic step. Scoring is cheap, fast, and deterministic. So when you fix a scorer bug, and you will, you rescore the cached outputs instead of paying to regenerate 500 model calls. A rescore takes seconds and produces a report that differs from the old one only where the scorer changed. Without the cache, every scorer fix costs a full regeneration, and worse, the new outputs are not the same outputs, so you can no longer attribute the score change to the scorer.
Pin everything, record everything
What gets pinned, and why
| Field | Why it is pinned |
|---|---|
| model snapshot | A floating model alias can change under you mid comparison |
| prompt hash | The prompt is code. The hash proves which text produced this run |
| temperature 0 | Minimizes sampling variance between runs |
| max tokens | Truncation is a behavior. It must not drift silently |
| retrieval k, index | Retrieval changes are generation changes. Index version is part of the system |
Temperature 0 reduces variance. It does not guarantee identical outputs. Provider side batching, mixture of experts routing, and floating point nondeterminism can still flip individual generations at temperature 0. That is exactly why the manifest and the raw output cache exist: you do not assume reproducibility, you record what actually ran, so any report can be traced back to the exact outputs that produced it.
One contract for every scorer
What the contract buys you
- Composable: the runner treats every scorer identically. Adding a scorer is adding one function and one registry entry.
- Unit testable: a pure function of two dicts can be tested with fixtures, no mocking, no API keys. Slide 21.
- Attributable: value, passed, and reason are recorded per case per scorer, so every aggregate in the report can be drilled back to individual decisions.
- No scorer calls the model. The moment a scorer calls an LLM it stops being deterministic, stops being free, and belongs to Layer 3 in Video 5, with a different contract.
Is the output even the right shape?
The cheapest failures to catch are structural: the answer cannot be parsed, ignores the requested format, or blows the token budget. Check shape before you check content.
| Check | Method | Failure it catches |
|---|---|---|
| JSON validity | json.loads, then jsonschema validate | Prose wrapped around JSON, trailing commas, missing required fields, wrong types |
| Parse rate | parsed cases / total cases | Format regressions at the suite level. Example run: 493 / 500 = 98.6% |
| Exact bullet count | count lines matching the bullet marker | Instruction drift: asked for exactly 3 bullets, got 5 and a paragraph |
| Token budget | tokenizer count vs. case budget | Runaway verbosity that passes eyeball review and doubles cost |
| Language check | script and language detection on the text | Answers in the wrong language after a prompt or model change |
jsonschema: use it when structure is a contract
Validates types, required fields, enums, nesting, and additional properties, and returns which constraint failed as the reason string. If downstream code consumes the output, this is the check.
Regex: use it only for surface patterns
Fast for spotting a marker or a pattern, but it cannot express types or nesting, and it fails silently on the malformed inputs you most need to catch. A regex that says the JSON looks fine is how broken payloads reach production.
Contains checks, done right
must_contain and must_not_contain from the Chapter 3 schema are substring checks, and raw substring matching is wrong often enough to matter. Normalize both sides first, identically.
| Model wrote | Raw substring | After normalization | What normalization cannot fix |
|---|---|---|---|
| "30 days" | MATCH | MATCH | The easy case. It is also the rarest in the wild |
| "30 Days." | MISS | MATCH | Case and punctuation. Fixed by the pipeline above |
| "thirty days" | MISS | MISS | Number words. Needs explicit numeral canonicalization (thirty to 30) as a tested pipeline step, or the case belongs to reference scoring |
| "a month" | MISS | MISS | A paraphrase, not a formatting variant. No string pipeline fixes this. That is a semantic judgment, Video 5 |
Citations: three checks, one hard boundary
1 · It parses
Extract citation markers from the output with a strict pattern. A malformed marker is a format failure, scored here, not a mystery.
2 · It resolves
Every cited doc ID must exist in the index at the pinned index version. A citation to a deleted or invented doc is a hard fail.
3 · It is expected
Compare the cited set against expected_sources from the case. This is a set operation, fully deterministic.
Per case citation precision and recall
| Worked example | Sets | Arithmetic | Result |
|---|---|---|---|
| Case cites 3 docs, case expects 2, both expected docs are cited, plus 1 extra | cited = {refunds, billing, sso} expected = {refunds, billing} | precision = 2 / 3 recall = 2 / 2 | precision 0.667 recall 1.000 |
Read it as: the answer grounded itself in everything required (recall 1.000) but padded with one unneeded source (precision 0.667). Two different fixes, two different numbers.
The boundary: existence is deterministic. Whether the cited doc actually supports the claim is a semantic judgment, and it is a judge task in Video 5. Teams blur this line constantly, then wonder why their deterministic citation score says 98 percent while users keep finding unsupported claims.
Tool calls are data. Validate them like data.
For agents, the most consequential output is often not the prose, it is the tool call. Every part of its validity is checkable without a model.
| Check | Method |
|---|---|
| Tool name in allowlist | Set membership against the tools registered for this agent. An unregistered name is a hallucinated capability |
| Arguments validate | jsonschema against the tool's declared argument schema: required fields present, enums respected |
| Argument types correct | Schema types, strictly. "30" the string is not 30 the integer, and coercing it silently hides model errors |
| No invented parameters | Schema declares additionalProperties: false. An extra key like force fails validation instead of reaching the tool |
Why this matters more than prose quality: a wrong tool call executes. It refunds the wrong order or queries the wrong account, in production, at machine speed. Validity here is not pedantry, it is the last deterministic checkpoint before a side effect. Whether the agent picked the right tool for the task is a separate question: that is trajectory scoring in Video 7.
Record the run, not just the answers
The generator records latency, tokens, and errors per case into the raw output cache. These are deterministic measurements of the run itself, and they belong in the same report as quality.
| Metric | How it is computed | Example run |
|---|---|---|
| Latency p50 | Median per case wall clock, generation only | 1.2 s |
| Latency p95 | The value 95 percent of cases finish under | 2.6 s |
| Latency mean | Arithmetic mean, recorded but never reported alone | 1.4 s |
| Tokens per case | Mean input and output tokens from the API response | 812 in · 214 out |
| Cost per case | (812 × price_in + 214 × price_out) / 1,000,000 | $0.0043 |
| Timeout rate | timed out cases / total = 3 / 500 | 0.6% |
| Retry rate | cases needing at least 1 retry / total = 12 / 500 | 2.4% |
Why p50 and p95, not only the mean
The mean of 1.4 s hides the tail: one case in twenty waits past 2.6 s, and users remember the tail. A prompt change that adds a reasoning paragraph often leaves the mean nearly flat while p95 climbs. Percentiles are where operational regressions actually live.
Timeouts and retries are scores too
A timed out case produced no answer, so quality scorers skip it, and a suite that only reports quality will call a slow, flaky candidate an improvement. The report carries timeout and retry rates next to the quality numbers, and the gate in Video 8 can block on them.
Refusal is a classifier. Score it like one.
Each case carries expected_behavior from Chapter 3. The scorer detects the output's behavior from deterministic markers (refusal phrasing, absence of an answer body, zero citations on an answer expected case) and compares. Example baseline run, all 500 cases:
Confusion matrix · example baseline run · n = 500
The two error cells are different incidents. 14 answered anyway is a grounding and safety problem. 19 false refusals is a usefulness problem. One number would merge them.
The three rates, with arithmetic
Of the cases that required a refusal, how often the system answered anyway. The high risk direction.
Of the cases with supporting evidence, how often the system refused anyway. The silent usefulness leak.
Behavior matched the spec, either direction. Report it only alongside the two rates above.
Note the denominators: each rate is computed on its own population. Missed refusals over all 500 cases would read 2.8 percent and look harmless. Rates are only meaningful against the cases that could have produced them.
Exact match: the right tool, in a narrow band
Normalization is the slide 7 pipeline, applied to both sides before comparing. Exact match on raw strings measures formatting luck, not correctness.
Where exact match is correct
- Classifications: the predicted label either is the expected label or it is not. Refusal detection on slide 11 is this shape.
- IDs and codes: order IDs, SKUs, error codes, doc IDs. There is exactly one right string.
- Short factual answers: "30 days", a version number, a date. The answer space is effectively closed.
The counterexample
| Answer | Correct? | EM |
|---|---|---|
| "You can request a refund within 30 days of purchase." | YES | 1 |
| "Refunds are available for 30 days after checkout." | YES | 0 |
| "File your refund request in the first 30 days." | YES | 0 |
Reference is the first row. All three answers are correct, two score zero. On prose, exact match mostly measures how the reference author happened to phrase it, and a suite built on it punishes every correct paraphrase.
Token F1, worked end to end
| Side | Tokens after normalization | Count |
|---|---|---|
| Reference | you · can · request · a · refund · within · 30 · days · of · purchase | 10 |
| Prediction | you · can · request · a · refund · during · 30 · days · after · any · purchase · online | 12 |
| Overlap | you · can · request · a · refund · 30 · days · purchase (multiset intersection: within, of missing on one side; during, after, any, online extra on the other) | 8 |
Step 1 · Precision
8 / 12 = 0.667
Of the tokens the model produced, two thirds appear in the reference. The rest is added wording.
Step 2 · Recall
8 / 10 = 0.800
Of the tokens the reference needed, 80 percent are present. Nothing factual was dropped except connective wording.
Step 3 · Harmonic mean
2 × 0.667 × 0.800 / (0.667 + 0.800)
= 1.067 / 1.467 = 0.727
The harmonic mean punishes imbalance: a scorer gaming precision by answering in three words cannot hide the recall collapse.
What F1 does not know: the missing tokens here are harmless connectives, and F1 cannot tell them apart from a missing "not". Token overlap is a similarity signal with partial credit, useful for tracking drift across versions of the same system. It is not a correctness verdict: that distinction is why F1 sits in Layer 2 as a diagnostic, and why the gate in Video 8 never blocks on F1 alone.
BLEU and ROUGE, honestly
Both are n-gram overlap metrics, built for a specific job in a specific era. Using them well starts with knowing what that job was.
BLEU · built for machine translation
- pn: precision of n-grams up to 4 against reference translations
- BP: brevity penalty, punishes outputs shorter than the reference
- Designed as a corpus metric: meaningful averaged over thousands of sentences, unstable and noisy on single answers
ROUGE · built for summarization
- Recall oriented: how much of the reference summary's wording the candidate summary kept
- Assumes reference wording is the target, which is defensible for extractive style summaries and wrong for open answers
- ROUGE-L swaps n-grams for longest common subsequence, same assumption underneath
| Use | Verdict | Why |
|---|---|---|
| RAG answer quality | WEAK · DO NOT GATE | Penalizes correct paraphrase, rewards fluent copying of a wrong reference, and has no concept of whether a claim is supported by the retrieved context |
| Fixed style generation | ACCEPTABLE SIGNAL | Templated summaries, formatted reports, style locked outputs: the reference wording genuinely is close to the target, so overlap tracks something real |
| Release gate, alone | NEVER | No threshold on an overlap metric distinguishes a factual regression from a phrasing change. Slide 12's counterexample scores near zero on both metrics while being fully correct |
Embedding similarity: calibrate the threshold, fear the near miss
The computation
- Embed prediction and reference with the same pinned embedding model, record its version in the manifest
- Score is the cosine: 1.0 is identical direction, and real paraphrase pairs typically land well below it
- Above threshold counts as a match. The threshold is the scorer, everything else is arithmetic
Calibrate, do not guess
- Label a dev set of pairs by hand: match or no match, a few hundred pairs, including adversarial near misses
- Sweep the threshold across the score range and read the false pass and false fail rates at each candidate value
- Pick the operating point by the error you can afford. For a gating scorer, false passes are the expensive direction
- Freeze it in the scorer version. Changing the threshold or the embedding model is a scorer change, slide 21
| Pair (reference vs. prediction) | Cosine | Similarity verdict at 0.85 | Reality |
|---|---|---|---|
| "A refund is available within 30 days" vs. "A refund is not available within 30 days" | 0.93 | MATCH | Opposite claim. One token flips the meaning, embeddings barely move |
| "Refund window is 30 days" vs. "Refund window is 60 days" | 0.97 | MATCH | Wrong number. The single most important token in the answer is the one that changed |
| "Enterprise plans include SSO" vs. "Team plans include SSO" | 0.95 | MATCH | Wrong entity. Plan names live close together in embedding space, far apart in the contract |
Embeddings encode topical similarity, and these pairs are maximally on topic. Use similarity as a soft signal and a triage tool. Never let it be the only thing standing between a negated or renumbered answer and a release.
Score retrieval before generation
expected_sources in each case turns retrieval into a scored component with its own ground truth. If the right chunk never arrives, no prompt can fix the answer, so this failure gets measured at its source.
| Metric | Formula | What it answers |
|---|---|---|
| Recall@k | relevant retrieved in top k / all relevant | Did the evidence the answer needs show up at all? |
| Precision@k | relevant retrieved in top k / k | How much of the context window is signal vs. noise? |
| MRR | mean over queries of 1 / rank of first relevant | How high does the first useful doc rank, on average? |
| Hit Rate@k | queries with at least 1 relevant in top k / all queries | For what fraction of questions did retrieval give generation a chance? |
Worked example · one case, k = 5
| Rank | Retrieved doc | Relevant? | Arithmetic |
|---|---|---|---|
| 1 | docs/billing/refunds.md | YES | Relevant set for this case: 3 docs. Top 5 contains 2 of them, at ranks 1 and 4. Recall@5 = 2 / 3 = 0.667 Precision@5 = 2 / 5 = 0.400 Reciprocal rank = 1 / 1 = 1.000 Hit = 1 (at least one relevant present) |
| 2 | docs/billing/invoices.md | NO | |
| 3 | docs/account/profile.md | NO | |
| 4 | docs/billing/refund-policy-2026.md | YES | |
| 5 | docs/support/faq.md | NO |
The third relevant doc never entered the top 5. If the answer then misses its content, that failure is booked to retrieval, with Recall@5 as the evidence, not to the model. Two failure classes, two owners, zero mixing.
Micro and macro: report both, or be misled by either
| Capability slice | n | Passed | Rate | Bar |
|---|---|---|---|---|
| Factual | 180 | 171 | 95.0% | |
| Reasoning | 70 | 56 | 80.0% | |
| Refusal (both directions combined) | 120 | 108 | 90.0% | |
| Ambiguous / clarify | 30 | 18 | 60.0% | |
| Instruction format | 50 | 47 | 94.0% | |
| Adversarial | 50 | 35 | 70.0% |
Micro average · every case equal
Total passes over total cases. This is the honest overall number, and it is dominated by the biggest slice: factual cases alone are 36 percent of the set.
Macro average · every slice equal
Mean of the six slice rates. It surfaces what micro hides: the ambiguous slice is collapsing at 60 percent. If that slice fell to zero, micro would move by only 18 cases in 500, about 3.6 points, while users who ask ambiguous questions would get a broken product.
The reporting rule: every aggregate ships with its n, and every slice ships with its own rate. A score without an n is a claim without evidence, and Chapter 3's sample size math is what decides whether a slice is even allowed to gate.
Freeze the baseline. Everything after this is a diff.
What gets frozen
- The system: current production prompt and model snapshot, run once through the full pipeline from slides 3 to 17
- report.json: the per capability scores, micro and macro, with n per slice
- raw_outputs.jsonl: all 500 outputs verbatim, so any baseline number can be re-derived or rescored
- manifest.json: the pinned config from slide 4 that produced it
These artifacts are saved permanently, in the repo, next to the dataset version they ran against. A baseline that lives in a chat message or a dashboard screenshot is not a baseline.
The rule this creates
Every future number in this course is a diff against this baseline.
Not "the candidate scored 86.4 percent", but "the candidate scored 86.4 percent against a baseline of 89.6 percent, on the same 500 cases, with these 40 cases changing verdict". Absolute scores describe a run. Diffs make release decisions.
Re-baselining is a decision, not a habit
When production legitimately changes, you may promote a new run to baseline, in a reviewed change that records why: what shipped, what the old baseline measured, and what comparability you are giving up. Re-baselining quietly after a bad week is how suites learn to always pass.
Same 500 cases, both runs: read the flips, not just the delta
Because baseline and candidate ran the identical cases, every case is a pair. Cross-tabulating the pairs shows movement the headline delta erases. Example run:
Per case flips · example run · n = 500
The 28 regression case IDs are the review list. They land in failures.jsonl with reasons, and a human reads those before anyone argues about the release.
The arithmetic, and what it hides
This asymmetry is also the statistical core of paired tests like McNemar's: only the discordant pairs carry information about which system is better, and their imbalance is the evidence.
Artifacts you debug from, not dashboards you glance at
If a failure has no reason string, the scorer is not finished. The reason is what turns a failed case into an assignable bug. A developer handed refusal-041 and that reason can reproduce and fix the failure without rerunning anything or guessing what the scorer meant.
What the pair of artifacts gives you
- report.json is the diff surface: machine readable, comparable across runs, ready for the CI gate in Video 8
- failures.jsonl is the work queue: filterable by capability and scorer, every row actionable
- Together they answer both questions a release review asks: how much changed, and exactly where
The scorers are code. Test them like code.
Unit fixtures per scorer
| Fixture | What it pins down |
|---|---|
| Known pass output | A hand written output that must score passed, with the expected value and reason shape |
| Known fail output | A hand written output that must fail, including the exact violation it demonstrates |
| Boundary outputs | The edge the scorer is most likely to get wrong: the citation that almost resolves, the bullet list with a stray line, the refusal phrased as an apology with an answer attached |
Fixtures are cheap because of the slide 5 contract: score(case, output) on two literal dicts, no model, no network. A scorer without fixtures is an opinion with an API.
Scorer changes are versioned like model changes
- Every report carries scorer_version, slide 20. Diffs are only valid within one scorer version, exactly like dataset versions in Chapter 3
- A scorer fix triggers a rescore of cached outputs, slide 3, so its effect is measured in isolation
The trap this prevents
Example: a unicode normalization fix in the contains scorer moves the factual pass rate up 1.8 points, example run, with zero model change. Reported naively, that reads as a model improvement. Reported correctly, the diff attributes it to scorers 1.3.0 to 1.3.1, model and prompt untouched. Measurement changes and system changes must never share a headline number.
Scoring without an LLM, in five lines
Key takeaways
- Separate generation from scoring. Cache raw outputs, rescore freely, never pay twice for the same generation.
- Deterministic first. It is free, fast, and reproducible, and it catches more than most teams expect: structure, citations, refusals, tool calls, cost.
- Every metric has a formula and a failure mode. Know both before you put a number in a report.
- Aggregate per capability, micro and macro with n, and compare paired against a frozen baseline.
- Test the scorers and version them. A score is only as trustworthy as the code that produced it.
Next video · Chapter 5
LLM-as-a-judge, implemented. What Layers 1 and 2 cannot measure is precisely defined now: faithfulness to the retrieved context, whether a citation actually supports its claim, helpfulness, completeness. Video 5 builds the judge for exactly those questions, and Video 6 calibrates it against human labels before it is allowed to gate anything.
Bring: report.json, failures.jsonl, and the frozen baseline. Next time the suite learns to judge, carefully.