Building the Golden Dataset
The dataset is the specification. Everything else is measurement. This chapter ends with a real file in the repo: versioned, linted, and ready for the scorers in Videos 4 to 6.
By the end, you can answer three questions
- Where do eval examples come from, and which sources can you trust?
- What does each example need to contain for a scorer to consume it?
- How do you know the dataset is good enough to gate a release?
How this lecture runs
- 20 to 25 minutes, 22 slides: the first build of the course
- Section 1: four sources, one pipeline
- Section 2: composition, stratification, and sample size
- Section 3: trusting the dataset itself: review, linting, versioning
- Same running example throughout: the Support RAG assistant
Last time: the architecture. Today: its input.
We established
- Four layers: deterministic checks, reference scoring, LLM judge, human review
- Capabilities, not models: retrieval, faithfulness, refusal, tool use, each scored separately
- Policies, not averages: per capability thresholds decide merge or block
Today we build the input to all of it
No dataset, no eval. Only anecdotes.
- Every layer from Chapter 2 consumes cases: Layer 2 is impossible without reference answers
- The gate from Chapter 2 is only as honest as the cases it runs on
- So the dataset gets built first, and built like a system, not collected like screenshots
Engineering question: where do eval examples come from, what must each one contain, and when is the set good enough to gate on?
The wrong way to build one
Three shortcuts that produce a dataset on day one and a false sense of safety from day two.
1 · Borrow a benchmark
Download a public benchmark and call it yours.
It tests someone else's product, someone else's documents, and failure modes your users will never hit, while missing the ones they hit daily.
2 · Generate and ship
Ask an LLM for 1,000 test questions.
Volume without review. The questions are clean, templated, and drawn from the same distribution the model already handles well. See slide 10.
3 · Save the demos
Save 20 demos that worked.
Survivorship as a dataset. Every case passed before it was saved, so the suite can only ever confirm what you already believed.
Bottom line: none of these represent your users, your docs, or your failures. A golden dataset is defined by whose behavior it specifies, and all three shortcuts specify someone else's, or no one's.
What a golden dataset actually is
Formal definition
A versioned, reviewed set of cases, where each case specifies input, context, expected behavior, and the capability it tests.
Versioned
Every change is a new version with a changelog. Scores are comparable only inside one version. Slide 20.
Reviewed
A human verified the expected behavior. Unreviewed cases are candidates, not gold. Slide 17.
Specified
Each case states what correct looks like before any model runs. No scoring against vibes.
Attributed
Each case names the capability it tests, so failures aggregate into slices, not anecdotes.
It is a specification of correct behavior, not a pile of questions. Questions are the easy part. The value lives in the expected behavior attached to each one, because that is what every scorer in Videos 4 to 6 measures against.
One case, field by field
Every field exists because a scorer consumes it
| Field | Why it exists | Consumed by |
|---|---|---|
| id | Stable identity for diffs, failure reports, and review history | All scorers · join key |
| query | The user input, verbatim, including typos and ambiguity | Runner · all layers |
| context | Pinned retrieved context when the case must be reproducible | Faithfulness judge · Video 5 |
| expected_answer | The reference answer for answerable cases | Reference scoring · Video 4 |
| expected_sources | Doc IDs that must support the answer; empty means refuse | Retrieval Recall@K · Video 4 |
| expected_behavior | answer, refuse, or clarify: the behavior being specified | Deterministic + judge · Videos 4 to 5 |
| must_contain | Facts every correct answer must state | Deterministic contains · Video 4 |
| must_not_contain | Claims that are failures if present, however fluent | Deterministic contains · Video 4 |
| capability | Which capability slice this case scores | Aggregation + report · Video 4 |
| difficulty | easy, medium, hard: where regressions hide first | Aggregation + report · Video 4 |
| source | Provenance: logs, coverage, synthetic, or failure archive | Audit + datasheet · this video |
The rule: if a field has no scorer, delete it. If a scorer needs a field you do not have, add it. The schema is a contract between this video and Videos 4 to 6, nothing more and nothing decorative.
Four sources, one pipeline
Production logs
What users actually asked, failures included. Slide 8.
Document coverage
What the corpus can answer, worked backwards. Slide 9.
Failure archive
What already broke in production, kept forever. Slide 11.
Synthetic / adversarial
What nobody asked yet but the system must survive. Slides 10 to 11.
No single source is sufficient alone. Logs miss what users never tried. Coverage misses how users actually phrase things. Synthetic misses reality. The failure archive only knows yesterday's bugs. The pipeline exists so each source covers the others' blind spots.
Production logs: sample reality, do not curate it
The method
- Sample real queries. Random or stratified sampling over a time window, not a scroll through highlights.
- Strip PII before anything else. Names, emails, account IDs come out before the file touches a repo or a reviewer outside the data boundary.
- Dedupe near identical queries. Fifty copies of one popular question are one case with a frequency note, not fifty cases.
- Label expected behavior in review. The log gives you the query. Gold status still requires a human verdict, slide 17.
Do not cherry pick successes.
Filtering logs to interactions that ended well rebuilds the demo problem at larger scale. A log sampled honestly will contain vague, angry, misspelled, and unanswerable queries. That is the point.
Keep the failures
They are the highest value cases you own. A logged failure is a query distribution, a real user intent, and a proven system weakness in one record. Failures sampled from logs feed the failure archive rule on slide 11.
Sampling rule of thumb: if you recognize every query in your sample, you did not sample. You browsed.
Document coverage: work backwards from the corpus
Logs tell you what was asked. Coverage tells you what could be asked. Every doc cluster gets questions, including the boring ones nobody demos.
How to build it
Cluster the corpus, list the question types each cluster should answer, and write cases into the empty cells first. The heatmap turns "we should cover more" into named, countable holes.
Read the zeros
The highlighted gaps are where regressions will ship silently: no cases means no score, and no score means the gate cannot see that capability at all.
Synthetic generation, with verification
LLMs generate volume. Humans grant gold status.
Generation is a drafting tool: propose questions, paraphrases, and edge variants at scale. Every synthetic case still passes the same review as a logged case before it enters the golden set.
Risk: too clean
Generated queries are grammatical, scoped, and polite. Real users are none of those reliably. A set of only clean questions overstates every score.
Risk: too templated
One prompt produces one shape of question with swapped nouns. Fifty variants of a template are closer to one case than fifty.
Risk: inherited blind spots
The generator will not invent questions about distinctions it does not know exist. Its gaps become your dataset's gaps.
The filter that keeps synthetic honest
- Run the generator's own model on its cases. A synthetic case that model cannot fail teaches you nothing. Keep the ones it gets wrong or shaky, review the rest harder.
- Mix, never replace. Synthetic fills coverage gaps and paraphrase diversity. It does not substitute for logged reality.
- Tag provenance. The source field records synthetic origin, so slice level audits can check whether synthetic cases behave differently.
Adversarial and failure cases: manufacture the hard part
Near miss documents
A doc that almost answers the question: right product, wrong plan tier. Retrieval finds it, and a careless model cites it.
Outdated policy versions
Last year's refund window sitting in the corpus next to this year's. The case checks which version wins.
Injection in a retrieved doc
Instructions planted inside a document the retriever returns. The expected behavior is to answer from policy and ignore the planted instruction.
Looks answerable, is not
Questions with the shape of a supported question and no supporting evidence. These are the refusal cases from slide 5, built on purpose.
The standing rule
Every production failure becomes a permanent case.
Not a ticket that closes, a case that stays. The failure archive is the one part of the dataset that grows by itself, and it guarantees a fixed bug can never silently return: the gate runs its case on every release, forever.
The 500-case build, stratified on purpose
Difficulty cross cut
| Difficulty | Cases | Share |
|---|---|---|
| Easy | 200 | 40% |
| Medium | 200 | 40% |
| Hard | 100 | 20% |
Every case type above is split across all three difficulties. Stratification is a grid, not two separate lists.
Why stratify at all
Refusals are 24 percent of this set by design, not by accident of sampling. Left to raw log frequencies, the set would be mostly easy factual questions, and the gate would be blind exactly where Chapter 1's regressions lived.
How many examples do you actually need?
A score is a proportion, and proportions carry uncertainty. For a capability score near the middle of the range, the 95 percent confidence band shrinks slowly with n.
Band width scales with 1 over the square root of n. Quadrupling the slice only halves the band, which is why slice size is a budgeting decision, not a detail.
The punchline
A capability slice with 50 cases cannot detect a 5 point regression. Its noise band is wider than the regression you are trying to catch, so the gate either fires on noise or sleeps through real drops.
The working rule
- Start at 150 to 200 total cases to build against
- Gate only on slices with enough cases to see the regression you care about
- Reach about 500 for release gating, with the gated slices sized first
Expected behavior, not expected strings
Reference answers are specs, not scripts
| Spec field | What it pins down |
|---|---|
| must_contain | The facts a correct answer must state, in any wording |
| must_cite | The sources a correct answer must ground itself in |
| must_not_contain | Claims that fail the case outright |
| Acceptable variation | Length, ordering, and phrasing the spec deliberately leaves free |
One case, two correct answers
Two correct answers in different words must both pass. The moment a scorer demands one exact string, paraphrase becomes failure, teams start writing to the test string, and the score stops describing quality. Exact strings belong to slide 6 fields with deterministic scorers, and nowhere else.
Building refusal cases deliberately
Answerable questions occur naturally in logs. Should refuse questions mostly do not, because users rarely ask them on purpose and demos never include them. You have to manufacture the negatives.
No evidence questions
Well formed questions about topics the corpus simply does not cover. Expected behavior: refuse, and say what is missing.
Near miss docs
Pair the question with a doc that is close enough to tempt an answer and wrong enough to forbid one. Refusal 041 on slide 5 is this pattern.
Expired docs
The only evidence is a superseded policy version. Answering from it is a faithfulness failure wearing a citation.
Out of scope requests
Asks outside the product entirely: personal advice, other companies' policies, general chat. 40 of the 500 cases on slide 12.
If your dataset has no should refuse cases, your refusal score from Chapter 2 does not exist. There is nothing to score: every case expects an answer, so a model that answers everything, including the unanswerable, posts a perfect record right up to the incident.
Difficulty is a design variable
Same capability, three tiers. Capability: citation correctness on refund policy questions. Watch what changes across the tiers: not the topic, the trap.
Easy
Single doc, direct fact.
"What is the refund window for monthly plans?"
One doc holds the answer verbatim. Failure here means something basic broke: retrieval, reading, or formatting.
Hard
Two docs, conflicting versions.
"What is the refund window for annual plans?"
The 2025 and 2026 policies disagree. The case checks the model cites the current one and does not blend the two.
Adversarial
Right answer present, wrong answer prominent.
"What is the refund window for annual plans?"
Same question, but the outdated doc ranks first and is quoted in a banner. The prominent answer is the wrong one, by construction.
Why tiers matter: regressions show up in the hard and adversarial slices first, while the easy slice stays green. Report difficulty slices separately, the same way Chapter 2 reported capabilities separately, or the average hides the early warning.
Who writes, who verifies, what happens on disagreement
The labeling guide is an artifact
A written guide in the repo: how to spec a refusal, what counts as a citation, worked examples. If the rule lives only in one person's head, the dataset drifts with whoever labels that week.
Second reviewer, selectively
Refusal and safety cases get a second reviewer by default, because a wrong spec there teaches the gate to enforce the wrong behavior. Routine factual cases get one verifier.
Keep a disagreement log
Every disagreement is recorded with its resolution. Disagreements usually mean an ambiguous case, not a bad labeler. An ambiguous case gets rewritten until two reviewers read it the same way, or it leaves the set.
Datasets have bugs. Run checks on them like code.
What the lint catches
| Check | Failure it prevents |
|---|---|
| Schema validation | A case a scorer silently skips |
| Expected source exists | Scoring against a doc that was deleted or renamed |
| Duplicate queries | One question counted five times in a slice |
| Capability present | Cases that vanish from every slice in the report |
| Stale doc references | A correct model failing on a reference that moved |
The principle: a broken dataset produces confident, wrong scores, and nobody suspects the ruler. Linting is cheap, runs on every dataset PR, and keeps measurement bugs out of release decisions.
Protect the set from the system it measures
Never train or tune on it
The golden set measures generalization to user questions. Fitting to it converts measurement into memorization, and the score stops predicting production.
Never paste it into the prompt
Few shot examples drawn from the golden set leak the test into the candidate. Prompt examples come from the dev set below, never from the gating set.
Never let builders hand pick cases
Whoever ships the candidate does not choose what it is graded on. Case selection follows slides 7 to 16, not release convenience.
THE SPLIT THAT ENFORCES IT
Prompt work, debugging, few shot examples. Scores here are diagnostics, not gates.
The golden set from this chapter. Changes only through versioned review, never mid release.
Sealed cases swapped in when the frozen set gets memorized by process: quoted in docs, tickets, and postmortems.
Version the dataset like you version code
| Version | Change | Cases |
|---|---|---|
| golden_v1.2 | Added 34 failure archive cases, retired 6 cases pointing at deleted docs, reasons recorded per case | 500 |
| golden_v1.1 | Added adversarial slice, rebalanced refusal cases after the Chapter 1 regression review | 472 |
| golden_v1.0 | First frozen release set: logs, coverage, and reviewed synthetic | 410 |
Added and retired cases always carry a reason. A case retired because the product changed is maintenance. A case retired because models keep failing it is a red flag, and review should treat it as one.
Scores are comparable only within a version
A faithfulness score on golden_v1.1 and the same score on golden_v1.2 are measurements on different instruments. Reports always print the dataset version next to the score, and baseline diffs never cross versions.
The diagnostic rule
A score jump after a dataset edit is a dataset change until proven otherwise.
Check the changelog before celebrating the model. Most sudden improvements in mature suites are edits, retirements, or leakage, and versioning is what lets you prove which one.
A real file, in the repo
Inside DATASHEET.md
- Sources: how many cases came from logs, coverage, synthetic generation, and the failure archive, and how each was reviewed
- Composition: the stratification table from slide 12 and the difficulty cross cut, as shipped
- Known gaps: the heatmap zeros from slide 9 that remain open, named so nobody mistakes absence of cases for absence of risk
- Rules: contamination policy from slide 19 and the versioning policy from slide 20
The datasheet is what lets a new engineer, or a reviewer six months later, judge whether a score from this dataset deserves trust.
The dataset, in five lines
Key takeaways
- The dataset is the specification. Scorers only measure against it.
- Four sources, reviewed into one set. Synthetic alone is not gold.
- Stratify by capability and difficulty, or averages hide the holes.
- Size the slices to the regression you need to detect.
- Version, lint, and protect it from contamination.
Next video · Chapter 4
Video 4 runs this file. Deterministic and reference based scoring: the Layer 1 and Layer 2 scorers from Chapter 2, consuming the schema from slide 6, producing the first baseline report the rest of the course diffs against.
Bring: golden_v1.jsonl, linted and frozen. Next time it stops being a document and starts being a measurement.