Chapter 2 · Architecture
Anatomy of an LLM Evaluation Suite
Designing an evaluation system that can actually block bad releases. No implementation yet: this chapter is the blueprint that Videos 3 to 8 build, layer by layer.
By the end, you can answer one question
"If I have an LLM app, what exactly should my eval suite look like?"
- Name the four scoring layers and what each one is for
- Score capabilities, not models: retrieval, faithfulness, refusal, tool use
- Turn a report into a release decision with an explicit policy
How this lecture runs
- 20 to 25 minutes, 22 slides: architecture first, code starts in Video 3
- Section 1: the four layers that score any output
- Section 2: capability-based evaluation: what gets scored
- Section 3: the full pipeline, the report, and the release gate
- Same running example throughout: the Support RAG assistant from Chapter 1
Audience: engineers with a working LLM app
Scope: architecture, not implementation
Output: a blueprint for your own suite
SG
Saloni GargSenior Machine Learning Engineer, Adobe
SLIDE 01 / 22PACE 0:00
Slide 02 · Recap
Last time: the problem. Today: the design.
We learned
- Demos lie. A hand-picked sample is filtered by the outcome before it is shown.
- Manual testing does not scale. Five spot checks cannot see a failure mode that fires on a small slice of real traffic.
- Every release needs automated evaluation. A prompt change that warmed up the tone silently cost faithfulness, citations, and refusals.
Today we answer
What should an evaluation suite actually contain?
- The layers that score an output, cheapest and most objective first
- The capabilities each output is scored on
- The policy that turns scores into a merge or block decision
Engineering question: if I have an LLM app, what exactly should my eval suite look like? By slide 22, you will be able to draw it.
SLIDE 02 / 22PACE 1:05
Slide 03 · The wrong mental model
Most teams think an eval suite looks like this
Dataset
A folder of test prompts
Model
Run the candidate over the prompts
Accuracy = 87%
One number out the bottom
This tells you almost nothing.
- Which cases failed? The number does not say, so nobody can fix them.
- Which capability regressed? Retrieval, faithfulness, and refusal are averaged into invisibility.
- What is it averaged over? Easy and adversarial cases count the same.
- What decision does it drive? There is no threshold and no owner, so the release ships on a feeling anyway.
A single score is a summary with the evidence deleted. The next slide shows what that costs.
SLIDE 03 / 22PACE 2:10
Slide 04 · Why one score fails
Suppose accuracy stays the same
Baseline vs. candidate, scored per capability. The headline number does not move. Everything a user would notice does.
| Metric | Baseline | Candidate | Reading |
|---|---|---|---|
| Accuracy | 87% | 87% | FLAT |
| Hallucinations | 2% | 11% | ↑ 9 pts, worse |
| Refusal | 96% | 73% | ↓ 23 pts |
| Citation correctness | 99% | 81% | ↓ 18 pts |
The question
Would you deploy this model?
The accuracy row says nothing changed. The other three rows describe a system that invents facts five times as often, refuses far worse, and cites sources it did not use.
Why the average hides it
Capabilities move in opposite directions and cancel out in the mean. Gains on easy cases pay for regressions on hard ones, and the single number reports the trade as "no change." A release decision needs the rows, not the average of the rows.
SLIDE 04 / 22PACE 3:15
Slide 05 · The dimensions
Evaluation is multi-dimensional
"Quality" is not one thing. It is a set of capabilities and costs, each measured separately, because each one fails separately.
Retrieval
Did the system fetch the evidence the answer needs?
Faithfulness
Is every claim supported by the retrieved context?
Refusal
Does it decline what it should decline, and only that?
Tool Use
For agents: right tool, right arguments, right order?
Safety
Does it stay inside the lines on harmful or out-of-scope asks?
Cost
What does one run of the system cost, in tokens and retries?
Latency
How long does the user wait, at the tail as well as the median?
Format
Does the output obey the contract: structure, length, citations?
Every capability can regress independently. Slide 4 is the proof: accuracy flat while hallucinations, refusal, and citation correctness all moved. A suite that reports one number per capability can see that. A suite that reports one number cannot.
SLIDE 05 / 22PACE 4:20
SECTION 1
The Four Layers
SLIDES 06 TO 10 · HOW AN OUTPUT GETS SCORED
Slide 06 · Architecture overview
Every response passes through four layers
Responses
Outputs of the system under test, one per dataset case
L1
Deterministic Checks
Code decides: format, schema, budgets
DECIDES
L2
Reference Scoring
Compare the output against ground truth
DECIDES
L3
LLM Judge
A model scores what only judgment can settle
JUDGES
L4
Human Review
People settle what the other layers cannot
DECIDE
Why this order
- Cheapest and most objective first. A rule that runs in microseconds should never wait behind a judge call.
- Each layer handles what the layer above cannot decide. Format is a fact. Faithfulness is a judgment. Ambiguity is a human call.
- Volume shrinks as cost rises. Everything passes through Layer 1. Only the residue reaches human review.
- Layers are independent. Swap the judge model or add a check without rebuilding the suite.
Design rule: never pay for judgment where a fact will do.
SLIDE 06 / 22PACE 5:25
Slide 07 · Layer 1 of 4
Deterministic checks: questions with objective answers
checks/deterministic.py · plain asserts, no model call
def check(out):
assert json.loads(out.text) # JSON valid?
assert valid_schema(out.tool_call) # tool call valid?
assert out.citations # citation exists?
assert out.tokens < 500 # under 500 tokens?
assert out.latency_s < 3.0 # under 3 s?Properties
- Free and instant: runs on every case, every commit
- Perfectly reproducible: same input, same verdict, forever
- Binary: pass or fail, with the failing case ID attached
What belongs here
| Question | Decided by |
|---|---|
| JSON valid? | Parse it |
| Tool call valid? | Schema check |
| Citation exists? | Lookup in retrieved set |
| Response under 500 tokens? | Token count |
| Latency under 3 s? | Timer |
These should never use an LLM.
Asking a model whether JSON is valid buys you cost, latency, and nondeterminism in exchange for a worse answer than a parser gives for free. If a rule can decide it, a rule decides it.
SLIDE 07 / 22PACE 6:30
Slide 08 · Layer 2 of 4
Reference-based metrics: compare against ground truth
When the dataset carries the right answer, scoring is arithmetic: measure the distance between the output and the reference.
| Metric | What it measures | Use it when |
|---|---|---|
| Exact match | Output equals the reference, character for character | Answers are short and canonical: labels, codes, IDs |
| F1 | Token-level overlap: precision and recall of the answer's words | Extractive QA, short factual answers |
| BLEU SOMETIMES | N-gram precision against the reference | Wording is tightly constrained, translation-like tasks |
| ROUGE SOMETIMES | N-gram recall against the reference | Summarization-style outputs with a fixed target |
| Semantic similarity | Embedding distance between output and reference | Paraphrase is fine but meaning must match |
| Retrieval Recall@K | Did the expected chunk appear in the top K retrieved? | Scoring the retriever before generation even starts |
The requirement
Requires reference answers. No ground truth, no Layer 2. This is why the golden dataset in Video 3 stores an expected answer and expected sources for every case.
The limit
Overlap is not correctness. A wrong answer can share most of its words with the right one, and a right answer can share almost none. Layer 2 is strong where wording is constrained, weak where answers are open-ended.
SLIDE 08 / 22PACE 7:35
Slide 09 · Layer 3 of 4
LLM-as-a-judge: questions humans answer well
What the judge scores
- Is the answer faithful? Every claim traceable to the retrieved context
- Is it helpful? It addresses what the user actually asked
- Does it follow instructions? Constraints honored, not just attempted
- Is reasoning complete? No skipped steps, no unsupported leaps
Each question gets a rubric: defined criteria the judge applies case by case, so the verdict comes with a reason a human can audit.
Why this layer exists
Cannot be reliably measured with string matching. Two answers can share zero words and both be right. A fluent answer can match every keyword in the reference and still be wrong. Meaning is the thing being scored, and meaning needs a reader.
What this chapter does not do
The judge is one component of the suite, not the suite. Videos 5 and 6 build it properly: writing rubrics it can follow, exposing its biases, and calibrating it against human labels before any gate is allowed to trust it.
Engineering question: how do I trust an LLM judge? Short answer for now: you do not, until it agrees with humans on cases humans labeled first. That is Video 6.
SLIDE 09 / 22PACE 8:40
Slide 10 · Layer 4 of 4
Human review: the scarcest layer, spent deliberately
Ambiguous outputs
The rubric has no clean answer and the judge is guessing. A human settles what the case should have scored.
Safety
The cost of a wrong automated verdict is highest here. Safety cases get human eyes before a release ships.
New capabilities
No rubric exists yet for behavior the system just learned. Humans define what good looks like first.
Judge disagreements
Where judge runs conflict, or the judge conflicts with a reference score, the case escalates instead of averaging out.
Humans are expensive. Use them strategically.
Human review is the slowest and most expensive layer in the stack, and the only one that does not scale with compute. The architecture protects it: Layers 1 to 3 absorb the volume so humans see only the cases where their judgment changes the outcome.
Human labels do double duty
Every human verdict is also training signal for the suite itself: it becomes a golden dataset case, a rubric clarification, or a calibration point for the judge. Review is not a cost center bolted on at the end; it is how the cheaper layers get trustworthy.
SLIDE 10 / 22PACE 9:45
SECTION 2
Capability-Based Evaluation
SLIDES 11 TO 17 · WHAT GETS SCORED
Slide 11 · The unit of scoring
Don't score models. Score capabilities.
Instead of
GPT-5 = 91%
- No owner: when it drops, nobody knows which part of the system to fix
- No diagnosis: the number moves and the cause stays hidden
- No policy: there is no threshold you can defend for a number that means everything at once
Think
Retrieval
Faithfulness
Refusal
Tool Use
Safety
Latency
One score per capability, each with its own scorer, its own baseline, and its own gate. The next six slides take the core capabilities one at a time.
A model score tells you something changed. A capability score tells you what changed, and who fixes it.
SLIDE 11 / 22PACE 10:50
Slide 12 · Capability 1 of 6
Retrieval: score it before generation exists
Query
One dataset case
→
Retriever
Returns top-k chunks
→
Check vs. Expected Sources
Was the gold chunk in the top k?
→
Recall@K
Layer 2, deterministic scoring
The questions
- Did we retrieve the right chunk?
- Was supporting evidence available in the retrieved set at all?
- Was recall sufficient, or did the answer need evidence that never arrived?
Generation hasn't happened yet.
Retrieval is scored on the retriever's output alone, before the model writes a word. If the evidence never arrived, the failure belongs to retrieval, and no prompt edit will fix it. Scoring the stages separately is what makes a failure attributable.
SLIDE 12 / 22PACE 11:55
Slide 13 · Capability 2 of 6
Faithfulness: grounded, or invented?
The question
Did the model invent anything not present in retrieved context?
Not "is the answer true in the world?" but "is it true to the evidence the system retrieved?" A correct-sounding claim with no support in context is still a faithfulness failure, because the next one will be wrong the same way.
POSSIBLE OUTPUTS
Faithful
Every claim in the answer is supported by the retrieved context.
Partially faithful
The core answer is supported, but details were added that the context never states.
Hallucinated
Claims contradict the context, or rest on evidence that was never retrieved.
Scored by: the LLM judge (Layer 3), because support is a judgment about meaning. String matching cannot tell a paraphrase from an invention.
SLIDE 13 / 22PACE 13:00
Slide 14 · Capability 3 of 6
Refusal: a two-way decision, scored both ways
Should answer?
YES: answer is in scope and in the docs
Respond
Answer, grounded and cited
Should answer?
NO: out of scope, or no evidence retrieved
Refuse
Decline clearly, do not guess
Measure both error directions
- False refusals: the system refused a question it should have answered. Users hit a wall for no reason.
- Missed refusals: the system answered a question it should have refused. This is where hallucination and safety failures enter.
Why one number is not enough
The two errors trade off. Tune the system to never refuse and missed refusals explode; tune it to refuse freely and false refusals do. The dataset labels every case must_refuse: true / false, and the report shows both directions separately.
SLIDE 14 / 22PACE 14:05
Slide 15 · Capability 4 of 6
Instruction following: the layer is chosen by the instruction
| Example instruction | How it is checked | Layer |
|---|---|---|
| Exactly three bullets | Count the bullets | DETERMINISTIC |
| JSON only | Parse the output, reject prose | DETERMINISTIC |
| Markdown table | Check the structure renders as a table | DETERMINISTIC |
| Use citations | Citations present (Layer 1), and they support the claims (Layer 3) | BOTH |
| No opinions | Judgment call about tone and stance | JUDGED |
Often deterministic. Sometimes judged. Count what can be counted, parse what can be parsed, and reserve the judge for instructions about meaning and tone. Routing each instruction to the cheapest layer that can decide it is the whole architecture in miniature.
SLIDE 15 / 22PACE 15:10
Slide 16 · Capability 5 of 6
Tool use: for agents, score the trajectory
Correct tool
Did the agent call the tool the task actually needed, not a plausible neighbor?
Check: deterministic, against the expected call.
Correct arguments
Right values, right types, nothing invented to fill a required field.
Check: schema plus value comparison.
Correct order
Steps sequenced so each call has what it needs from the ones before it.
Check: trajectory comparison.
Recovery after failure
A tool errored or returned junk: did the agent adapt, retry sensibly, or escalate?
Check: judged, plus step counts.
Success ≠ final answer only. An agent can land on the right answer through the wrong calls, a lucky guess, or a side effect you would never accept in production. If you score only the outcome, all of that ships. Video 7 extends the full framework to agents.
SLIDE 16 / 22PACE 16:15
Slide 17 · Capability 6 of 6
Operational metrics: don't forget the cost of quality
| Metric | What it catches |
|---|---|
| Latency | A candidate that answers better but slower, past the point users tolerate |
| Token cost | Prompt and reasoning growth that multiplies spend at production volume |
| Retries | Hidden instability: the system quietly paying twice for one answer |
| Throughput | Whether the candidate still holds up at real request volume |
| Timeout rate | Requests that never produce an answer at all, invisible to quality scores |
Quality isn't free. Operational metrics belong in the same report as quality metrics, because the release decision weighs both. A slightly better answer at much higher latency and cost can be the wrong release, and slide 20 turns exactly that trade into policy.
SLIDE 17 / 22PACE 17:20
SECTION 3
Putting Everything Together
SLIDES 18 TO 22 · PIPELINE, REPORT, GATE
Slide 18 · Complete evaluation pipeline
The layers and capabilities, assembled
Golden Dataset
Fixed, versioned cases with expected behavior
Candidate Model
The exact build under test, pinned
Generate Outputs
One response per case, recorded with latency and cost
Scoring · Layers 1 to 3 run in order
Deterministicformat, schema, budgets
Referencevs. ground truth
LLM Judgefaithfulness, quality
Aggregate Report
One score per capability, diffed against the baseline
Release Gate
Policy decides: merge, or block the release
Human review (Layer 4) sits beside the pipeline: escalations, safety cases, and disagreements route to people, and their verdicts flow back into the golden dataset.
SLIDE 18 / 22PACE 18:25
Slide 19 · Example report
What the report actually looks like
EXAMPLE EVALUATION REPORT · SCORED PER CAPABILITY
Retrieval Recall93%
Faithfulness95%
Citation Accuracy98%
Refusal88%
Instruction Following97%
Latency1.8 s
No single overall score.
There is deliberately no blended number at the bottom. The moment you average these rows, slide 4 happens: regressions hide inside the mean.
How to read it
Each row is a capability with its own baseline and its own threshold. Refusal at 88% is the weakest row here, and it is visible at a glance. The report is the input to the release decision, which is the next slide.
SLIDE 19 / 22PACE 19:30
Slide 20 · Release decision
Release decisions are policies, not averages
An example gate policy, written before any candidate is run. The candidate is compared against the baseline, per capability, and the policy decides.
EXAMPLE POLICY · FAIL THE RELEASE IF ANY ROW TRIPS
| Capability | Fail condition | Why this threshold shape |
|---|---|---|
| Faithfulness | ↓ more than 2% | Invented facts are core product risk, so the band is tight |
| Refusal | ↓ more than 5% | Noisier to measure, wider band, still gated |
| Safety | ↓ any amount | No tolerance band: any regression blocks |
| Latency | ↑ more than 25% | Operational regressions gate too, not just quality ones |
Engineering teams deploy policies, not averages. Thresholds are agreed in advance, per capability, so when the report lands the decision is mechanical: the gate fires or it does not. Tune the numbers to your application and its noise; the structure, per-capability thresholds with an explicit fail action, is the part to copy.
SLIDE 20 / 22PACE 20:35
Slide 21 · The road ahead
What we'll build, in dependency order
✓ Golden dataset
The input every layer consumes
Deterministic metrics
Layer 1, implemented first because it is free
Reference scoring
Layer 2, against the dataset's ground truth
LLM judge
Layer 3, built and then calibrated against humans
CI pipeline
The suite runs on every pull request
Automatic release gates
Policy from slide 20, enforced by the pipeline
Which video builds which piece
| Video | Builds |
|---|---|
| 3 | The golden dataset |
| 4 | Deterministic and reference-based scoring |
| 5 to 6 | The LLM judge: implementation, then validation against humans |
| 7 | The framework extended to RAG and agents |
| 8 | The architecture becomes a CI/CD release gate |
One coherent engineering project, not nine independent topics: each video fills in one part of the blueprint from this chapter.
SLIDE 21 / 22PACE 21:40
Slide 22 · Key takeaways
The blueprint, in five lines
Key takeaways
- Evaluation is multi-dimensional.
- Different capabilities require different metrics.
- Use deterministic checks whenever possible.
- LLM judges are only one component.
- Release decisions should be based on capability regressions, not a single aggregate score.
Next video · Chapter 3
Build the Golden Dataset. Every layer in this architecture consumes it, and Layer 2 is impossible without its reference answers, so it comes first: what a case contains, how to stratify it, and the rule that the dataset never gets edited to make a failing model pass.
Bring: the questions your users actually ask, including the ones your app should refuse.
SG
Saloni GargSenior Machine Learning Engineer, Adobe
SLIDE 22 / 22PACE 22:45 · END 24:00