Prompt Engineering for RAG: From Prototype to Production
Engineering decisions that decide whether a retrieval system stays trustworthy when real users, real corpora and real traffic arrive.
Senior Machine Learning Engineer, Adobe
Co-author, RAG Artificial Intelligence (De Gruyter)
Four pillars, one system
A failure in any one pillar looks like a failure in all of them. We will treat them as one pipeline, not four topics.
Prompt
Anatomy, context assembly, structured output, versioning.
Retrieval
Chunking, hybrid search, reranking, freshness.
Evaluation
Golden sets, metrics that separate cause from symptom.
Production
Latency, cost, guardrails, incidents.
Why prototypes break in production
Demo data is kind
Clean documents, short queries, one happy path. Production brings scanned PDFs, duplicates and questions nobody anticipated.
Success hides debt
A fluent answer can be wrong, stale or ungrounded, and still get a thumbs up. Fluency is not evidence.
Change is constant
Models, prompts, indexes and corpora all change at different speeds. Without versioning, you cannot explain a regression.
Rule for today: every design choice should answer, how will I know when this breaks?
One request, six decisions
Each stage is independently testable. If you cannot test a stage alone, you cannot debug it in production.
Prompt Engineering
In production, a prompt is not text. It is a versioned interface between retrieval, policy and the model, with a contract on both sides.
Anatomy of a production prompt
Fixed layers
- Role and scope: what the system answers, and what it refuses
- Grounding rule: answer only from supplied context, cite it
- Output contract: format, length, citation style
- Fallback: exactly what to say when context is insufficient
Variable layers
- Retrieved passages, ranked and labelled by source
- Few-shot examples, selected per query type
- User query, clearly delimited from instructions
- Session context, summarised, never raw and unbounded
Decision: keep fixed and variable layers separate in code. Mixing them is how prompt edits become untraceable.
Context assembly: what earns a token
Context is a budget, not a bucket. Every passage competes for tokens against instructions, examples and the answer itself.
More context past the point of coverage usually lowers accuracy. It dilutes attention and imports contradictions.
Patterns and anti-patterns
Patterns that hold
- Explicit insufficiency behaviour: say what is missing
- Source labels the model can cite verbatim
- One instruction per idea, ordered by priority
- Examples that show the refusal case, not only success
Anti-patterns that fail
- Politeness padding that consumes budget and adds no signal
- Contradictory instructions accumulated over months of edits
- Asking the model to both answer and grade itself in one pass
- Hidden instructions inside retrieved documents, treated as trusted
Structured output and schema enforcement
Ask for structure
Fields for answer, citations, confidence signal and a no-answer flag. Downstream code should never parse prose.
Validate, then repair
Schema validation on every response. One bounded repair retry, then a safe fallback, never an infinite loop.
Treat failure as data
A rising parse failure rate is an early warning of model, prompt or retrieval drift. Log it like an error, because it is one.
Prompt versioning and regression
prompts/ answer_v14.yaml # current, gated by eval answer_v13.yaml # previous, rollback target on every prompt change: 1. run golden set (retrieval + answer checks) 2. compare against v13, flag any slice that drops 3. ship to a small traffic slice first 4. keep model, index and prompt versions together in every log line
A prompt change without an eval run is a production experiment on your users.
Few-shot in production: selection and drift
Select, do not accumulate
Retrieve examples by query type and difficulty, the same way you retrieve passages. A fixed block of examples goes stale silently and biases answers toward old phrasing.
Watch for drift
- Examples drawn from a corpus that has since changed
- Examples that contradict a newer policy instruction
- Example answers in a format the schema no longer accepts
Retrieval
The model can only reason over what retrieval gives it. Most so-called hallucinations in RAG systems start here, one stage earlier than they appear.
Chunking trade-offs
Small chunks
Precise matching, low noise per passage.
Large chunks
Full context, but diluted relevance.
Structure aware
Split on headings, tables and code boundaries first, size second.
Decision: chunk for the question shape, not for a round number of tokens. Measure per document type.
Embeddings and hybrid search
Why hybrid wins
Users mix natural language with exact identifiers in the same query. Each method alone fails on half of that query.
Embedding choice
Evaluate on your corpus and your queries, not a public leaderboard. Domain vocabulary is where generic models quietly lose.
Reranking: the cheapest quality win
Retrieve wide for recall. Rerank narrow for precision.
Retrieve 50 to 100
Cheap first stage, optimised so the right passage is somewhere in the set.
Rerank to 5 to 8
A cross-encoder reads query and passage together. Slower per item, applied to few items.
Spend where it counts
Reranking usually buys more answer quality per unit of latency than a larger generator.
Query rewriting and routing
Rewrite for the index
- Expand abbreviations using corpus vocabulary
- Split compound questions into sub-queries
- Carry conversational context into a standalone query
Route for the task
- Factual lookup to the document index
- Fresh or personal data to a live source, not a stale index
- Out of scope queries to a refusal, early and cheaply
Guardrail: log the original and rewritten query side by side. Rewriting bugs are invisible otherwise.
Index freshness and corpus governance
Freshness is a feature
Decide per source: minutes, hours or daily. Show the as-of date in the answer when staleness matters.
Deletions must propagate
A removed document that stays in the index is a correctness and compliance bug, not a cleanup task.
One owner per source
Every corpus needs a named owner for quality, access control and retirement. Orphan corpora rot.
Retrieval failure modes
Miss
Right document exists, never retrieved. Fix vocabulary, chunking or recall depth.
Distraction
Topically similar, factually wrong passage ranks first. Fix reranking.
Staleness
Correct once, superseded now. Fix freshness and version preference.
Contradiction
Two sources disagree and both are supplied. Fix source priority and prompt behaviour.
Name the failure before you fix it. Each mode has a different owner and a different remedy.
Evaluation
If you cannot tell a good answer from a fluent one, you do not have a system yet. You have a demo with users.
Offline vs online evaluation
Offline: before release
- Golden set, run on every prompt, model or index change
- Deterministic checks first: citations present, schema valid
- Blocks a regression before users see it
Online: after release
- Real queries, real failure discovery
- Feedback, abandonment and follow-up question signals
- Finds the questions your golden set never imagined
Feed online failures back into the golden set. That loop is the evaluation system.
Retrieval metrics vs answer metrics
| Layer | Question it answers | Typical measures |
|---|---|---|
| Retrieval | Was the right evidence in the set, and ranked high? | Recall at k, ranking quality, coverage by slice |
| Grounding | Does the answer follow from the evidence supplied? | Faithfulness checks, citation accuracy |
| Answer | Is it correct, complete and usable for this user? | Correctness vs reference, task success, human review |
A single end-to-end score hides which layer failed. Keep the layers separate, always.
Golden sets: building one that survives
Start from real pain
Seed with real user questions, support tickets and known failures, not questions invented to be easy.
Cover the hard slices
No-answer cases, contradictory sources, stale documents, exact identifier lookups, multi-part questions.
Maintain it like code
Version it, review additions, retire cases when the corpus legitimately changes. A frozen set slowly becomes fiction.
LLM-as-judge: uses and limits
Good uses
- Scaling first-pass scoring across many answers
- Consistent rubric application: grounded, complete, clear
- Triaging which answers deserve human review
Hard limits
- Prefers fluent, longer and confident answers
- Shares blind spots with the model it judges
- Drifts when the judge model or prompt changes
Calibrate the judge against human labels on a regular cadence, and report agreement, not just scores.
Tracing a bad answer to its source
Full trace logging turns a vague complaint, the AI is wrong, into a fixable ticket in one specific stage.
Production Reliability
Users forgive a system that says it does not know. They do not forgive one that is slow, expensive and confidently wrong.
Latency and cost budgets
Budget the whole request, stage by stage, before optimising any single stage.
Cost follows context size and answer length. The cheapest token is the one you never send.
Caching, fallbacks and model routing
Cache at the right layer
Embeddings and retrieval results for repeated queries. Cache answers only where freshness and permissions allow.
Fallbacks with dignity
Smaller model, reduced context, or a clear no-answer with sources to read. Never a silent quality cliff.
Route by difficulty
Simple lookups to fast models, synthesis and conflict resolution to stronger ones. Measure the router like a model.
Guardrails and prompt injection
The threat is in the data
Retrieved documents are untrusted input. A passage that says ignore previous instructions is content to quote, never an instruction to follow.
Defence in layers
- Delimit and label retrieved content in the prompt
- Least privilege: tools and data scoped per request
- Output checks for leakage and policy before release
- Red team with malicious documents in the corpus, not only malicious users
Observability and incident patterns
Detect
Golden set canary, parse failures, no-answer rate and latency alerts fire first.
Triage
Pull full traces for failing queries. Which stage changed: model, prompt, index or corpus?
Mitigate
Roll back the changed version, pin the index, or route to the fallback path.
Learn
Every incident adds golden set cases. The postmortem is a test suite update.
Log query, rewritten query, retrieved IDs, prompt version, model version and scores for every request. Without that, incidents are folklore.
Build the loop, not just the demo
1. Grounding is a system property
No single prompt trick makes an answer trustworthy. Retrieval, prompting and evaluation do it together.
2. Version everything
Prompt, model, index and golden set, together, or you cannot explain your own system.
3. Fail honestly
A clear no-answer with sources beats a fluent guess, in trust and in incident counts.
Saloni Garg, Senior Machine Learning Engineer, Adobe
Co-author, RAG Artificial Intelligence, De Gruyter
linkedin.com/in/saloni-garg
Scan to pick your subject
saloni-garg.github.io/courses.html