Lectures and course slides
Slide decks from invited lectures and from the LLM Evals From Zero course. Each deck is a self-contained HTML presentation; use the arrow keys to move through the slides.
New courses are being planned. Leave your email and pick the subject you want to learn, and you will be notified when enrollment opens for that course. Responses are sent directly to Saloni.
Your email is only used to notify you about the course you pick. No spam, no sharing. If the form does not load above, open it in a new page.
Want to see what a course looks like first? The current lecture slides are open:
Virginia Tech distinguished lecture, October 30, 2026. A 45-minute technical lecture for graduate students, researchers, and practitioners, on the engineering decisions that take a retrieval-augmented generation system from prototype to production. Four pillars: prompt engineering, retrieval, evaluation, and production reliability. 30 slides.
A nine-chapter course on evaluating LLM applications: why demos mislead, how to design an evaluation suite and build the golden dataset it runs on, deterministic and reference-based scoring, LLM-as-a-judge and its calibration, evaluation for RAG pipelines and agents, release gates in CI, and online evaluation and drift in production. 196 slides.
Why "looks good" is not a metric. A support RAG system under test, a prompt change that appears to improve quality, and why manual spot-checks pass while the system is broken.
What an eval suite actually looks like, and how to design one: evaluation is multi-dimensional, and a single accuracy number can stay flat while quality moves.
The input the suite runs on: what a golden dataset actually is, the wrong way to build one, and one case examined field by field.
Deterministic and reference-based scoring, implemented: the eval runner, the scorer interface, exact match and token F1, retrieval metrics, and paired comparison against a frozen baseline.
A scorer made of a model: judge prompt and rubric design, structured verdicts, evidence-first claim checking, and bias experiments for position, verbosity, and self-preference.
Calibrating the LLM judge against blind human labels: confusion matrices, Cohen's kappa, false passes, and per-capability trust decisions (gate, warn only, or human always).
Stage-wise evaluation for RAG pipelines and multi-step agents: retrieval metrics and failure attribution, trajectory scoring, fault injection, and repeated-run success rates.
Evaluation in CI that blocks bad releases: policy as code in gate.yaml, thresholds set from measured baseline noise, a two-tier pipeline, and baseline management.
Online evaluation, drift, and maintenance: production sampling, shadow and canary evaluation, provider fingerprinting, and the eval health dashboard.