It usually starts well. You build a retrieval-augmented assistant, try it on five questions before the demo, and it answers them beautifully. Two weeks later it is confidently citing the wrong policy document — and when you tweak one prompt to fix it, three other answers quietly get worse. How would you even know?
That gap — between “it felt fine when I tried it” and “I can prove it works” — is the entire reason evaluations exist. This guide covers what evals actually are, the two very different things people mean when they say “evaluate an LLM,” and how to build a real eval pipeline for a RAG system.
The problem: vibes don’t scale
Eyeballing a handful of outputs is fine for the first day of a prototype. It falls apart the moment you need to answer questions like:
- Is prompt B actually better than prompt A, or does it just feel better?
- Did upgrading the model improve quality, or did it quietly regress on edge cases?
- My retriever changed — are answers still grounded in the right documents?
LLM applications are non-deterministic: the same input can produce a great answer today and a hallucination tomorrow. You cannot manage what you cannot measure, and you cannot measure with vibes. The mental model that makes it click: evals are the unit tests and regression tests for non-deterministic systems.
What an evaluation actually is
An evaluation is a repeatable measurement of output quality against a defined expectation, producing a score you can track over time. Every eval, however fancy, is built from three ingredients:
- Something to test on — a dataset of inputs, and ideally expected outputs.
- Something to judge with — a metric or scorer.
- Something you get back — a score, plus a pass/fail verdict.
Two very different things people call “evaluation”
When someone says “I want to evaluate an LLM,” they could mean one of two completely different activities. Conflating them is the single most common mistake in this space.
Evaluating the model
Here the question is “which model is better?” and the tooling is mostly off-the-shelf.
- Benchmarks test raw capability: MMLU (broad knowledge), GSM8K and MATH (math), HumanEval and MBPP (code), HellaSwag and ARC (reasoning), TruthfulQA (honesty), IFEval (instruction following).
- Leaderboards aggregate these into rankings: LMArena (Elo from human pairwise votes), the HuggingFace Open LLM Leaderboard, HELM, and Artificial Analysis.
- Reference-based metrics compare output against a known-good reference. N-gram overlap: BLEU (precision-oriented, built for translation), ROUGE (recall-oriented — ROUGE-N counts n-gram overlap, ROUGE-L uses the longest common subsequence; built for summarization), and METEOR (synonym- and stem-aware). Model-based: BERTScore, which compares contextual embeddings instead of exact tokens, so paraphrases score well. And perplexity — the exponential of the average negative log-likelihood; it measures how “surprised” a model is by held-out text (lower is better), useful during pretraining and fine-tuning.
A word of honesty: ROUGE and BLEU correlate poorly with human judgment on open-ended generation — they shine for summarization and translation, and are weak for chatbots. Perplexity needs token-level probabilities (so it works for open-weight models, not closed APIs), is not comparable across different tokenizers, and tells you nothing about whether the output is actually useful.
Evaluating the application
Here the question is “is my RAG or agent doing its job?” No public benchmark knows your documents, your users, or your task. This requires custom development: your own dataset, task-specific metrics, and usually an LLM acting as a judge. This is where the real work lives.
The three components of any eval
Goldens (the test cases). A golden is one row of your dataset: an input (the query), an expected output (the ground-truth answer), and optionally the reference context. It is a template before you run your system; it becomes a full test case once you attach the actual output your application produced.
The evaluator (the metric). A scoring function plus a pass threshold. Evaluators come in two families — deterministic ones (ROUGE, exact match) and LLM-as-judge ones (faithfulness, answer relevancy, custom rubrics). The judge-based scorers have a superpower: they return a reason for the score, which turns a failing number into an actionable debugging clue.
The result. A score (usually 0 to 1), a pass/fail verdict against the threshold, the reason, and an aggregate across the whole dataset that you can re-run on every change.
Evaluating a RAG system
A quick recap of retrieval-augmented generation: you embed the user’s query, retrieve the most relevant chunks from a vector store, stuff those chunks into the prompt as context, and let the LLM generate a grounded answer. The key insight for evaluation is that RAG has two distinct failure surfaces — retrieval can fail (you fetched the wrong chunks) or generation can fail (you had the right chunks but the model ignored or contradicted them).
The RAG triad
The cleanest mental model for which metric catches which failure is the RAG triad. There are three artifacts — the query, the retrieved context, and the generated answer — and each metric measures one relationship between them.
- Context relevancy — are the retrieved chunks actually about the query? (query ↔ context)
- Context precision — are the relevant chunks ranked at the top instead of buried under noise? (retrieved context vs. the expected answer)
- Faithfulness — is every claim in the answer grounded in the retrieved context, with no hallucination? (context ↔ answer)
- Answer relevancy — does the answer actually address what was asked? (query ↔ answer)
The diagnostic power is in the combinations. If faithfulness fails but answer relevancy passes, the model wrote a fluent answer that is not supported by your documents — a hallucination. If context relevancy fails, your retriever is the problem, not the LLM. You stop guessing and start fixing the right component.
What goes in a RAG golden dataset
For each golden you want the question, the ideal expected answer (your ground truth), and optionally the ideal reference context. A good dataset is not just happy-path questions — deliberately include edge cases, adversarial phrasings, and “the answer isn’t in the docs, so the system should say I don’t know” cases. Those refusal cases catch the most embarrassing production failures.
Wiring it together: the eval pipeline
Here is the crucial idea — evals don’t change your RAG, they observe it. Same retriever, same generator, same prompt. You swap the live user query for a curated golden, capture what the pipeline already produces, and add a measurement layer on the side.
Walking the flow: the golden supplies the query and the expected answer. The query goes into your live RAG, which produces the retrieved context and the actual answer — these only exist after you run the system. You assemble all four fields into a test case, hand it to the evaluator (the triad metrics plus an LLM judge), and get back per-question, per-metric scores with reasons. Because this layer is bolted on the side, you can drop an eval harness onto an existing RAG without touching the inference code — and re-run the whole suite on every prompt tweak, model swap, or retriever change.
Before and after evals
Before: shipping on vibes and a handful of manual checks; no objective way to compare two prompts or two models; regressions slip through silently; “it works on my examples” is the quality bar.
After: an objective, numeric quality signal; data-driven model and prompt selection; regression tests wired into CI so a bad change fails the build; the confidence to ship, with hallucinations caught before users ever see them.
The takeaway
Evaluation turns “I think it got better” into “I can prove it got better.” Start small: write ten golden questions for your application, pick the two or three metrics that map to your actual failure modes, and wire up a pipeline you can re-run. The first time an eval catches a regression you would have shipped, it pays for itself. If you are building RAG specifically, anchor everything on the triad — it tells you not just that something is wrong, but where. At Orthonity, this evaluation-first discipline is built into everything we ship, so the AI we build for you keeps earning its trust in production.
