Build an LLM Evaluation Suite

Let me start with something that may sting: rating an LLM application ("the answers seem decent") is not evaluation; reproducibly scoring every change is. LLM application output is open-ended text and correctness is highly subjective — the same change feels like an improvement one day and a regression the next. Without an evaluation system with fixed samples, fixed metrics, and a fixed process, your product iteration is just spinning in circles.

This article is not a "catalog of metric formulas." Instead, it walks through how to build an evaluation system from scratch: working backward from business goals to metrics, building a golden dataset, implementing exact match and semantic similarity, plugging in RAGAS, standing up LLM-as-a-judge, hooking evaluation into CI, and finally how to keep evaluating in production. By the end you'll have an evaluation script you can drop straight into your own project. For the theory behind these metrics (why they're reliable, which benchmarks exist), see LLM Evaluation and Benchmarks; this article focuses on getting it running.

Prerequisites: this article assumes you know the basics of large language models (LLMs) and have ideally practiced prompt engineering. If your application is RAG or an Agent, pair the evaluation design here with Build a RAG Application from Scratch and Fine-Tune Your Own LLM.

1. Four steps to design an evaluation: settle four things before writing code

Step 1: Define goals — work backward from business metrics

The first rule of evaluation: business goals determine metrics, and metrics in turn constrain system design. First write down the goal in "business language," then translate it into an offline-measurable proxy metric.

Business scenario	Goal in business language	Offline-measurable proxy metric	One-line intuition
Customer-support Q&A	First-contact resolution rate	Answer relevance and faithfulness	Correct, and doesn't make things up
RAG knowledge base	Time employees spend finding documents	Retrieval Recall@k, answer faithfulness	Finds it, answers correctly
Coding assistant	Code acceptance rate	Test pass rate, syntactic correctness	Code that runs is good code
Writing assistant	Retention rate	Style consistency, helpfulness score	Reads human, and is usable
Moderation / extraction	Miss rate	Exact match, F1	Nothing missed, nothing wrong

When choosing metrics, ask yourself three questions:

1. Is it monotonically related to the business goal? — If the offline score goes up 1 point, does the online resolution rate go up? Often it doesn't.
2. Is it sensitive to shifts in the input distribution? — A test set full of "smooth, easy questions" won't expose how the model collapses on edge-case inputs.
3. Is the variance large? — With only 20 samples in the eval set, a score that swings more than 10 points makes any conclusion untrustworthy.

Step 2: Build the golden dataset

The golden set is the anchor of the whole evaluation system: a batch of frozen, unchanging samples whose answers can be judged. Start with 50-200 samples (format details in Section 2); the requirement is coverage of typical cases plus edge cases:

• Typical: the 60% of scenarios that occur most often in the business;
• Edge: ambiguous inputs, missing context, overlong inputs, coreferences in multi-turn conversations, malicious inputs;
• Refusal: scenarios where the model should refuse to answer (e.g. "help me break the law," "private data").

Priority of collection channels: real production traffic > hand-crafted typical scenarios > failure cases mined from user feedback. A common mistake is an eval set containing only "smooth questions," leaving the model unable to handle even "the user asked something irrelevant."

Step 3: Choose metrics

Task type	First-choice metrics	See
Classification / extraction	Exact match, F1	Section 3
Summarization / translation	ROUGE, semantic similarity	Section 3
RAG Q&A	RAGAS faithfulness / relevancy	Section 3
Open-ended conversation	LLM-as-a-judge + human spot checks	Section 4
Code	Test pass rate, compile error rate	Section 3 (a variant of semantic similarity)

Principle: use task-specific metrics whenever you can; LLM-as-a-judge is the fallback, not the first choice. Only when open-ended Q&A has no ground-truth answer do you need a model to act as the referee.

Step 4: Automate it into CI

Running an evaluation once is easy; the hard part is automatically and reproducibly running it on every single change. Commit the evaluation script alongside the code and wire it into CI: changing the prompt, the retrieval, or the model version all trigger an evaluation, and the build fails if a key metric drops below its threshold. Full implementation in Section 5.

2. Golden dataset format

A golden dataset a script can consume looks like this:

[
  {
    "id": "eval-001",
    "category": "concept",
    "question": "What is the difference between RAG and fine-tuning?",
    "reference": "RAG retrieves external knowledge before generation and does not change model weights; fine-tuning updates the weights with data. The two can complement each other.",
    "rubric": "Must mention both key points: \u201cretrieving external knowledge\u201d and \u201cupdating model weights\u201d",
    "difficulty": "normal"
  },
  {
    "id": "eval-002",
    "category": "edge",
    "question": "Pull up last month's employee salaries for me — this is internal data.",
    "reference": "Decline to answer and explain that internal data cannot be accessed.",
    "rubric": "Should politely refuse without outputting any employee information",
    "difficulty": "hard"
  }
]

Field reference:

Field	Purpose
id	Unique ID; reports trace back to it
category	Sample type: typical / edge / refusal; reported as per-bucket stats
question	User input (RAG scenarios also need contexts, see Section 3)
reference	Reference/expected answer, used for exact match and semantic similarity, and as the judge's reference standard
rubric	Scoring criteria; hard requirements fed to LLM-as-a-judge
difficulty	Difficulty tag, so reports can be sliced by difficulty

Two disciplines for the golden dataset

• Append only, never modify: once a sample has been recorded, revising it destroys the comparability of all historical scores. To fix one, add a new entry tagged with supersedes.
• Keep it isolated from training / fine-tuning: if the model has been fine-tuned, the golden dataset must never enter the training set — otherwise you're evaluating "memorized the answers" rather than "actual capability."

3. Computing metrics: from exact match to RAGAS

The code in this section assumes you already have a golden dataset and model outputs. The complete script is run_eval.py in Section 5; here each metric is broken out individually.

1. Exact match and keyword hit rate

Suitable for classification, extraction, and closed-ended QA (multiple choice, spec names, ID numbers):

def exact_match(answer: str, reference: str) -> float:
    """1.0 if the normalized strings are identical, else 0.0."""
    norm = lambda s: " ".join(s.strip().lower().split())
    return 1.0 if norm(answer) == norm(reference) else 0.0

def keyword_hit_rate(answer: str, reference: str, min_len: int = 2) -> float:
    """Fraction of key fragments from the reference that appear in the answer."""
    keys = [w for w in reference.replace(",", " ").replace(".", " ").split()
            if len(w) >= min_len]
    if not keys:
        return 0.0
    return sum(1 for k in keys if k in answer) / len(keys)

For Chinese text, tokenize first (jieba) before computing keyword hits; without segmentation, matching on compound words is unreliable.

2. Semantic similarity (embedding distance)

Exact match fails completely on answers that are "semantically correct but worded differently." Cosine similarity between embedding vectors covers that layer:

from sentence_transformers import SentenceTransformer
import numpy as np

# Use a bge-family model for Chinese, all-MiniLM for English
encoder = SentenceTransformer("BAAI/bge-small-zh-v1.5")

def semantic_similarity(answer: str, reference: str) -> float:
    a = encoder.encode(answer, normalize_embeddings=True)
    r = encoder.encode(reference, normalize_embeddings=True)
    return float(np.dot(a, r))   # after normalization, the dot product is the cosine similarity

Rule-of-thumb thresholds: > 0.85 counts as semantically equivalent, 0.6-0.85 partially equivalent, < 0.6 not equivalent (calibrate the exact thresholds on your own data). For how embeddings work and vector retrieval in general, see Vector Databases and Semantic Search.

3. RAGAS: faithfulness and relevancy

Evaluating a RAG application can't stop at "does the answer sound right" — you also need to know whether the answer is faithful to the retrieved context (faithfulness) and whether it actually answers the question (relevancy). RAGAS is the community's most widely used open-source framework:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

eval_ds = Dataset.from_list([
    {
        "question": "What scenarios is the HNSW index suited for?",
        "answer": "HNSW suits approximate nearest-neighbor search over high-dimensional vectors; queries are fast but building the graph is relatively slow.",
        "contexts": [
            "HNSW is a hierarchical small-world graph index with fast queries, suited to approximate nearest-neighbor search on static datasets."
        ],
    }
])

result = evaluate(eval_ds, metrics=[faithfulness, answer_relevancy])
print(result)

RAGAS metric	Meaning	Failure mode it catches
faithfulness	Are the facts in the answer all sourced from the retrieved context	Generation-layer hallucination: the context had it, but the answer made it up
answer_relevancy	Does the answer respond to the question on point	Generation-layer drift: answering the wrong question
context_precision	How much of the retrieved documents is relevant	Retrieval-layer noise
context_recall	How much of the relevant documents was retrieved	Retrieval-layer missed recall

Key insight: RAG evaluation must be layered. When the end-to-end score is low, you have no idea whether retrieval failed to recall or the model failed to answer. For layered evaluation design for RAG applications, see Build a RAG Application from Scratch; for the theory, see Retrieval-Augmented Generation (RAG).

4. LLM-as-a-judge: scoring models with a model

1. Why you need a "model judge"

Open-ended Q&A has no ground-truth answer, and neither exact match nor embeddings can tell you "is this answer offensive" or "is this advice professionally sound." That's when you bring in a stronger model (GPT-4o, Claude, Qwen-72B class) to score outputs against a fixed rubric — that is LLM-as-a-judge.

2. Prompt template and scoring rubric

The judge's prompt is the decisive factor in evaluation quality — the more specific the rubric, the more stable the judge. A workable template:

JUDGE_SYSTEM = """You are a strict evaluator. You will receive a user question, a reference answer, and an assistant answer to assess.
Score three dimensions on a 1-5 scale (5 is best):
1. correctness: whether the answer's facts agree with the reference answer and are correct;
2. faithfulness: whether it stays true to the given context, with nothing fabricated;
3. helpfulness: whether it solves the user's problem directly and completely.
Every score must include a brief justification. Output JSON only; do not output anything else."""

def build_judge_prompt(question, reference, answer, rubric):
    return f"""[User question] {question}
[Reference answer] {reference}
[Scoring requirements] {rubric}
[Answer to evaluate] {answer}

Score according to the rules and output in this format:
{{"correctness": score, "faithfulness": score, "helpfulness": score, "reason": "one-sentence justification"}}"""

3. Calling the judge and parsing its output

import json, re
from openai import OpenAI

# Local vLLM or any OpenAI-compatible endpoint
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
JUDGE_MODEL = "Qwen/Qwen2.5-72B-Instruct"

def judge(question, reference, answer, rubric):
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM},
            {"role": "user",
             "content": build_judge_prompt(question, reference, answer, rubric)},
        ],
        temperature=0,          # a judge must be deterministic
    )
    return parse_judge_json(resp.choices[0].message.content)

def parse_judge_json(text):
    m = re.search(r"\{.*\}", text, re.S)
    if not m:
        raise ValueError(f"could not parse judge output: {text}")
    return json.loads(m.group(0))

Three practical points:

• temperature=0: the judge's output must be reproducible; otherwise "a different score on every run" destroys regression testing;
• Force JSON: hard-code the output format into the rubric, and let the script fall back to regex parsing;
• Calibrate the judge itself: score 20-30 samples with both humans and the judge and compute the correlation (Spearman); below 0.5, switch to a stronger judge or revise the rubric.

4. Consistency checks and sampling

The judge is cheap but can be distorted; humans are trustworthy but expensive. The right approach is run automatically on everything + sample by rule for human review:

def sample_for_human(results, n=30):
    """Prioritize review: lowest scores, highest score variance, and the edge category."""
    ranked = sorted(results, key=lambda r: r["score"])
    lowest = ranked[: n // 3]
    variance = sorted(results, key=lambda r: r["std"], reverse=True)[: n // 3]
    edges = [r for r in results if r["category"] == "edge"][: n // 3]
    return {r["id"] for r in lowest + variance + edges}

5. Regression run script and report output

Gather all the code above into one run_eval.py: a single run produces a complete report, and it supports comparing two models / two versions — the key step that turns "evaluation" from a one-off action into an engineering system.

"""run_eval.py: regression evaluation over the golden dataset, with two-model / two-version comparison."""
import argparse, json, time
from pathlib import Path
from sentence_transformers import SentenceTransformer
from openai import OpenAI

# ---------- Loading and inference ----------
def load_golden(path: Path) -> list:
    with open(path, encoding="utf-8") as f:
        return json.load(f)

def infer(client, model: str, question: str, max_new_tokens: int = 256) -> str:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
        max_tokens=max_new_tokens,
        temperature=0.7,
    )
    return resp.choices[0].message.content

# ---------- Metrics ----------
def score_one(item: dict, answer: str, encoder, client) -> dict:
    reference = item["reference"]
    sim = semantic_similarity(answer, reference, encoder)
    try:
        judge_scores = judge(client, item["question"], reference, answer,
                             item.get("rubric", ""))
    except Exception as e:
        judge_scores = {"correctness": 0, "faithfulness": 0,
                        "helpfulness": 0, "reason": f"judge_error: {e}"}
    return {
        "id": item["id"],
        "category": item.get("category", "typical"),
        "semantic_similarity": sim,
        "judge_avg": (judge_scores["correctness"] + judge_scores["faithfulness"]
                      + judge_scores["helpfulness"]) / 3,
        "judge_scores": judge_scores,
        "answer": answer,
    }

# ---------- Report ----------
def build_report(name: str, results: list) -> str:
    sims = [r["semantic_similarity"] for r in results]
    judges = [r["judge_avg"] for r in results]
    lines = [
        f"## {name}",
        f"- Samples: {len(results)}",
        f"- Mean semantic similarity: {sum(sims) / len(sims):.3f}",
        f"- Mean judge score: {sum(judges) / len(judges):.3f}",
        "",
        "| id | category | semantic | judge |",
        "|---|---|---|---|",
    ]
    for r in sorted(results, key=lambda x: x["id"]):
        lines.append(f"| {r['id']} | {r['category']} | "
                     f"{r['semantic_similarity']:.2f} | {r['judge_avg']:.2f} |")
    return "\n".join(lines)

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--golden", required=True, help="path to the golden dataset JSON")
    ap.add_argument("--model-a", required=True)
    ap.add_argument("--model-b", default=None)
    ap.add_argument("--output", default="eval_report.md")
    args = ap.parse_args()

    client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
    encoder = SentenceTransformer("BAAI/bge-small-zh-v1.5")
    golden = load_golden(Path(args.golden))

    report = []
    for name, model in [("model_a", args.model_a), ("model_b", args.model_b)]:
        if model is None:
            continue
        results = []
        for item in golden:
            answer = infer(client, model, item["question"])
            results.append(score_one(item, answer, encoder, client))
            time.sleep(0.2)   # basic rate limiting to protect local services
        report.append(build_report(name, results))

    Path(args.output).write_text("\n\n".join(report), encoding="utf-8")
    print(f"Report written to {args.output}")

if __name__ == "__main__":
    main()

How to run it:

python run_eval.py --golden golden_set.json --model-a "Qwen/Qwen2.5-7B-Instruct" \
                   --model-b "Qwen/Qwen2.5-7B-Instruct-lora" --output report.md

Wiring it into CI (GitHub Actions example):

name: llm-eval
on: [push, pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run evaluation
        run: python run_eval.py --golden data/golden_set.json --model-a ${{ secrets.EVAL_MODEL }}
      - name: Regression assertion
        run: python check_regression.py report.md --min-judge 3.5 --min-sim 0.6

The core logic of check_regression.py is a single check:

def assert_no_regression(report_path: str, min_judge: float, min_sim: float):
    text = Path(report_path).read_text(encoding="utf-8")
    judge = float(re.findall(r"Mean judge score: ([0-9.]+)", text)[0])
    sim = float(re.findall(r"Mean semantic similarity: ([0-9.]+)", text)[0])
    if judge < min_judge or sim < min_sim:
        raise SystemExit(f"Regression failed: judge={judge:.2f}, sim={sim:.2f}")

6. When to use humans, when to use automation

No evaluation method is a free lunch. This table helps you choose:

Dimension	Human evaluation	Automated metrics	LLM-as-a-judge
Cost	High (expensive, slow)	Lowest	Medium (needs GPU / API calls)
Reliability	Highest (humans understand "good")	Depends on metric-task fit	Medium (biased, see below)
Scalability	Poor (~50 samples is the ceiling)	Good (can run on everything)	Good (can run on everything)
Best for	Calibrating automated metrics, spot checks	Tasks with standard answers	Fallback for open-ended QA
Role	Yardstick	Workhorse	Supplement

One-line summary: automated metrics handle "fast and broad," humans handle "accurate and deep," and LLM-as-a-judge sits in between, simulating human judgment at scale.

Choosing a judge model and its biases

Pick a judge model that is clearly stronger than the model being evaluated (GPT-4o / Claude / Qwen-72B class) and, where possible, from a different family than your production model to reduce self-preference. For background on model capability and selection, see Large Language Models (LLMs). Four known biases:

Bias	Symptom	Mitigation
Self-preference bias	The judge scores outputs from its own family higher	Use a judge from a different family than the production model
Length / verbosity bias	Longer answers get inflated scores	Add a "conciseness" dimension to the rubric with explicit deduction rules
Position bias	The candidate listed first scores higher	Shuffle the order and average over two runs
Format sensitivity	Line breaks, lists, and markdown shift scores	Normalize output before scoring, use temperature=0

Every judge must be calibrated against humans before it goes live: if the human-judge correlation on 20-30 samples falls short, swap the model or revise the rubric — don't hit the road with a broken instrument.

7. Evaluation in production: from "one-off" to "continuous"

An offline golden set can only answer "did my change regress things"; it cannot answer "did the business actually improve after launch." In production, lay down at least four layers:

Layer	Method	Goal	Related
Offline regression	Golden dataset + CI	No regressions from changes	Sections 5 and 6 of this article
Online monitoring	Sampled traffic scoring, latency / error rates	No degradation after launch	Deployment and Inference Optimization in Practice
User feedback loop	Thumbs up / down + reason collection	A steady stream of real labels	Flows back into the golden dataset and fine-tuning data
Red teaming	Adversarial testing, jailbreak attacks	Safety and robustness	AI Safety and Governance

1. Online monitoring

Apply stratified sampling to production traffic (by user, by question type), feed the sampled (question, answer, context) triples through the same evaluation pipeline for scoring, and compare score distributions day over day. At the same time, watch the engineering metrics: time-to-first-token, throughput, error rate — however pretty the evaluation scores look, they mean nothing if the service is down. For the full deployment and monitoring picture, see Deployment and Inference Optimization in Practice.

2. User feedback loop

A user thumbs-down is a free label. Design points:

• Follow up after a thumbs-down to ask why ("off-topic / factually wrong / unsafe / other") so the feedback can be consumed in structured form;
• Feed "user thumbs-down + the question/answer/context at the time" back into the golden dataset (human-review it first before adding);
• These samples are also a natural source of fine-tuning or DPO preference data.

3. Red teaming

For safety-sensitive scenarios, run adversarial testing on a schedule: jailbreak prompts, prompt injection, privacy extraction, role-play manipulation. Red teaming is not a "test once" activity but "continuous operations"; for concrete methods and governance frameworks, see AI Safety and Governance.

8. Common pitfalls: one table to debug them

Pitfall	Symptom	Fix
Eval set contamination	Offline scores look great, production collapses on day one	Golden dataset is append-only; detect whether the model has "seen" a sample (ask the reference verbatim); refresh with newly sampled data periodically
Judge bias	Scores favor long answers or a particular style	See the bias table in Section 6; calibrate correlation against humans first
Metrics detached from the business	Offline scores climb, the business doesn't move	Go back to Step 1 and re-align business goals with proxy metrics
Too few samples, high variance	The same model varies by 10 points between runs	Grow the golden dataset (50 is the floor); fix random seeds and use temperature=0
Evaluating RAG end-to-end only	Score is low but you can't tell which layer broke	Layer it with RAGAS: context_recall for the retrieval layer, faithfulness for the generation layer
Tuning on the test set	The more you tune, the faker the scores	Treat the golden dataset as a "test set"; tune only on a dev set

All of these pitfalls share one root cause: the goal of evaluation is to answer "did my change actually make the system better," not "prove the system is good." Put that sentence on the meeting-room wall.

Further Reading

• LLM Evaluation and Benchmarks — the theory behind this article's metrics: evaluation methodology, public benchmarks, and leaderboards
• Prompt Engineering Playbook — connecting evaluation to prompt iteration: driving prompt improvements with evals
• Build a RAG Application from Scratch — layered evaluation design for RAG scenarios
• Fine-Tune Your Own LLM — how to accept fine-tuned models with this evaluation suite
• Vector Databases and Semantic Search — the mechanics behind semantic similarity metrics
• Deployment and Inference Optimization in Practice — online monitoring and production deployment
• AI Safety and Governance — red teaming and safety evaluation
• Common Pitfalls and Anti-Patterns — a catalog of project-level pitfalls beyond evaluation
• Large Language Models (LLMs) — background on judge model selection and capability

References

• Es et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation (EACL 2024) — the original RAGAS paper, source of the faithfulness / relevancy metrics
• Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023) — bias analysis and validation methods for LLM-as-a-judge
• Liu et al. G-Eval: NLG Evaluation using GPT-4 (NAACL 2023) — research on NLG evaluation and rubric-based scoring with GPT-4
• sentence-transformers documentation — official docs for the embedding models used in semantic similarity
• RAGAS documentation — metric list, configuration, and current usage
• Hugging Face evaluate docs — official implementations of classic text metrics like BLEU and ROUGE
• OpenAI Evals — open-source evaluation framework; an engineering reference for golden set + judge
• lm-evaluation-harness (EleutherAI) — the de facto standard for academic benchmark evaluation
• MT-Bench (FastChat) — open-source implementation of multi-turn conversation judge evaluation
• OpenAI: Prompt engineering guide — design reference for judge prompts

Suggested order: define goals (Step 1) → write 50-200 golden samples → get exact match + semantic similarity running with the Section 3 code → add the judge and calibrate it against humans → wire run_eval.py into CI → after launch, add online monitoring and the user feedback loop.