Posted on
• Artificial Intelligence

RAG Evaluation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG Evaluation on Linux: Measure Before You Ship

If your Retrieval-Augmented Generation (RAG) system crushes demos but stumbles in production, you’re not alone. RAG pipelines are deceptively complex: the retriever can silently miss the right passages, the generator can hallucinate, and latency can spike under real workloads. The fix is systematic, repeatable evaluation—run from Bash, versioned with your code, and enforced in CI.

This post explains what to measure, why it matters, and how to stand up a lightweight, Linux-friendly RAG evaluation workflow with simple commands you can drop into your scripts and pipelines.

Why RAG evaluation is worth your time

  • It separates retrieval quality from generation quality. When answers are wrong, you’ll know whether to tune your vector store or your model prompts.

  • It prevents silent regressions. A small index change or model update can tank Recall@k without anyone noticing—unless you gate merges on metrics.

  • It speeds iteration. Fast, local metrics let you A/B embeddings, chunking, and prompting without subjectivity.

  • It aligns with ops. You can track accuracy and latency together to avoid “fast but dumb” or “smart but slow” releases.

What to measure (and what each metric tells you)

  • Retrieval Recall@k: Of the ground-truth relevant docs, how many did you retrieve? Low recall means your search/indexing/embedding setup is missing targets.

  • Context Precision@k: Of the retrieved docs, how many are actually relevant? Low precision points to query formulation or overly broad retrieval.

  • Answer Similarity: How close is the model’s answer to a known-good reference? A cosine similarity on sentence embeddings is a strong, language-agnostic baseline.

  • Latency: End-to-end time per query. Accuracy means little if users bounce before results render.

You can start with these four and add advanced checks (faithfulness, groundedness) later.

Step 1 — Install prerequisites

You’ll need Python, pip, and a few CLI helpers. Use your distro’s package manager.

  • Ubuntu/Debian (apt):

    • sudo apt update && sudo apt install -y python3 python3-venv python3-pip git jq curl

  • Fedora/RHEL (dnf):

    • sudo dnf install -y python3 python3-pip git jq curl

  • openSUSE (zypper):

    • sudo zypper refresh && sudo zypper install -y python3 python3-pip git jq curl

Create and activate a virtual environment (recommended):

  • python3 -m venv .venv

  • source .venv/bin/activate

Then install Python libs for evaluation:

  • pip install -U pip wheel

  • pip install pandas numpy scikit-learn sentence-transformers faiss-cpu tqdm

Why these?

  • sentence-transformers gives you high-quality text embeddings for answer similarity.

  • faiss-cpu is handy if you also want to prototype retrieval locally.

  • pandas/numpy for data wrangling; tqdm for progress.

Step 2 — Prepare a small test dataset

Start with a JSONL file where each line is one query. Include what you retrieved and what the model answered. Example (dev.jsonl):

  • {"id":"q1","question":"Who wrote the Linux kernel?","gt_answer":"Linus Torvalds wrote the original Linux kernel.","gt_doc_ids":["doc_42"],"retrieved_doc_ids":["doc_11","doc_42","doc_99"],"model_answer":"Linus Torvalds created the Linux kernel.","latency_ms":380}

  • {"id":"q2","question":"What is RAG in NLP?","gt_answer":"Retrieval-Augmented Generation combines document retrieval with text generation.","gt_doc_ids":["doc_A","doc_B"],"retrieved_doc_ids":["doc_Z","doc_Y"],"model_answer":"RAG means mixing retrieval and generation to answer questions.","latency_ms":615}

Notes:

  • gt_doc_ids are the IDs of documents you consider relevant (from your labeling or curated pairs).

  • retrieved_doc_ids should be whatever your system actually fetched for that question.

  • gt_answer is a short canonical answer. It doesn’t need to be perfect; it’s a reference.

If you don’t have labels yet, start with a tiny, hand-labeled set (20–50 examples). You’ll get value immediately.

Step 3 — Drop in a tiny evaluator (Python)

Save this as eval_rag.py and commit it with your codebase.

#!/usr/bin/env python3
import argparse, json, sys, math
from statistics import mean
from typing import List
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

def jread(path):
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line=line.strip()
            if line:
                yield json.loads(line)

def jdump(obj, path):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(obj, f, indent=2, ensure_ascii=False)

def safe_div(a,b):
    return float(a)/float(b) if b else math.nan

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--input", required=True, help="JSONL with fields: gt_doc_ids, retrieved_doc_ids, gt_answer, model_answer, latency_ms")
    p.add_argument("--output", required=True, help="Path to write metrics JSON")
    p.add_argument("--k", type=int, default=3, help="Top-k to evaluate for retrieval metrics")
    p.add_argument("--embed-model", default="sentence-transformers/all-MiniLM-L6-v2")
    args = p.parse_args()

    data = list(jread(args.input))
    if not data:
        print("No records in input.", file=sys.stderr)
        sys.exit(2)

    # Truncate retrieved docs to k
    for r in data:
        r["retrieved_doc_ids"] = (r.get("retrieved_doc_ids") or [])[:args.k]

    # Retrieval metrics
    recalls, precisions = [], []
    for r in data:
        gt = set(r.get("gt_doc_ids") or [])
        rt = set(r.get("retrieved_doc_ids") or [])
        inter = len(gt & rt)
        recalls.append(safe_div(inter, len(gt)) if len(gt)>0 else math.nan)
        precisions.append(safe_div(inter, len(rt)) if len(rt)>0 else math.nan)

    # Answer similarity via embeddings
    model = SentenceTransformer(args.embed_model)
    gt_answers = [r.get("gt_answer","") for r in data]
    model_answers = [r.get("model_answer","") for r in data]
    emb_gt = model.encode(gt_answers, convert_to_numpy=True, normalize_embeddings=True)
    emb_ans = model.encode(model_answers, convert_to_numpy=True, normalize_embeddings=True)
    sims = cosine_similarity(emb_gt, emb_ans).diagonal().tolist()

    # Latency
    lats = [r.get("latency_ms") for r in data if isinstance(r.get("latency_ms"), (int,float))]

    # Aggregate means (ignoring NaN)
    def nanmean(xs):
        xs = [x for x in xs if isinstance(x,(int,float)) and not math.isnan(x)]
        return mean(xs) if xs else math.nan

    report = {
        f"retrieval_recall@{args.k}": nanmean(recalls),
        f"context_precision@{args.k}": nanmean(precisions),
        "answer_similarity": nanmean(sims),
        "latency_ms_p50": (sorted(lats)[len(lats)//2] if lats else math.nan),
        "latency_ms_p95": (sorted(lats)[int(len(lats)*0.95)-1] if lats else math.nan),
        "count": len(data),
        "embed_model": args.embed_model,
        "k": args.k,
    }
    jdump(report, args.output)
    print(json.dumps(report, indent=2))

if __name__ == "__main__":
    main()

Run it:

  • python3 eval_rag.py --input dev.jsonl --output out/metrics.json --k 3

Pretty-print the results:

  • jq . out/metrics.json

You’ll see something like:

{
  "retrieval_recall@3": 0.75,
  "context_precision@3": 0.58,
  "answer_similarity": 0.84,
  "latency_ms_p50": 410,
  "latency_ms_p95": 820,
  "count": 48,
  "embed_model": "sentence-transformers/all-MiniLM-L6-v2",
  "k": 3
}

Interpretation:

  • If retrieval_recall@3 is low while answer_similarity is okay, your reference set might be forgiving—or your retriever is failing but the generator is guessing right. Investigate.

  • High recall with low precision means you’re fetching too many distractors. Tighten query or reduce k.

Step 4 — Automate from Bash

Add a Make target and a quality gate to catch regressions.

Makefile snippet:

RAG_K ?= 3
RAG_INPUT ?= dev.jsonl
RAG_OUT ?= out/metrics.json
RAG_MIN_RECALL ?= 0.70
RAG_MIN_SIM ?= 0.80

.PHONY: eval-rag
eval-rag:
    @mkdir -p out
    python3 eval_rag.py --input $(RAG_INPUT) --output $(RAG_OUT) --k $(RAG_K)

.PHONY: gate-rag
gate-rag: eval-rag
    @actual_recall=$$(jq -r '.["retrieval_recall@$(RAG_K)"]' $(RAG_OUT)); \
     actual_sim=$$(jq -r '.answer_similarity' $(RAG_OUT)); \
     echo "Recall@$(RAG_K): $$actual_recall (min $(RAG_MIN_RECALL))"; \
     echo "Answer similarity: $$actual_sim (min $(RAG_MIN_SIM))"; \
     awk -v a="$$actual_recall" -v t="$(RAG_MIN_RECALL)" 'BEGIN{ exit(a<t) }' || { echo "FAIL: recall below threshold"; exit 1; }; \
     awk -v a="$$actual_sim" -v t="$(RAG_MIN_SIM)" 'BEGIN{ exit(a<t) }' || { echo "FAIL: similarity below threshold"; exit 1; }; \
     echo "PASS"

Then wire make gate-rag into your CI job. Any regression fails the build.

Step 5 — Iterate with confidence (a mini playbook)

  • Tune k with intent: Run --k 1, --k 3, --k 5. If Recall@1 is far below Recall@3 but precision plummets, consider smarter reranking rather than indiscriminately increasing k.

  • A/B embeddings: Swap --embed-model to sentence-transformers/all-mpnet-base-v2 or multi-qa-MiniLM-L6-cos-v1 and compare. Lock in the best performer.

  • Test chunking strategies: Smaller chunks often boost recall but hurt precision. Run your eval after each change—keep what improves the aggregate, not anecdotes.

  • Track latency with load: Feed a larger dev.jsonl and compare latency_ms_p95 before/after indexing/prompt changes.

  • Grow your dataset: Add failure cases from production to dev.jsonl. Your metrics become more predictive as the set diversifies.

Real-world example: One team saw “good answers” in demos but low Recall@3 (~0.35). It turned out their indexer trimmed section headers, breaking entity recall. Fixing chunk metadata doubled recall and similarity, and reduced p95 latency by lowering k.

Optional: Add a faithfulness check later

Once you’re stable on the basics, add a groundedness/faithfulness check using a small NLI model or rule-based overlap against retrieved contexts. This requires storing the actual retrieved passages, not just IDs. Keep it optional at first; the four core metrics above already catch most regressions.

Conclusion and next steps

RAG isn’t magic—it’s engineering. With a small JSONL, a 100-line evaluator, and a Make target, you can quantify improvements, prevent backslides, and ship with confidence.

Your next steps: 1. Install the prerequisites with your package manager. 2. Create a 20–50 example dev.jsonl from real queries. 3. Drop in eval_rag.py, run it, and set thresholds. 4. Wire make gate-rag into CI and iterate.

Have a bigger stack (LangChain, LlamaIndex, or vector DBs)? Keep this evaluator as your neutral ground truth. Glue code changes; metrics don’t.

Related posts