Posted on
• Artificial Intelligence

RAG Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG Best Practices: Make Your LLM Stop Hallucinating (The Bash-Friendly Way)

If your LLM keeps “confidently” making things up about your product, APIs, or internal docs, you don’t need more temperature tweaking—you need Retrieval-Augmented Generation (RAG). RAG plugs your model into your actual knowledge base, so answers are grounded in your content, not random priors.

This post walks Linux users through practical, command-line-friendly RAG best practices. You’ll get a clean workflow to prepare data, embed it, retrieve it intelligently, and prompt defensively—plus copy-pasteable commands for apt, dnf, and zypper.

What you’ll get:

  • Why RAG matters and how it saves you time and credibility

  • A minimal, reproducible pipeline that runs on any Linux distro

  • 3–5 actionable practices with code you can adapt today

  • A small end-to-end example (index + query + rerank + prompt)

Why RAG on Linux?

  • Control: Keep your data local, scriptable, and automatable (cron, systemd, CI).

  • Performance: Use efficient embeddings and caching; iterate quickly.

  • Reproducibility: Version-lock your pipeline (requirements, corpora, configs).

  • Cost and privacy: Reduce API calls, keep sensitive content offline if needed.

Prereqs: System packages

Install common tooling for Python, building wheels, and text extraction.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip build-essential git curl jq ripgrep poppler-utils

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y python3 python3-pip python3-virtualenv gcc gcc-c++ make git curl jq ripgrep poppler-utils

Zypper (openSUSE):

sudo zypper refresh
sudo zypper install -y python3 python3-pip gcc-c++ make git curl jq ripgrep poppler-tools

Create an isolated Python env and install core libs:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install chromadb sentence-transformers langchain tiktoken openai pypdf rapidfuzz ragas

Notes:

  • poppler-utils/poppler-tools gives you pdftotext for fast PDF extraction.

  • If you prefer local LLMs, you can wire in Ollama (HTTP) or text-generation-webui; the retrieval/indexing process stays the same.

Best Practice 1: Clean, normalize, and chunk your corpus

Garbage in = garbage out. Before embedding, convert to plain UTF-8, strip noise, and chunk with overlap.

Example: convert PDFs and collect MD/TXT into a normalized folder.

mkdir -p corpus/txt

# Convert PDFs to text (preserving layout helps tables/code)
find ~/docs -type f -iname '*.pdf' -print0 | xargs -0 -I{} sh -c \
'out="corpus/txt/$(basename "{}" .pdf).txt"; pdftotext -layout "{}" "$out"'

# Copy existing .md and .txt
rsync -a --include='*/' --include='*.md' --include='*.txt' --exclude='*' ~/docs/ corpus/txt/

# Normalize to UTF-8, strip CRs and trailing spaces
find corpus/txt -type f \( -iname '*.txt' -o -iname '*.md' \) -print0 | while IFS= read -r -d '' f; do
  tmp="$f.norm"
  iconv -f utf-8 -t utf-8 -c "$f" | tr -d '\r' | sed -E 's/[ \t]+$//' > "$tmp" && mv "$tmp" "$f"
done

# Quick sanity check: look for obvious junk
rg -n --hidden --max-columns 200 '�|<script|BEGIN CERTIFICATE' corpus/txt || true

Chunking and metadata-aware splitting (Python + LangChain):

python - <<'PY'
import os, glob, json, pathlib
from langchain.text_splitter import RecursiveCharacterTextSplitter

root = "corpus/txt"
out = "corpus/chunks"
pathlib.Path(out).mkdir(parents=True, exist_ok=True)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=150,
    separators=["\n\n", "\n", " ", ""]
)

docs = []
for f in glob.glob(f"{root}/**/*", recursive=True):
    if not os.path.isfile(f): 
        continue
    with open(f, "r", encoding="utf-8", errors="ignore") as ih:
        text = ih.read()
    if not text.strip():
        continue
    chunks = splitter.split_text(text)
    for i, chunk in enumerate(chunks):
        meta = {"source": os.path.relpath(f, root), "chunk_id": i}
        # Save one-file-per-chunk as JSONL-like
        base = os.path.basename(f)
        out_f = f"{out}/{base}.{i}.json"
        with open(out_f, "w", encoding="utf-8") as oh:
            json.dump({"text": chunk, "metadata": meta}, oh, ensure_ascii=False)
print("Chunking complete.")
PY

Tips:

  • Keep chunk size near your model’s attention sweet spot (800–1500 chars works well for general docs).

  • Always carry metadata (source, section, timestamp) for citations and filtering.

Best Practice 2: Embed with the right model and persist vectors

Start with a small, fast sentence-transformer (great quality/speed tradeoff). Persist to disk so you can query without re-embedding.

python - <<'PY'
import os, json, glob, pathlib
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer

# Choose a strong baseline embedder; swap later as needed
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"

client = chromadb.Client(Settings(anonymized_telemetry=False, persist_directory="db"))
col = client.get_or_create_collection(name="docs", metadata={"hnsw:space": "cosine"})

embedder = SentenceTransformer(EMBED_MODEL)

ids, texts, metas = [], [], []
for f in glob.glob("corpus/chunks/*.json"):
    with open(f, "r", encoding="utf-8") as ih:
        obj = json.load(ih)
    doc_id = os.path.basename(f)
    ids.append(doc_id)
    texts.append(obj["text"])
    metas.append(obj["metadata"])

# Batch encode for speed
embs = embedder.encode(texts, batch_size=64, show_progress_bar=True, normalize_embeddings=True)
col.add(ids=ids, documents=texts, metadatas=metas, embeddings=embs.tolist())

client.persist()
print(f"Indexed {len(ids)} chunks into ./db")
PY

Notes:

  • normalize_embeddings=True improves cosine similarity behavior.

  • Persisting means subsequent queries are instant; back up ./db for reproducibility.

Best Practice 3: Retrieve smartly (recall first, then rerank)

A common anti-pattern: n_results=3 and hope for the best. Instead: 1) Pull a wider candidate set with vector search (e.g., top 20–50). 2) Rerank with a cross-encoder for precision (it reads query+passage together).

python - <<'PY'
import os
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer, CrossEncoder

EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
RERANK_MODEL = "cross-encoder/ms-marco-MiniLM-L-6-v2"

client = chromadb.Client(Settings(persist_directory="db", anonymized_telemetry=False))
col = client.get_collection("docs")

embedder = SentenceTransformer(EMBED_MODEL)
reranker = CrossEncoder(RERANK_MODEL)

def retrieve(query, top_k_candidates=30, top_k_final=5):
    q_emb = embedder.encode([query], normalize_embeddings=True)[0].tolist()
    res = col.query(
        query_embeddings=[q_emb],
        n_results=top_k_candidates,
        include=["documents", "metadatas", "distances"]
    )
    docs = list(zip(res["documents"][0], res["metadatas"][0]))

    # Rerank
    pairs = [(query, d[0]) for d in docs]
    scores = reranker.predict(pairs).tolist()
    ranked = sorted(zip(docs, scores), key=lambda x: x[1], reverse=True)[:top_k_final]

    contexts = [{"text": d[0], "metadata": d[1], "score": s} for (d, s) in ranked]
    return contexts

if __name__ == "__main__":
    q = "How do I authenticate to our internal API and rotate tokens?"
    ctxs = retrieve(q)
    for i, c in enumerate(ctxs, 1):
        print(f"#{i} {c['metadata']['source']} (score={c['score']:.3f})\n{c['text'][:300]}\n---")
PY

Optional hybrid hint:

  • Combine lexical pre-filters (e.g., rg -l "token|api|auth") to scope retrieval, then vector search on that subset. This is fast and often boosts precision on keyword-heavy domains.

Best Practice 4: Prompt defensively and manage token budget

Don’t let the model wander. Tell it exactly how to behave, cite sources, and admit uncertainty. Trim context to fit.

python - <<'PY'
import os, tiktoken
from textwrap import dedent
from openai import OpenAI

def build_prompt(question, contexts, max_ctx_tokens=1500, model="gpt-4o-mini"):
    # Prepare a compact, citeable context block
    def cite(d):
        src = d["metadata"].get("source", "unknown")
        return f"[Source: {src}]\n{d['text'].strip()}\n"
    ctx_block = "\n\n".join(cite(c) for c in contexts)

    # Token-aware trimming
    enc = tiktoken.get_encoding("cl100k_base")  # works for many chat models
    tokens = enc.encode(ctx_block)
    if len(tokens) > max_ctx_tokens:
        ctx_block = enc.decode(tokens[:max_ctx_tokens])

    system = dedent("""\
      You are a concise, accurate assistant.
      Use ONLY the provided Context. If the answer is not in the Context, say "I don't know".
      Always cite sources in square brackets at the end of each sentence when applicable.
    """)
    user = f"Question: {question}\n\nContext:\n{ctx_block}\n\nAnswer:"

    return system, user

def answer(question, contexts):
    api_key = os.getenv("OPENAI_API_KEY")
    if not api_key:
        print("OPENAI_API_KEY not set; printing prompt instead.\n")
        s,u = build_prompt(question, contexts)
        print("SYSTEM:\n", s, "\nUSER:\n", u[:2000], "...")
        return

    client = OpenAI()
    s, u = build_prompt(question, contexts)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"system","content":s},{"role":"user","content":u}],
        temperature=0.1
    )
    print(resp.choices[0].message.content)

if __name__ == "__main__":
    # Fake minimal contexts (in real code, feed from the retriever above)
    contexts = [
        {"text": "Use POST /v1/token/rotate with a valid admin token.", "metadata": {"source":"api/auth.md"}},
        {"text": "Tokens expire after 24h; refresh via /v1/token/refresh.", "metadata": {"source":"api/tokens.md"}},
    ]
    answer("How do I rotate API tokens?", contexts)
PY

Prompting checklist:

  • Give the assistant strict rules.

  • Include only necessary context; keep margin for the model’s output.

  • Temperature low (0–0.2) for factual tasks.

  • Ask for citations.

If you run a local model (e.g., Ollama), post the prompt to its HTTP endpoint:

curl http://localhost:11434/api/generate -d @- <<'JSON'
{
  "model": "llama3",
  "prompt": "SYSTEM: You are ...\nUSER: Question: ...\nContext: ...\nAnswer:",
  "stream": false
}
JSON

Best Practice 5: Evaluate and monitor

Measure retrieval and answer quality so you can iterate with confidence.

Minimal RAG evaluation with ragas:

python - <<'PY'
import os
from datasets import Dataset
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from ragas import evaluate

# Example test set; in practice, keep 20–100 QAs your team cares about
data = Dataset.from_dict({
    "question": [
        "How do I rotate API tokens?",
        "Where is the backup policy documented?",
    ],
    "answer": [
        "Use POST /v1/token/rotate with a valid admin token. [api/auth.md]",
        "Backups are explained in the SRE runbook. [sre/runbook.md]"
    ],
    "contexts": [
        ["Use POST /v1/token/rotate...", "Tokens expire after 24h..."],
        ["SRE runbook section 3 covers backups", "Weekly snapshots in region us-east-1"]
    ],
})

# Requires an LLM for grading; set OPENAI_API_KEY or wire your local grader
report = evaluate(data, metrics=[faithfulness, answer_relevancy, context_precision])
print(report)
PY

Operational tips:

  • Keep an evaluation set in Git and run it on every embedding/model change.

  • Log retrieval hits and misses; inspect wrongly-ranked chunks and tweak chunking/overlap.

  • Version-lock: pip freeze > requirements.txt and tag both code and index snapshots.

Real-world mini-pipeline: index + query + rerank + answer

1) Prepare corpus (convert + normalize). 2) Chunk and embed to Chroma. 3) Retrieve top 30, rerank to top 5. 4) Prompt with strict rules and citations. 5) Evaluate on a small QA set.

Try it end-to-end with your ~/docs, then:

  • Swap in a larger embedding model for better recall (e.g., bge-base/bge-large).

  • Add filters: by team, date, product version.

  • Containerize and schedule re-index in cron or CI.

Conclusion and Next Steps

RAG is the difference between “sounds right” and “is right.” On Linux you can wire everything with a few scripts, keep your data local, and iterate fast.

Your next step:

  • Point the scripts at a real repo or wiki and build your first index.

  • Add the reranker and defensive prompt, then run a tiny evaluation set.

  • Automate re-indexing and snapshot both code and data.

Questions or want a deeper dive (e.g., hybrid BM25+vector, Qdrant/Milvus, or GPU serving)? Tell me about your stack and data shape, and I’ll share a tailored recipe.

Related posts