Posted on
• Artificial Intelligence

RAG Performance Optimisation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG Performance Optimisation from the Linux Command Line

If your Retrieval-Augmented Generation (RAG) system answers “pretty well” but latency spikes, recall misses, or cloud bills say otherwise, you’ve got performance on the table. The good news: you can claw back accuracy and speed with a handful of tactical changes you can drive entirely from your Linux shell.

This guide explains why RAG performance work matters, and gives you 5 concrete, CLI-friendly steps you can implement today—complete with multi-distro install commands and reproducible scripts.

Why optimise RAG?

  • Latency compounds: retrieval, reranking, and generation each add delay. A slow retriever starves your LLM.

  • Accuracy drives trust: better chunking and indexing improve recall and precision, which directly improves answer quality (and reduces “hallucinated glue”).

  • Cost is performance: fewer calls, leaner reranking, and right-sized indexes reduce compute and hosting bills.

Prerequisites

Install core tooling. Pick the command set for your distro:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl jq ripgrep apache2-utils
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-virtualenv python3-pip git curl jq ripgrep httpd-tools
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y python3 python3-virtualenv python3-pip git curl jq ripgrep apache2-utils

Create an isolated Python environment and install common RAG libs:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install faiss-cpu sentence-transformers scikit-learn chromadb qdrant-client pymilvus tiktoken ragas datasets fastapi uvicorn

Tip: If you compile anything from source and hit build errors, install compilers:

  • apt: sudo apt install -y build-essential

  • dnf: sudo dnf install -y @development-tools

  • zypper: sudo zypper install -y gcc gcc-c++ make

Export a few CPU threading knobs (tune to your core count):

export OMP_NUM_THREADS=8
export MKL_NUM_THREADS=8

1) Choose the right index and tune it

Dense vector search is the workhorse of RAG. Two battle-tested paths:

  • Local and lightweight: FAISS or Chroma (HNSW under the hood).

  • Production/clustered: Qdrant or Milvus.

Start local with FAISS, then swap to a service when you outgrow it.

Quick FAISS baseline (IVF Flat with tunable nlist/nprobe):

python3 - <<'PY'
import faiss, numpy as np
from sentence_transformers import SentenceTransformer

# Config
model_name = "sentence-transformers/all-MiniLM-L6-v2"
nlist = 1024   # number of coarse clusters
nprobe = 16    # how many clusters to search at query time
d = 384        # embedding dimension for the model above

# Toy corpus
docs = [f"Document {i} about Linux Bash and RAG {i%5}" for i in range(20000)]
model = SentenceTransformer(model_name)
emb = np.array(model.encode(docs, batch_size=256, normalize_embeddings=True), dtype="float32")

quantizer = faiss.IndexFlatIP(d)
index = faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_INNER_PRODUCT)
index.train(emb)
index.add(emb)
index.nprobe = nprobe

def search(q, k=5):
    qv = np.array(model.encode([q], normalize_embeddings=True), dtype="float32")
    D, I = index.search(qv, k)
    return [docs[i] for i in I[0]]

print(search("optimize retrieval latency with bash"))
PY

Tuning notes:

  • Raise nlist for larger corpora (e.g., 4096–16384); raise nprobe for better recall at the cost of latency.

  • Prefer inner product with normalized vectors for speed and stability.

  • For even faster recall at scale with lower RAM, move to IVF+PQ or HNSW. When using Chroma, look at HNSW params: ef_construction, M, and query-time ef_search.

When graduating to a service:

  • Qdrant: good defaults, strong HNSW performance, simple Docker.

  • Milvus: IVF/HNSW/PQ flexibility, GPU options for FAISS-based pipelines.

2) Chunk smarter, not just smaller

Chunk size, overlap, and boundaries strongly affect both recall and latency. Too small hurts context; too large hurts match precision and retrieval speed. Start with 256–512 tokens and 10–64 overlap and measure.

Simple CLI-friendly chunker (word-based with approximate token sizing):

cat > chunk.py <<'PY'
import math, sys, json
from pathlib import Path

path = Path(sys.argv[1])
text = path.read_text(encoding="utf-8", errors="ignore")

# naive token proxy: 1 token ~ 4 chars (adjust as needed)
target_tokens = int(sys.argv[2]) if len(sys.argv) > 2 else 384
overlap = int(sys.argv[3]) if len(sys.argv) > 3 else 32
chars_per_token = 4
size = target_tokens * chars_per_token
ovlp = overlap * chars_per_token

chunks = []
i = 0
while i < len(text):
    end = min(len(text), i + size)
    chunk = text[i:end]
    # try to end at a sentence boundary to keep semantics tight
    j = chunk.rfind(". ")
    if j > size * 0.5:
        chunk = chunk[:j+1]
        end = i + j + 1
    chunks.append(chunk.strip())
    i = max(end - ovlp, end) if ovlp < size else end

for c in chunks:
    if c:
        print(json.dumps({"text": c}))
PY

Use it like:

python3 chunk.py docs/handbook.md 384 32 | jq -c . > chunks.jsonl

Evaluate chunk strategies quickly:

for size in 256 384 512; do
  for ov in 16 32 64; do
    echo "size=$size overlap=$ov"
    /usr/bin/time -f "user=%U sys=%S elapsed=%E" python3 chunk.py docs/handbook.md $size $ov | wc -l
  done
done

What to look for:

  • Fewer, larger chunks reduce index size and speed up build times, but can hurt retrieval specificity.

  • Slight overlap (10–15% of chunk size) maintains continuity for boundary-spanning facts.

3) Speed up embeddings without sacrificing too much quality

Your embedding throughput often gates ingestion and live updates.

  • Pick a strong, fast small model: all-MiniLM-L6-v2, gte-small, bge-small.

  • Normalize embeddings and use inner product search.

  • Batch aggressively; align to CPU cores or GPU memory.

Batched encoder with optional GPU use:

python3 - <<'PY'
import os, numpy as np, torch
from sentence_transformers import SentenceTransformer

docs = [f"Doc {i} about Linux and vector search {i%7}" for i in range(50000)]
model_name = os.environ.get("EMBED_MODEL","sentence-transformers/all-MiniLM-L6-v2")
batch = int(os.environ.get("BATCH", "512"))

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SentenceTransformer(model_name, device=device)

embs = model.encode(
    docs, batch_size=batch, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=True
)

print(embs.shape, embs.dtype, "device:", device)
PY

Tips:

  • CPU-only? Set OMP_NUM_THREADS to physical cores; batch size 256–1024 is often fine.

  • GPU available? Try BATCH=2048 and mixed precision:

export CUDA_VISIBLE_DEVICES=0
export TOKENIZERS_PARALLELISM=false
python3 - <<'PY'
import torch
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda")
model = model.half()  # fp16
print("Ready:", next(model.parameters()).dtype)
PY

Cache embeddings for re-ingestion speedups (simple SQLite cache):

python3 - <<'PY'
import sqlite3, hashlib, json, os
from sentence_transformers import SentenceTransformer

db = sqlite3.connect("emb_cache.sqlite")
db.execute("CREATE TABLE IF NOT EXISTS emb (k TEXT PRIMARY KEY, v BLOB)")
db.commit()

def key(x): return hashlib.sha256(x.encode()).hexdigest()
def get_cached(keys):
    q = "SELECT k,v FROM emb WHERE k IN (%s)" % ",".join(["?"]*len(keys))
    return {k: json.loads(v) for k,v in db.execute(q, keys)}

def set_cached(kv):
    db.executemany("INSERT OR REPLACE INTO emb (k,v) VALUES (?,?)", [(k, json.dumps(v)) for k,v in kv.items()])
    db.commit()

docs = ["some text", "another doc", "some text"]  # dedupe by hash
uniq = list({key(d): d for d in docs}.items())
keys, vals = zip(*uniq)

cached = get_cached(keys)
todo = [(k,v) for k,v in uniq if k not in cached]

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
newvecs = {}
if todo:
    embs = model.encode([v for _,v in todo], normalize_embeddings=True).tolist()
    for (k,_), e in zip(todo, embs):
        newvecs[k] = e
    set_cached(newvecs)

print("cached:", len(cached), "new:", len(newvecs))
PY

4) Retrieve better: hybrid search and lightweight reranking

Dense-only misses exact-phrase and rare-term queries. Hybrid = BM25 (sparse) + dense gives the best of both.

Quick hybrid prototype in Python (TF-IDF for sparse + FAISS for dense, score blending):

python3 - <<'PY'
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
import faiss
from sentence_transformers import SentenceTransformer

docs = [f"Linux Bash trick #{i}: use ripgrep for RAG preprocessing {i%11}" for i in range(10000)]

# Sparse
tfidf = TfidfVectorizer(ngram_range=(1,2), max_features=200000)
X = tfidf.fit_transform(docs)

# Dense
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
E = np.array(model.encode(docs, normalize_embeddings=True), dtype="float32")
d = E.shape[1]
index = faiss.IndexFlatIP(d)
index.add(E)

def hybrid_search(q, k=5, alpha=0.5):
    # alpha weights dense vs sparse (0..1)
    q_dense = np.array(model.encode([q], normalize_embeddings=True), dtype="float32")
    D, I = index.search(q_dense, k*5)
    dense_scores = dict(zip(I[0], D[0]))

    q_sparse = tfidf.transform([q])
    sp = (X @ q_sparse.T).toarray().ravel()
    sp_top = np.argpartition(-sp, k*5)[:k*5]
    sparse_scores = {i: sp[i] for i in sp_top}

    candidates = set(dense_scores) | set(sparse_scores)
    blended = [(i, alpha*dense_scores.get(i,0.0) + (1-alpha)*sparse_scores.get(i,0.0)) for i in candidates]
    blended.sort(key=lambda x: -x[1])
    return [docs[i] for i,_ in blended[:k]]

print(hybrid_search("exact phrase ripgrep trick", k=5, alpha=0.6))
PY

Add a light reranker to re-order top-20 candidates:

python3 - <<'PY'
from sentence_transformers import CrossEncoder
pairs = [("query about ripgrep", "snippet text 1"), ("query about ripgrep", "snippet text 2")]
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = reranker.predict(pairs)
print(scores)
PY

Use reranking for top-20 only; it’s expensive. If using CPU-only, consider skipping rerank for short queries or cache frequent queries.

CLI helpers matter too:

  • Use ripgrep to prefilter candidate files before embedding:
rg -n --json "RAG|retrieval" ./docs | jq -r '.data.path.text' | sort -u > hit_files.txt

5) Measure what matters and iterate

Focus on:

  • Retrieval metrics: recall@k, MRR, hit rate of gold answers in top-k.

  • End-to-end latency: p50/p95 across query patterns.

  • Cost: tokens per answer, reranker invocations per query.

Minimal FastAPI wrapper to benchmark:

cat > app.py <<'PY'
from fastapi import FastAPI, Query
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

app = FastAPI()
docs = [f"Doc {i} about Linux and RAG {i%5}" for i in range(20000)]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
E = np.array(model.encode(docs, normalize_embeddings=True), dtype="float32")
index = faiss.IndexFlatIP(E.shape[1]); index.add(E)

@app.get("/search")
def search(q: str = Query(...), k: int = 5):
    qv = np.array(model.encode([q], normalize_embeddings=True), dtype="float32")
    D,I = index.search(qv, k)
    return [{"doc": docs[i], "score": float(D[0][j])} for j,i in enumerate(I[0])]
PY

uvicorn app:app --host 0.0.0.0 --port 8000

Benchmark with ApacheBench:

  • apt/zypper: apache2-utils already installed above

  • dnf: httpd-tools already installed above

ab -n 200 -c 20 "http://127.0.0.1:8000/search?q=optimize+rag&k=5"

Watch p95 and failed requests. Iterate: adjust nprobe, chunk size, hybrid alpha, and rerank depth. Keep a simple CSV of config vs. metrics.

Optionally, add RAG evaluation with ragas to track retrieval quality over a labeled set:

python3 - <<'PY'
from ragas.metrics import faithfulness, answer_relevancy
from ragas import evaluate
from datasets import Dataset

# toy example; replace with your labeled triples
data = Dataset.from_dict({
  "question": ["How to speed up FAISS?"],
  "answer": ["Tune nlist and nprobe; normalize vectors."],
  "contexts": [["Use IndexIVFFlat with higher nprobe; normalize."]],
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy])
print(result)
PY

Real-world optimisation patterns that pay off

  • Go hybrid early: Dense-only often misses exact queries; hybrid boosts hit rate 5–15% with negligible infra.

  • Tune before scaling: A 2× nprobe bump can fix recall with a 10–30% latency hit—cheaper than new servers.

  • Chunk once, reuse everywhere: Canonical chunking + embedding cache saves hours on re-indexing.

  • Cap reranking: Rerank top-10 or top-20 only; skip for short queries; cache reranker outputs for common queries.

  • Concurrency limits: Set OMP_NUM_THREADS and batch sizes explicitly; don’t let Python spawn unbounded workers.

Conclusion and next steps

RAG performance is a loop, not a leap. Start with a sane index, chunk smartly, batch embeddings, retrieve hybrid, and measure. Small, CLI-driven changes routinely cut p95 latency in half and raise retrieval accuracy without wholesale rewrites.

Next steps:

  • Wire the FAISS + hybrid example into your pipeline.

  • Benchmark three chunk configs and two nprobe settings with ab; keep a simple results log.

  • Add a top-20 rerank guarded by a feature flag and measure the delta.

Have a RAG bottleneck you can reproduce from the shell? Send it my way—I’ll help you shave milliseconds and boost hits.

Related posts