RAG Troubleshooting from the Linux Shell: Practical Steps That Actually Fix Things

Has your RAG pipeline started returning great-sounding but wrong answers—or gone from snappy to sluggish overnight? You’re not alone. Retrieval-Augmented Generation is powerful, but when it misbehaves the failure can hide in any stage: data ingestion, indexing, retrieval, context assembly, or generation. The good news: you can diagnose most issues quickly with standard Linux tools and a few small scripts.

This article shows how to troubleshoot RAG from the command line with reproducible checks, tiny Bash utilities, and a lightweight evaluation harness. You’ll learn how to confirm your index is healthy, prove what your retriever actually returns, spot latency culprits, validate prompt/context assembly, and measure retrieval quality—without heavy frameworks.

Why this matters

• RAG often fails silently. Stale or partial indexes, mismatched embedding models, chunking mistakes, or prompt truncation produce “confidently wrong” answers that look like LLM hallucinations.

• Tight loops win. Shell-based smoke tests verify each stage quickly before you dig into application code.

• Portable tools. jq, ripgrep, sqlite3, curl, strace, and GNU parallel are available across Linux distros and integrate nicely with CI/CD.

Common failure modes we’ll target:

• Documents never made it into the index (or ended up in the wrong namespace/tenant).

• Retrieval returns plausible but irrelevant chunks due to chunk size, overlap, or tokenization differences.

• Context is truncated or poorly delimited, confusing the LLM.

• Latency spikes from network DNS issues or oversized payloads.

• Regression sneaks in: a new embedding model or a re-chunking job changes retrieval behavior.

Prerequisites: install a few CLI tools

We’ll use curl, jq, ripgrep (rg), sqlite3 (for quick full-text checks), strace, and GNU parallel.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq ripgrep sqlite3 parallel strace

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y curl jq ripgrep sqlite strace parallel
# Note: on some RHEL/CentOS versions ripgrep may require EPEL.

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq ripgrep sqlite3 strace
# GNU parallel may be packaged as 'parallel' or 'gnu_parallel':
sudo zypper install -y parallel || sudo zypper install -y gnu_parallel

Verify:

curl --version
jq --version
rg --version
sqlite3 --version
strace -V
parallel --version || true

1) Is your corpus actually retrievable?

Before blaming embeddings or the LLM, verify your source text is present and queryable. Two quick checks:

A) Coverage audit with ripgrep

• Confirm key terms and doc IDs exist in the raw corpus directory (e.g., ./data/docs):

export CORPUS=./data/docs
rg -n --stats 'warranty|SLA|refund' "$CORPUS"
rg -n --stats 'DOC-12345' "$CORPUS"

• Red flags:
  • Near-zero hits for critical terms you expect everywhere.
  • IDs present in files but missing later from retrieval results.

B) Scratch full-text search with SQLite FTS

• Useful to validate that keyword retrieval (baseline) can find a known doc quickly.

Create a tiny FTS index from plain text files:

rm -f scratch.db
sqlite3 scratch.db <<'SQL'
PRAGMA journal_mode=WAL;
CREATE VIRTUAL TABLE docs USING fts5(path, body, tokenize = 'porter');
SQL

# Ingest files (one row per file)
find "$CORPUS" -type f -print0 | while IFS= read -r -d '' f; do
  sqlite3 scratch.db "INSERT INTO docs (path, body) VALUES ('$(printf "%q" "$f")', readfile('$f'))";
done

# Query it
sqlite3 scratch.db "SELECT path, snippet(docs, 1, '[', ']', '…', 10) FROM docs WHERE docs MATCH 'refund NEAR/5 policy' LIMIT 5;"

If FTS can reliably surface key documents but your vector retriever can’t, you likely have:

• Broken ingestion/indexing in the vector store.

• Mismatch of tenant/namespace, or filters unintentionally excluding docs.

• Embedding or chunking settings that obscure important terms.

2) Prove what your retriever actually returns

Create a portable smoke test that hits your retriever directly. Replace $RETRIEVER_URL with your service.

Example API expectation:

• POST $RETRIEVER_URL/search with JSON {"query": "...", "k": 5}.

• Response: {"docs":[{"id":"...","score":0.87,"text":"..."}, ...]}.

cat > retrieval_smoke_test.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

: "${RETRIEVER_URL:?Set RETRIEVER_URL env var, e.g. https://retriever.local/search}"

query="${1:-"What is the warranty period for product X?"}"
k="${2:-5}"

curl -sS -X POST "$RETRIEVER_URL" \
  -H 'Content-Type: application/json' \
  -d "{\"query\": \"$query\", \"k\": $k}" \
| jq -r '
  .docs[]
  | {id, score, preview: (.text | gsub("\n"; " ") | .[0:160])}
  | "\(.score)\t\(.id)\t\(.preview)"
'
EOF
chmod +x retrieval_smoke_test.sh

Run:

RETRIEVER_URL="https://retriever.local/search" ./retrieval_smoke_test.sh "refund policy"

What to look for:

• Are expected document IDs present? If not, suspect namespace/filters or stale index.

• Are scores extremely compressed (e.g., all ~0.50)? That suggests normalization or embedding mismatch.

• Do previews look like metadata headers only? Chunking might be swallowing bodies or you’re indexing titles without content.

Tip: Test known “gold” queries where you know the right doc ID. Keep a small list you can run on every deployment.

3) Find and fix latency at each stage

Latency often hides in network calls, DNS, or oversized payloads. Measure each hop from the shell.

A) Measure HTTP timings

curl -sS -o /dev/null -w 'dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' \
  "$RETRIEVER_URL/health"

• High time_namelookup: DNS issue or misconfigured resolver inside containers.

• Big ttfb: server-side processing slow (embedding, DB, or I/O).

• total >> ttfb: large response body or slow network.

B) End-to-end tracing with shell timing

time ./retrieval_smoke_test.sh "SLA uptime"

C) Spot network stalls with strace (dev or staging only)

strace -cf -e trace=network -o net.trace \
  curl -sS "$RETRIEVER_URL/health" > /dev/null
cat net.trace

D) Check concurrency limits Run multiple queries concurrently to see head-of-line blocking:

seq 1 20 | parallel -j 10 --bar \
  './retrieval_smoke_test.sh "refund policy" >/dev/null'

Red flags:

• Large, variable DNS times across runs.

• Response bodies in MBs when k is small (send less text or compress).

• One slow upstream call (vector DB or embedding API) dominating traces.

4) Validate prompt and context assembly

Even perfect retrieval fails if context is malformed or truncated. Validate how your app composes the prompt.

A) Create a local prompt composer Assume you’ve saved retrieved chunks as files; this script joins them with clear separators and measures size.

cat > compose_prompt.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
system_msg="You are a concise support assistant."
user_q="${1:-"What is the warranty period for product X?"}"

shift || true
chunks=( "$@" )

{
  echo "<system>"
  echo "$system_msg"
  echo "</system>"
  echo
  echo "<user>"
  echo "$user_q"
  echo "</user>"
  echo
  echo "<context>"
  for f in "${chunks[@]}"; do
    echo "----- BEGIN CHUNK: $f -----"
    sed -E 's/[[:cntrl:]]//g' "$f" | head -c 4000
    echo
    echo "----- END CHUNK: $f -----"
  done
  echo "</context>"
} | tee composed.txt >/dev/null

echo "Bytes: $(wc -c < composed.txt)"
echo "Lines: $(wc -l < composed.txt)"
EOF
chmod +x compose_prompt.sh

Run:

./compose_prompt.sh "refund policy" ./retrieved/chunk-1.txt ./retrieved/chunk-2.txt

What to check:

• Clear delimiters around chunks, and presence of doc IDs/metadata.

• Size within your model/input budget. If not, reduce k, compress text, or improve chunk selection.

• No control characters or binary junk that can confuse the model.

B) Diff app vs. local composer Dump your app’s final prompt to a file and compare:

diff -u app_prompt.txt composed.txt || true

If they differ a lot, fix your prompt building logic (separators, missing metadata, truncation).

5) Build a tiny retrieval evaluation harness (recall@k)

A small gold set prevents regressions. We’ll compute a simple recall@k from the shell.

A) Create queries.jsonl with expected doc IDs

cat > queries.jsonl <<'EOF'
{"query":"What is the warranty period for product X?","expected":["DOC-12345"]}
{"query":"How do I request a refund?","expected":["DOC-20001","DOC-20002"]}
{"query":"What is the SLA for enterprise plan?","expected":["SLA-ENT-01"]}
EOF

B) Write an evaluator that calls your retriever and scores recall@k

cat > eval_recall.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
: "${RETRIEVER_URL:?Set RETRIEVER_URL env var}"

k="${1:-5}"
tp=0
total=0

while IFS= read -r line; do
  q=$(jq -r '.query' <<<"$line")
  exp=$(jq -r '.expected[]' <<<"$line" | sort | tr '\n' ' ')
  got=$(curl -sS -X POST "$RETRIEVER_URL" -H 'Content-Type: application/json' -d "{\"query\": \"$q\", \"k\": $k}" \
    | jq -r '.docs[].id' | sort | tr '\n' ' ')

  # Compute intersection size
  hit=0
  for e in $exp; do
    if grep -qw "$e" <<<"$got"; then hit=$((hit+1)); fi
  done

  echo -e "Q: $q\nExpected: $exp\nGot: $got\nHits: $hit\n---"
  if [ "$hit" -gt 0 ]; then tp=$((tp+1)); fi
  total=$((total+1))
done < queries.jsonl

echo "Recall@${k} (query-level): $tp/$total = $(awk -v a="$tp" -v b="$total" 'BEGIN{printf("%.2f%%", (a/b)*100)}')"
EOF
chmod +x eval_recall.sh

Run:

RETRIEVER_URL="https://retriever.local/search" ./eval_recall.sh 5

Use this harness in CI to catch:

• Silent index drift.

• Embedding/chunking config regressions.

• Namespace or filter changes.

Real-world fixes when things look “fine” but answers are wrong

• Mismatched embedding model: Index built with one model and queried with another yields nonsense similarity. Rebuild or align both sides.

• Chunk size/overlap: Too large hides relevant spans; too small loses context. Try 300–800 tokens with 10–20% overlap as a starting point and validate with the harness.

• Normalization differences: If you stem/normalize during indexing but not during query (or vice versa), retrieval degrades. Standardize preprocessing.

• Metadata filters: Wrong tenant/project filters can quietly exclude everything you care about.

• Context spam: Duplicated headers, navigation, or boilerplate dominate the prompt. Strip template noise during ingestion.

Conclusion and next steps

Troubleshooting RAG doesn’t need heavyweight tooling. With a handful of Linux utilities you can:

• Verify corpus coverage and retrievability.

• See exactly what your retriever returns.

• Pinpoint latency bottlenecks.

• Ensure prompt/context assembly is correct and within budget.

• Guard against regressions with a simple recall@k harness.

Call to action:

• Add retrieval_smoke_test.sh and eval_recall.sh to your repo and CI.

• Keep a small queries.jsonl gold set updated whenever docs or configs change.

• Schedule a weekly scratch FTS check to catch ingestion drift early.

• If you spot systematic retrieval misses, iterate chunking and filters before touching the LLM.

Got a particularly tricky RAG failure? Turn it into a gold query and let your shell tests keep it fixed for good.