RAG Monitoring with Bash: Turn Your LLM Pipeline from Guesswork into Observability

If you’ve shipped a Retrieval-Augmented Generation (RAG) app, you’ve probably heard the same two complaints: “It’s slow,” and “It made something up.” RAG is powerful, but it’s a system of moving parts—embeddings, vector search, chunking, caching, generation—that can silently drift or degrade.

Good news: you don’t need a full-blown observability platform to get real, actionable insight. With a few Bash tools, you can monitor the health of your RAG pipeline, catch regressions early, and drive down both latency and hallucinations.

This post shows you how to do RAG monitoring using standard Linux tools, with concrete scripts you can drop into production. We’ll define the right metrics, structure logs, compute rolling stats, export to Prometheus, and even send alerts—all from Bash.

Why monitor RAG at all?

• Retrieval dominates answer quality. If the retriever misses the right passages, even the best model will hallucinate.

• Silent regressions happen. Re-embedding, index compaction, or a vendor model update can subtly degrade quality without obvious errors.

• Token costs creep. Quiet increases in context size or output length spike your bill.

• Latency isn’t just “model time.” Retrieval, reranking, tools, and generation each contribute—know where the time goes.

• SLOs for AI apps are different. You must measure groundedness, retrieval hit rate, and context utilization—not just uptime.

Install the essentials

We’ll use common CLI tools: jq, curl, bc, mailx (or mailutils), and optionally gnuplot for quick visuals. Pick the commands for your distro.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jq curl bc gawk gnuplot-nox mailutils python3 python3-venv sqlite3

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y jq curl bc gawk gnuplot mailx python3 python3-virtualenv sqlite

• openSUSE (zypper):

sudo zypper install -y jq curl bc gawk gnuplot mailx python3 python3-virtualenv sqlite3

Optional: if you want to export metrics to Prometheus via the node exporter’s textfile collector:

• Debian/Ubuntu:

sudo apt install -y prometheus-node-exporter

• Fedora/RHEL/CentOS:

sudo dnf install -y node_exporter || sudo dnf install -y golang-github-prometheus-node_exporter

• openSUSE:

sudo zypper install -y prometheus-node_exporter || sudo zypper install -y golang-github-prometheus-node_exporter

Note: package names may vary slightly by distro/version; the alternatives above cover common cases.

Step 1 — Decide what to measure

Here are pragmatic metrics that map directly to user experience and cost:

• Retrieval overlap rate (proxy for recall@k): fraction of cited references that were actually in the retrieved top-k.

• Groundedness/attribution rate: how often the answer cites at least one retrieved source.

• Latency breakdown: retrieval_ms, generation_ms, total_ms; plus p50/p95.

• Error rate: proportion of requests with errors or 5xx status.

• Token cost: rolling sum of (input_tokens, output_tokens) multiplied by your pricing.

• Cache hit rate: if you cache retrieval or generations, measure hit ratio.

• Index freshness: version/timestamp drift of embeddings vs. content.

You don’t need all of these on day one; start with retrieval overlap, latency, and error rate.

Step 2 — Log every RAG turn as JSONL

Output one JSON object per request (JSON Lines). Keep it flat and consistent. Example event:

{"ts":"2026-07-12T10:45:21Z",
 "query":"What ports does Postgres use?",
 "retrieved_docs":[{"doc_id":"net-ports-01","score":0.82},{"doc_id":"pg-admin-07","score":0.74}],
 "references":[{"doc_id":"net-ports-01"}],
 "generation":{"model":"my-llm","tokens_in":512,"tokens_out":86},
 "latency_ms":{"retrieval":92,"generation":780,"total":896},
 "status":200,
 "cache_hit":false,
 "answer":"PostgreSQL listens on TCP port 5432 by default..."
}

Tips:

• Always log doc_ids in both retrieved_docs and references so you can compute overlap.

• Log tokens_in/tokens_out to track cost and context bloat.

• Include status and an error field if something fails.

Append from your app or wrapper:

printf '%s\n' "$json_event" >> /var/log/rag/events.jsonl

Step 3 — Turn logs into metrics with Bash

The script below ingests the last N lines of your JSONL log and computes:

• latency p50/p95

• error rate

• retrieval-reference overlap rate

• token cost sum (uses IN_COST and OUT_COST env vars: dollars per 1K tokens)

• cache hit rate

It can also export Prometheus metrics via the node_exporter textfile collector.

Save as rag-metrics.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="${LOG_FILE:-/var/log/rag/events.jsonl}"
WINDOW="${WINDOW:-1000}"                       # how many recent events to sample
TEXTFILE_DIR="${TEXTFILE_DIR:-}"               # e.g. /var/lib/node_exporter/textfile_collector
IN_COST_PER_1K="${IN_COST_PER_1K:-0}"          # dollars per 1K input tokens
OUT_COST_PER_1K="${OUT_COST_PER_1K:-0}"        # dollars per 1K output tokens

if [[ ! -f "$LOG_FILE" ]]; then
  echo "No log file at $LOG_FILE" >&2
  exit 1
fi

tail_n() { tail -n "$WINDOW" "$LOG_FILE"; }

# Latency quantiles
sorted_total="$(tail_n | jq -r '.latency_ms.total // empty' | sort -n)"
count_total=$(wc -l <<< "$sorted_total")
p50_idx=$(( count_total==0 ? 0 : (count_total+1)/2 ))
p95_idx=$(( count_total==0 ? 0 : (95*count_total+99)/100 ))  # ceil(0.95*N)
p50_ms=$(awk -v i="$p50_idx" 'NR==i{print;exit}' <<< "$sorted_total")
p95_ms=$(awk -v i="$p95_idx" 'NR==i{print;exit}' <<< "$sorted_total")
p50_ms="${p50_ms:-0}"; p95_ms="${p95_ms:-0}"

# Error rate (errors or HTTP 5xx)
err_rate=$(tail_n | jq -r 'if ((.error != null) or ((.status // 200) >= 500)) then 1 else 0 end' \
  | awk '{s+=$1} END{if(NR==0) print 0; else printf "%.4f", s/NR}')

# Cache hit rate
cache_rate=$(tail_n | jq -r 'if (.cache_hit==true) then 1 else 0 end' \
  | awk '{s+=$1} END{if(NR==0) print 0; else printf "%.4f", s/NR}')

# Retrieval-reference overlap (proxy for recall@k on cited sources)
overlap=$(tail_n \
  | jq -r '(.retrieved_docs // [] | map(.doc_id|tostring) | unique) as $R
           | (.references    // [] | map(.doc_id|tostring) | unique) as $F
           | [ [ $F[]? | select(. as $x | $R | index($x)) ] | length,
               ($F | length) ] | @tsv' \
  | awk '{hit+=$1; total+=$2} END{ if(total==0) print 0; else printf "%.4f", hit/total }')
overlap="${overlap:-0}"

# Token cost over window
cost=$(tail_n | jq -r --argjson IN "$IN_COST_PER_1K" --argjson OUT "$OUT_COST_PER_1K" '
  (((.generation.tokens_in  // 0) * $IN  / 1000) +
   ((.generation.tokens_out // 0) * $OUT / 1000))' \
  | awk '{s+=$1} END{ printf "%.6f", s }')
cost="${cost:-0}"

# Output
ts=$(date -Iseconds)
echo "ts=$ts window=$WINDOW p50_ms=$p50_ms p95_ms=$p95_ms err_rate=$err_rate cache_rate=$cache_rate overlap_rate=$overlap window_cost_usd=$cost"

# Optional: Prometheus textfile exporter
if [[ -n "$TEXTFILE_DIR" ]]; then
  tmp="$TEXTFILE_DIR/rag.prom.$$"
  {
    echo "# HELP rag_latency_p50_ms Median end-to-end latency over recent window"
    echo "# TYPE rag_latency_p50_ms gauge"
    echo "rag_latency_p50_ms $p50_ms"
    echo "# HELP rag_latency_p95_ms 95th percentile end-to-end latency over recent window"
    echo "# TYPE rag_latency_p95_ms gauge"
    echo "rag_latency_p95_ms $p95_ms"
    echo "# HELP rag_error_rate Error rate over recent window"
    echo "# TYPE rag_error_rate gauge"
    echo "rag_error_rate $err_rate"
    echo "# HELP rag_cache_hit_rate Cache hit ratio over recent window"
    echo "# TYPE rag_cache_hit_rate gauge"
    echo "rag_cache_hit_rate $cache_rate"
    echo "# HELP rag_retrieval_overlap_rate Fraction of cited refs found in retrieved_docs"
    echo "# TYPE rag_retrieval_overlap_rate gauge"
    echo "rag_retrieval_overlap_rate $overlap"
    echo "# HELP rag_window_cost_usd Total token cost over recent window"
    echo "# TYPE rag_window_cost_usd gauge"
    echo "rag_window_cost_usd $cost"
  } > "$tmp"
  mv "$tmp" "$TEXTFILE_DIR/rag.prom"
fi

Example usage:

chmod +x rag-metrics.sh
LOG_FILE=/var/log/rag/events.jsonl WINDOW=2000 IN_COST_PER_1K=0.005 OUT_COST_PER_1K=0.015 ./rag-metrics.sh

To expose via Prometheus node exporter’s textfile collector:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown -R "$USER":"$USER" /var/lib/node_exporter/textfile_collector
TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector LOG_FILE=/var/log/rag/events.jsonl ./rag-metrics.sh

Point Prometheus at your node_exporter and scrape the textfile metrics.

Step 4 — Automate and alert

Run the metric job on a schedule and alert when thresholds are breached.

• Cron (simple):

*/1 * * * * LOG_FILE=/var/log/rag/events.jsonl WINDOW=2000 IN_COST_PER_1K=0.005 OUT_COST_PER_1K=0.015 TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector /opt/rag/rag-metrics.sh >/var/log/rag/metrics.last 2>&1

• systemd timer (more robust):

Service unit /etc/systemd/system/rag-metrics.service

[Unit]
Description=RAG metrics exporter

[Service]
Type=oneshot
Environment=LOG_FILE=/var/log/rag/events.jsonl
Environment=WINDOW=2000
Environment=IN_COST_PER_1K=0.005
Environment=OUT_COST_PER_1K=0.015
Environment=TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
ExecStart=/opt/rag/rag-metrics.sh

Timer /etc/systemd/system/rag-metrics.timer

[Unit]
Description=Run RAG metrics every minute

[Timer]
OnCalendar=*:0/1
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now rag-metrics.timer

• Minimal Slack and email alerts:

Set a threshold-check wrapper rag-alert.sh:

#!/usr/bin/env bash
set -euo pipefail
OUT=$(LOG_FILE=/var/log/rag/events.jsonl WINDOW=2000 IN_COST_PER_1K=0.005 OUT_COST_PER_1K=0.015 /opt/rag/rag-metrics.sh)
echo "$OUT"

# Parse fields
p95=$(awk -F' ' '{for(i=1;i<=NF;i++) if($i~"^p95_ms="){split($i,a,"="); print a[2]}}' <<<"$OUT")
err=$(awk -F' ' '{for(i=1;i<=NF;i++) if($i~"^err_rate="){split($i,a,"="); print a[2]}}' <<<"$OUT")
over=$(awk -F' ' '{for(i=1;i<=NF;i++) if($i~"^overlap_rate="){split($i,a,"="); print a[2]}}' <<<"$OUT")

alert=""
(( ${p95%.*} > 1500 )) && alert+="High latency p95=${p95}ms\n"
awk -v e="$err" 'BEGIN{if(e>0.05) exit 0; else exit 1}' && alert+="Error rate >5% (err_rate=$err)\n" || true
awk -v o="$over" 'BEGIN{if(o<0.5) exit 0; else exit 1}' && alert+="Retrieval overlap <50% (overlap_rate=$over)\n" || true

if [[ -n "$alert" ]]; then
  echo -e "$alert"
  # Slack (set SLACK_WEBHOOK_URL env)
  if [[ -n "${SLACK_WEBHOOK_URL:-}" ]]; then
    curl -s -X POST -H 'Content-type: application/json' --data "{\"text\":\"RAG Alert:\\n$alert\"}" "$SLACK_WEBHOOK_URL" >/dev/null
  fi
  # Email (ensure mail/mailx is installed)
  if command -v mail >/dev/null 2>&1; then
    echo -e "$alert" | mail -s "RAG Alert" ops@example.com || true
  elif command -v mailx >/dev/null 2>&1; then
    echo -e "$alert" | mailx -s "RAG Alert" ops@example.com || true
  fi
fi

Install email tools if you need them:

• Debian/Ubuntu (mail command):

sudo apt install -y mailutils

• Fedora/RHEL/CentOS (mailx command):

sudo dnf install -y mailx

• openSUSE (mailx provided by s-nail):

sudo zypper install -y s-nail

Schedule rag-alert.sh in cron or as a systemd timer.

Step 5 — Quick visualization (optional)

For a fast local graph, dump latency to CSV and plot with gnuplot:

Export:

tail -n 5000 /var/log/rag/events.jsonl | jq -r '[.ts, .latency_ms.total] | @csv' > /tmp/rag_latency.csv

Plot:

gnuplot -persist <<'GP'
set datafile separator ","
set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%H:%M"
set title "RAG total latency"
set ylabel "ms"
plot "/tmp/rag_latency.csv" using 1:2 with lines title "total_ms"
GP

A real-world pattern to watch for

If you suddenly see:

• overlap_rate dropping,

• p95_ms climbing,

• and window_cost_usd increasing,

you likely have context bloat and retrieval drift. Common causes:

• re-embedded docs with a different model but old index still in use,

• chunking changes without re-embedding,

• k increased “to be safe,” inflating context but not quality.

Use the overlap metric to validate retrieval actually surfaces cited sources. Then tune k, chunk size/overlap, or re-embed and rebuild the index.

Conclusion and next steps

RAG monitoring is not optional—it’s how you keep answers grounded, latency predictable, and costs in check. With a small set of Bash tools and structured logs, you can:

• Track retrieval-reference overlap (a groundedness proxy)

• Watch latency p50/p95 and error rate

• Control token cost growth

• Alert before users complain

Your next step: 1) Start logging JSONL exactly like the example. 2) Drop rag-metrics.sh into your host, set WINDOW and costs, and run it. 3) Wire it into Prometheus via the textfile collector or enable Slack/email alerts with rag-alert.sh. 4) Iterate on thresholds as you learn your app’s normal.

If you want a follow-up post with a full Bash-based evaluation harness (label sets, recall@k, answer grading with a local model like Ollama), let me know and I’ll publish it.