Posted on
• Artificial Intelligence

RAG Explained for Linux Users

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG Explained for Linux Users: Make Your Shell Smarter

If you’ve ever thought, “LLMs are cool, but they don’t know my system, my logs, or my internal docs,” Retrieval-Augmented Generation (RAG) is your fix. RAG lets you feed relevant, real documents from your machine into a language model so the model answers with context you control—no more vague, hallucinated replies about your infra.

In Linux terms: think of RAG as “grep + summarize.” You retrieve the right snippets from your knowledge base, then pipe them into a model to generate an answer. It’s composable, auditable, and scriptable.

This post explains the core idea, why it matters to Linux users, and gives you a minimal, practical RAG setup you can run from Bash using common tools plus a small Python script. You’ll get actionable steps, code you can copy, and distro-specific install commands.

What is RAG, and why should Linux users care?

  • Retrieval: Search your own files (docs, wikis, man pages, logs, READMEs, code) to find the most relevant chunks for a question.

  • Augmentation: Prepend those chunks to the prompt as context.

  • Generation: Ask a language model to answer using that context.

Why it’s valid and valuable:

  • Verifiable answers: You can show the exact sources that informed the reply.

  • Up-to-date and private: Your data stays on your machine; update the index anytime.

  • Unix-friendly: It’s “do one thing well” embodied—retrieval is one tool, generation is another, and Bash glues them together.

What we’ll build

  • A lightweight, local RAG pipeline you can run from Bash: 1) Index a folder of docs into a small vector database (FAISS). 2) Query by a question to retrieve the top-k chunks. 3) Feed the chunks into a local LLM (llama.cpp) or a remote API (optional).

You can adapt this to:

  • Query internal READMEs for onboarding.

  • Summarize incident timelines from logs.

  • Pull config snippets from /etc and explain them.

Prerequisites and installation

We’ll use:

  • System tools: curl, jq, ripgrep, git, cmake, build tools

  • Python: venv, pip, sentence-transformers, faiss-cpu, numpy

  • Optional: pandoc to convert docs to text (Markdown, HTML, PDF → .txt)

  • Optional: llama.cpp for fully local generation

Install system dependencies.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git build-essential cmake curl jq ripgrep pandoc

Fedora/RHEL (dnf):

sudo dnf install -y python3 python3-virtualenv python3-pip git gcc-c++ make cmake curl jq ripgrep pandoc

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-virtualenv python3-pip git gcc-c++ make cmake curl jq ripgrep pandoc

Set up a Python virtual environment and dependencies:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install sentence-transformers faiss-cpu numpy tqdm

Optional: build llama.cpp (local LLM inference):

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j
cd ..
  • Download a small GGUF model (example):
mkdir -p models
# Example: Phi-3 mini instruct quantized (check license/size/compatibility)
wget -O models/phi3-mini-instruct.gguf \
  https://huggingface.co/bartowski/Phi-3-mini-4k-instruct-GGUF/resolve/main/Phi-3-mini-4k-instruct.Q4_0.gguf

Note: llama.cpp’s binary may be named llama-cli or main depending on version.

Step 1: Prepare a knowledge base

Pick a directory with documents you care about, e.g., ~/kb. Put Markdown, text files, logs, READMEs, and exported wiki pages there.

Convert non-text docs (optional, via pandoc):

mkdir -p ~/kb_text
find ~/kb -type f \( -name '*.md' -o -name '*.txt' -o -name '*.log' \) -exec cp {} ~/kb_text/ \;

# Convert HTML/PDF to text (basic)
find ~/kb -type f -name '*.html' -exec sh -c 'for f; do pandoc "$f" -t plain -o ~/kb_text/"$(basename "$f" .html)".txt; done' sh {} +
find ~/kb -type f -name '*.pdf' -exec sh -c 'for f; do pandoc "$f" -t plain -o ~/kb_text/"$(basename "$f" .pdf)".txt; done' sh {} +

Quick sanity check:

rg -n "nginx|kubernetes|onboarding" ~/kb_text

Step 2: Build a vector index (FAISS)

Create build_index.py in your project directory:

#!/usr/bin/env python3
import os, json, math
from pathlib import Path
import numpy as np
from tqdm import tqdm
import faiss
from sentence_transformers import SentenceTransformer

KB_DIR = Path(os.environ.get("KB_DIR", str(Path.home() / "kb_text")))
OUT_DIR = Path(os.environ.get("OUT_DIR", "index"))
OUT_DIR.mkdir(parents=True, exist_ok=True)

CHUNK_SIZE = 800    # characters
OVERLAP = 200       # characters

def chunk_text(text, size=CHUNK_SIZE, overlap=OVERLAP):
    chunks = []
    i = 0
    n = max(1, size - overlap)
    while i < len(text):
        chunks.append(text[i:i+size])
        i += n
    return chunks

def collect_files(kb_dir):
    for p in kb_dir.rglob("*"):
        if p.is_file() and p.suffix.lower() in (".txt", ".md", ".log", ".conf"):
            yield p

def main():
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    texts, metas = [], []
    for f in tqdm(list(collect_files(KB_DIR)), desc="Reading files"):
        try:
            t = f.read_text(errors="ignore")
        except Exception:
            continue
        for i, ch in enumerate(chunk_text(t)):
            if ch.strip():
                texts.append(ch.strip())
                metas.append({"path": str(f), "chunk_id": i})
    if not texts:
        print("No texts found. Set KB_DIR or add files.")
        return

    embs = model.encode(texts, batch_size=64, show_progress_bar=True, normalize_embeddings=True)
    embs = np.asarray(embs, dtype="float32")

    index = faiss.IndexFlatIP(embs.shape[1])  # cosine via inner product on normalized vectors
    index.add(embs)

    faiss.write_index(index, str(OUT_DIR / "kb.index"))
    with open(OUT_DIR / "meta.jsonl", "w", encoding="utf-8") as w:
        for m, t in zip(metas, texts):
            m2 = m.copy()
            m2["text"] = t
            w.write(json.dumps(m2, ensure_ascii=False) + "\n")
    print(f"Indexed {len(texts)} chunks from {KB_DIR} into {OUT_DIR}")

if __name__ == "__main__":
    main()

Build the index:

export KB_DIR=~/kb_text
python3 build_index.py

Step 3: Query the index

Create query.py:

#!/usr/bin/env python3
import os, json, sys
from pathlib import Path
import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

OUT_DIR = Path(os.environ.get("OUT_DIR", "index"))
TOP_K = int(os.environ.get("TOP_K", "5"))

def load_meta(path):
    metas = []
    with open(path, "r", encoding="utf-8") as r:
        for line in r:
            metas.append(json.loads(line))
    return metas

def main():
    if len(sys.argv) < 2:
        print("Usage: query.py \"your question here\"", file=sys.stderr)
        sys.exit(1)
    q = sys.argv[1]

    index = faiss.read_index(str(OUT_DIR / "kb.index"))
    metas = load_meta(OUT_DIR / "meta.jsonl")

    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    qemb = model.encode([q], normalize_embeddings=True).astype("float32")
    D, I = index.search(qemb, TOP_K)

    results = []
    for rank, idx in enumerate(I[0]):
        m = metas[idx]
        results.append({
            "rank": rank+1,
            "path": m["path"],
            "chunk_id": m["chunk_id"],
            "score": float(D[0][rank]),
            "text": m["text"]
        })
    print(json.dumps(results, ensure_ascii=False, indent=2))

if __name__ == "__main__":
    main()

Query it:

python3 query.py "How do we deploy the staging stack?"

Or for logs:

python3 query.py "Why did nginx return 502 yesterday?"

Step 4: Generate an answer (two options)

A) Local model with llama.cpp:

QUESTION="Summarize the staging deploy steps."
CTX_JSON=$(python3 query.py "$QUESTION")

# Build a prompt with sources
CONTEXT=$(echo "$CTX_JSON" | jq -r '.[] | "Source: \(.path)#\(.chunk_id)\n-----\n\(.text)\n"')

PROMPT="You are a helpful assistant. Use ONLY the context below to answer.
If the answer isn't in the context, say you don't know.

Context:
$CONTEXT

Question: $QUESTION
Answer:"

LLAMA_BIN=$(command -v llama-cli || echo "./llama.cpp/llama-cli")
if [ ! -x "$LLAMA_BIN" ]; then
  LLAMA_BIN=$(command -v ./main || echo "./llama.cpp/main")
fi

"$LLAMA_BIN" -m models/phi3-mini-instruct.gguf -p "$PROMPT" -n 512 --temp 0.2

B) Remote API (example with OpenAI-compatible endpoint; requires API key):

export OPENAI_API_KEY=sk-...
QUESTION="Summarize the staging deploy steps."
CTX_JSON=$(python3 query.py "$QUESTION")
CONTEXT=$(echo "$CTX_JSON" | jq -r '.[] | "Source: \(.path)#\(.chunk_id)\n-----\n\(.text)\n"')

PROMPT="Use ONLY the context to answer. If not present, say you don't know.

Context:
$CONTEXT

Question: $QUESTION
Answer:"

curl -s https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
  -d "$(jq -n --arg p "$PROMPT" '{model:"gpt-4o-mini", messages:[{role:"user", content:$p}], temperature:0.2, max_tokens:512}')" \
| jq -r '.choices[0].message.content'

Replace the endpoint/model to match your provider. Keep secrets out of shell history.

Real-world examples you can try today

1) Onboarding docs search

  • Index your team’s README.md, runbooks, and wiki exports.

  • Ask: “What’s the process to rotate API keys?” or “How do I run the staging smoke tests?”

  • Benefit: junior teammates self-serve answers with sources.

2) Log-aided incident review

  • Index /var/log/nginx and app logs (rotate or sample to keep size manageable).

  • Ask: “What changed around 14:00 that caused 502s?” or “Summarize auth errors last hour.”

  • Bonus: pipe in grep pre-filters to shrink noise before indexing.

3) Config explainability

  • Index /etc/nginx, /etc/systemd, Helm charts, and Kubernetes manifests.

  • Ask: “Which nginx directives control client body size?” or “Where do we set memory limits in staging?”

  • Answers cite exact files/lines so you can verify quickly.

4) Codebase Q&A (lightweight)

  • Index docs/ and comments from src/ (you can limit to .md/.txt to stay small).

  • Ask: “What’s the contract for the payment retry job?” or “Where is the feature flag for checkout set?”

4 Practical tips for better RAG results

  • Chunking matters: Use overlaps (e.g., 800 chars with 200 overlap) so facts that cross boundaries aren’t lost.

  • Use smaller, faster models first: all-MiniLM-L6-v2 for embeddings is fast; for generation, small quantized GGUF models are fine for docs/logs.

  • Show sources: Always print file paths and chunk IDs. It builds trust and speeds verification.

  • Reindex incrementally: Re-run build_index.py via cron after doc syncs or deployments.

Troubleshooting

  • faiss-cpu install issues: On rare distros/architectures, conda may be easier. As a fallback: pip install faiss-cpu==1.7.4

  • RAM usage: If your corpus is huge, chunk fewer files at first or shard the index.

  • Slow generation: Reduce max tokens (-n) in llama.cpp, try a smaller GGUF, or switch to a remote API temporarily.

  • No relevant results: Increase TOP_K or improve your chunk size/overlap; ensure you indexed the right directory.

Conclusion and Next Steps (CTA)

RAG puts your Linux mindset into AI: retrieve with precision, then generate with context you trust. You now have:

  • A working local index of your docs/logs.

  • A Bash-friendly way to query and summarize with sources.

  • Options for fully local or remote generation.

Next steps:

  • Point KB_DIR at your real team docs and reindex.

  • Wrap the query + generation into a function in your shell profile.

  • Add a cron job to refresh the index nightly.

  • Explore advanced backends (e.g., Milvus, Chroma) and metadata filters (file type, date ranges).

Have a neat use-case or hit a snag? Share your script or question—and the sources your RAG cited—so others can learn and improve with you.

Related posts