Posted on
• Artificial Intelligence

Future of RAG

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

The Future of RAG (Retrieval-Augmented Generation): Linux-first Workflows You Can Run Today

If you’ve ever watched an LLM answer confidently but incorrectly, you already know why Retrieval-Augmented Generation (RAG) matters. RAG grounds a model’s output in your actual data—docs, logs, wikis—so it answers with citations instead of guesses. The “future of RAG” isn’t just bigger models; it’s better retrieval, fresher indexes, and private, local-first stacks you can run from Bash on your own Linux machine.

This post explains why RAG is only getting more important, then walks you through 3–5 practical steps to build and improve a local Linux RAG pipeline. You’ll get ready-to-run commands for apt, dnf, and zypper where relevant, plus short scripts you can adapt for your own environment.

Why RAG is the right bet

  • Trust and provenance: RAG lets you see exactly where an answer came from. In regulated or high-stakes environments, this matters more than raw fluency.

  • Cost and latency: You can pair small or quantized local models with fast vector search—often beating API-based giant models on cost and response time for domain-specific Q&A.

  • Privacy by default: Keep data on your Linux box, not on someone else’s GPU. Local RAG is a natural fit for teams with sensitive docs.

  • Continuous learning without retraining: Update your index, not your model. It’s faster, safer, and doesn’t risk forgetting old knowledge.

What’s changing next

  • From keyword to meaning to structure: RAG is moving beyond simple vector search to hybrid, reranked, and structured retrieval (graphs, SQL, events).

  • Multi-hop and tools: Pipelines now chain retrieval steps, call tools, and synthesize across many documents rather than a single top hit.

  • Observability and evals: Teams treat RAG like a production system—measuring faithfulness, grounding, and coverage with automatic checks.

  • Local-first by default: Thanks to efficient embeddings and quantized LLMs, you can do all of the above on a laptop or small server.

1) Local, private RAG stack in ~10 minutes

This section sets up a small, fully local RAG you can control from Bash:

  • Embeddings: fast, CPU-friendly

  • Vector search: FAISS

  • Model: local via Ollama

  • Glue: a couple of Python scripts

Install system prerequisites

Use your distro’s package manager to install common build and CLI tools.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-pip git curl jq inotify-tools build-essential cmake
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-pip git curl jq inotify-tools gcc-c++ make cmake
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y python3 python3-pip git curl jq inotify-tools gcc-c++ make cmake

Install a local model server (Ollama)

Ollama runs open models locally and exposes a simple CLI and REST API.

curl -fsSL https://ollama.com/install.sh | sh

Pull a model (change the model name if you prefer a different one):

ollama pull llama3

Ensure the service is running (usually auto-starts after first use):

systemctl --user start ollama || true

Test it:

echo "Say hello in one sentence." | ollama run llama3

Create a Python env and install libraries

We’ll use fastembed (ONNX-based, light), FAISS for vector search, and requests for HTTP calls to Ollama.

python3 -m pip install --user virtualenv
python3 -m virtualenv .venv
. .venv/bin/activate
pip install fastembed faiss-cpu requests

Script 1: Build your local index

Save as rag_build.py:

#!/usr/bin/env python3
import argparse, os, json, faiss, numpy as np
from fastembed import TextEmbedding

def iter_docs(root, exts=(".md", ".txt", ".rst", ".log", ".mdx")):
    for base, _, files in os.walk(root):
        for f in files:
            if f.lower().endswith(exts):
                path = os.path.join(base, f)
                try:
                    with open(path, "r", encoding="utf-8", errors="ignore") as fh:
                        text = fh.read()
                    if text.strip():
                        yield path, text
                except Exception:
                    pass

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--data", required=True, help="Directory of text/markdown docs")
    ap.add_argument("--out", default="index", help="Output dir for index + metadata")
    ap.add_argument("--model", default="BAAI/bge-small-en-v1.5", help="FastEmbed model")
    args = ap.parse_args()

    os.makedirs(args.out, exist_ok=True)
    docs, paths = [], []
    for path, text in iter_docs(args.data):
        paths.append(path)
        docs.append(text)

    if not docs:
        print("No documents found.")
        return

    embedder = TextEmbedding(model_name=args.model)
    vectors = np.array([v for v in embedder.embed(docs)], dtype="float32")

    # Normalize to use inner product as cosine similarity
    faiss.normalize_L2(vectors)

    index = faiss.IndexFlatIP(vectors.shape[1])
    index.add(vectors)

    faiss.write_index(index, os.path.join(args.out, "index.faiss"))
    with open(os.path.join(args.out, "meta.jsonl"), "w", encoding="utf-8") as out:
        for p, t in zip(paths, docs):
            out.write(json.dumps({"path": p, "text": t[:2000]}) + "\n")

    with open(os.path.join(args.out, "config.json"), "w", encoding="utf-8") as cf:
        json.dump({"model": args.model, "dim": vectors.shape[1]}, cf)

    print(f"Indexed {len(docs)} docs into {args.out}")

if __name__ == "__main__":
    main()

Build an index from a docs folder:

. .venv/bin/activate
python rag_build.py --data ./docs --out ./index

Script 2: Query with RAG + Ollama

Save as rag_query.py:

#!/usr/bin/env python3
import argparse, os, json, faiss, numpy as np, requests

from fastembed import TextEmbedding

def load_meta(meta_path):
    meta = []
    with open(meta_path, "r", encoding="utf-8") as fh:
        for line in fh:
            meta.append(json.loads(line))
    return meta

def format_context(hits):
    blocks = []
    for i, h in enumerate(hits, 1):
        blocks.append(f"[{i}] {h['path']}\n{h['text']}\n")
    return "\n---\n".join(blocks)

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--index", default="index", help="Index directory")
    ap.add_argument("--q", required=True, help="User question")
    ap.add_argument("--k", type=int, default=5, help="Top-k passages")
    ap.add_argument("--model", default="llama3", help="Ollama model name")
    ap.add_argument("--ollama-url", default="http://127.0.0.1:11434/api/generate")
    args = ap.parse_args()

    cfg = json.load(open(os.path.join(args.index, "config.json"), "r", encoding="utf-8"))
    meta = load_meta(os.path.join(args.index, "meta.jsonl"))

    index = faiss.read_index(os.path.join(args.index, "index.faiss"))
    embedder = TextEmbedding(model_name=cfg["model"])

    qv = np.array([v for v in embedder.embed([args.q])], dtype="float32")
    faiss.normalize_L2(qv)
    sims, ids = index.search(qv, args.k)
    ids = ids[0]
    hits = [meta[i] for i in ids if i >= 0]

    context = format_context(hits)
    prompt = f"""You are a helpful assistant. Use ONLY the context below to answer. Cite sources as [n].

Context:
{context}

Question: {args.q}
Answer:"""

    r = requests.post(args.ollama-url, json={"model": args.model, "prompt": prompt, "stream": False})
    r.raise_for_status()
    data = r.json()
    print(data.get("response", "").strip())

if __name__ == "__main__":
    main()

Ask a question grounded in your docs:

. .venv/bin/activate
python rag_query.py --index ./index --q "What do we say about installation on Linux?"

That’s a complete local RAG: no cloud, citations included. Swap in any Ollama model you like.

2) Make retrieval better than “just vectors”

Even a small pipeline becomes powerful when retrieval quality improves. Three quick wins:

  • Smarter chunking
    • Problem: giant documents get embedded as huge blobs or split arbitrarily.
    • Fix: chunk by semantic or structural boundaries (headings, paragraphs) and keep chunks in the 300–1000 token range depending on your model and domain.
    • Drop-in approach: pre-chunk files on read with simple rules.

Example tweak inside rag_build.py to chunk by paragraphs:

def chunk_text(text, max_chars=1200):
    parts, buf = [], []
    for line in text.splitlines():
        if not line.strip():  # paragraph separator
            if buf:
                parts.append("\n".join(buf))
                buf = []
        else:
            buf.append(line)
        if sum(len(x) for x in buf) > max_chars:
            parts.append("\n".join(buf))
            buf = []
    if buf:
        parts.append("\n".join(buf))
    return [p for p in parts if p.strip()]

Use chunk_text() when building docs:

for path, text in iter_docs(args.data):
    for chunk in chunk_text(text):
        paths.append(path)
        docs.append(chunk)
  • Hybrid retrieval (semantic + keyword)
    • Semantic search excels at meaning; BM25 excels at exact terms, numbers, and codes.
    • Combine both and merge rankings. It’s straightforward to add BM25 via rank-bm25:

Install:

pip install rank-bm25

Sketch (store tokenized docs, score both ways, then blend scores):

from rank_bm25 import BM25Okapi

# build time
tok_docs = [d.split() for d in docs]
bm25 = BM25Okapi(tok_docs)

# query time
q_tokens = args.q.split()
bm25_scores = bm25.get_scores(q_tokens)
# Normalize and add to vector similarity to re-rank
  • Lightweight reranking
    • After retrieving top-20 by vector/hybrid, use a tiny cross-encoder reranker to reorder the final top-5. If you want to stay light on dependencies, start with heuristic reranking: prefer chunks that contain rare query terms or exact matches.

3) Keep your knowledge fresh with a file watcher

RAG that’s stale won’t be trusted. Rebuild or incrementally update the index when docs change. The inotify-tools package lets you watch files and trigger an action.

Create watch_and_reindex.sh:

#!/usr/bin/env bash
set -euo pipefail

DOCS_DIR="${1:-./docs}"
INDEX_DIR="${2:-./index}"
VENV="${3:-.venv}"

echo "Watching $DOCS_DIR for changes..."
while inotifywait -r -e close_write,move,create,delete "$DOCS_DIR"; do
  echo "[watch] change detected, rebuilding..."
  . "$VENV/bin/activate"
  python rag_build.py --data "$DOCS_DIR" --out "$INDEX_DIR" || echo "[watch] rebuild failed"
  echo "[watch] rebuild complete"
done

Run it:

chmod +x watch_and_reindex.sh
./watch_and_reindex.sh ./docs ./index .venv

Now edits to your docs auto-refresh the index.

4) Add basic evaluation so you can trust changes

As you tweak chunking, hybrid scoring, or models, track whether answers are getting better. Start with a small set of question–answer pairs and use an automated judge.

Install:

pip install ragas datasets

Minimal example:

from ragas.metrics import faithfulness, answer_relevancy
from ragas import evaluate
from datasets import Dataset

data = Dataset.from_dict({
    "question": ["What’s our Linux install command?"],
    "answer":   ["Use apt or dnf or zypper as shown ..."],
    "contexts": [["[1] docs/install.md ..."]],
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy])
print(result)

This won’t replace human review, but it will catch regressions when you change retrieval or prompt templates.

5) A note on performance and model choices

  • CPU-friendly embeddings: fastembed with a small BGE model is fast and light. Great default.

  • Quantized local LLMs: With Ollama, try several models to see which balances speed and quality on your hardware. Smaller instruction-tuned models can be very effective with good retrieval.

  • Context windows: Favor models with larger context if you plan to stuff many chunks—just remember that better retrieval often beats dumping everything into the prompt.

Real-world mini demo

  • Point ./docs to:

    • your project’s README and docs
    • ops runbooks in Markdown
    • internal how-to notes

  • Build the index:

python rag_build.py --data ./docs --out ./index
  • Ask specific, grounded questions:
python rag_query.py --index ./index --q "How do I install prerequisites on Fedora?"

Expect an answer that cites the exact doc files it used.

Conclusion and next steps

RAG is moving from a research pattern to everyday infrastructure: local-first, observable, and tuned to your data. On Linux, you can stand it up quickly, keep it private, and iterate fast with Bash-friendly tooling.

Your next steps:

  • Implement the quickstart stack above and point it at your docs.

  • Improve retrieval with chunking and (optionally) hybrid search.

  • Add the watcher to keep indexes fresh.

  • Track quality with a tiny eval set and automated checks.

Have a favorite Linux trick or RAG tweak? Try it in this pipeline and measure the impact. If you want a follow-up post with GPU acceleration, hybrid search code, or Dockerized vector databases, let me know what you’d like to see next.

Related posts