Posted on
• Artificial Intelligence

RAG Pipelines on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG Pipelines on Linux: Turn Your Docs Into a Private, Searchable AI

Ever wished your shell could answer questions about your own docs, code, or logs—without sending data to the cloud? Retrieval-Augmented Generation (RAG) makes that possible. With a simple Linux-first workflow, you can build a local Q&A “brain” powered by your documents and a local LLM. It’s private, fast, automatable, and tailor-made for the Bash crowd.

In this guide you’ll:

  • Understand why RAG is worth your time on Linux.

  • Install only what you need (apt, dnf, zypper covered).

  • Build a minimal, hackable RAG pipeline with CLI and Python.

  • Run everything locally using Ollama + FAISS + fastembed.

  • Get actionable steps and real-world examples to extend it.

Why RAG on Linux?

  • Reduce hallucinations: The model retrieves grounded context from your own data before answering.

  • Privacy by default: Keep proprietary docs on your box—no SaaS round-trips.

  • Cost control: Run open models locally; no token bills.

  • DevOps-friendly: Script everything with Bash, automate re-indexing via cron/systemd, and keep artifacts under version control.

What we’ll build:

  • Ingest docs (txt, md, pdf) → chunk → embed (BAAI/bge-small-en-v1.5 via fastembed) → store in FAISS → retrieve top-k → prompt a local LLM (Ollama) → answer with sources.

Prerequisites (apt, dnf, zypper)

Install system packages for Python, virtualenv, curl, git, and PDF-to-text.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl poppler-utils build-essential cmake

Fedora/RHEL (dnf):

sudo dnf install -y python3 python3-virtualenv python3-pip git curl poppler-utils make gcc gcc-c++ cmake

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-virtualenv python3-pip git curl poppler-tools gcc gcc-c++ make cmake

Notes:

  • poppler-utils/poppler-tools gives you pdftotext for robust PDF ingestion.

  • build tools are optional but useful if wheels fall back to source.

Step 1: Install a local LLM with Ollama

Ollama runs LLMs locally and exposes a simple HTTP API.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

(Optional) Ensure the service is running:

sudo systemctl enable --now ollama
sudo systemctl status ollama

Pull a model (good balance of quality and size):

ollama pull llama3:8b

You can substitute another model later (e.g., mistral, qwen, phi3).

Security reminder: review install scripts before piping to sh in sensitive environments.

Step 2: Create a project and Python environment

mkdir -p ~/rag-linux/docs ~/rag-linux/rag_store
cd ~/rag-linux
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install fastembed faiss-cpu numpy requests tqdm

What we’re using:

  • fastembed: lightweight, fast CPU embeddings (no heavy PyTorch).

  • faiss-cpu: vector store for similarity search.

  • requests: call Ollama over HTTP.

Add your documents to ~/rag-linux/docs (txt, md, pdf). You can symlink or copy.

Step 3: Build the index (ingest → chunk → embed → FAISS)

Save as build_index.py:

#!/usr/bin/env python3
import argparse, json, os, subprocess
from pathlib import Path
from typing import List, Tuple
import numpy as np
import faiss
from tqdm import tqdm

def read_text_file(p: Path) -> str:
    return p.read_text(encoding="utf-8", errors="ignore")

def read_pdf_with_pdftotext(p: Path) -> str:
    try:
        out = subprocess.check_output(["pdftotext", "-layout", str(p), "-"], stderr=subprocess.DEVNULL)
        return out.decode("utf-8", errors="ignore")
    except Exception:
        return ""

def clean_text(s: str) -> str:
    return " ".join(s.replace("\r", " ").split())

def chunk_text(text: str, chunk_size: int = 800, overlap: int = 100) -> List[str]:
    # simple word-based chunking
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunk = " ".join(words[start:end])
        if chunk.strip():
            chunks.append(chunk)
        if end == len(words):
            break
        start = end - overlap
        if start < 0: start = 0
    return chunks

def collect_docs(docs_dir: Path) -> List[Tuple[str, str]]:
    items = []
    for p in docs_dir.rglob("*"):
        if not p.is_file():
            continue
        ext = p.suffix.lower()
        text = ""
        if ext in [".txt", ".md", ".log", ".rst"]:
            text = read_text_file(p)
        elif ext == ".pdf":
            text = read_pdf_with_pdftotext(p)
        if text.strip():
            items.append((str(p), clean_text(text)))
    return items

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--docs", default="docs", help="Directory with documents")
    parser.add_argument("--out", default="rag_store", help="Directory to store index & metadata")
    parser.add_argument("--chunk_size", type=int, default=800)
    parser.add_argument("--overlap", type=int, default=100)
    args = parser.parse_args()

    docs_dir = Path(args.docs).expanduser().resolve()
    out_dir = Path(args.out).expanduser().resolve()
    out_dir.mkdir(parents=True, exist_ok=True)

    items = collect_docs(docs_dir)
    if not items:
        print(f"No documents found in {docs_dir}")
        return

    # Chunk
    records = []
    for path, text in tqdm(items, desc="Chunking"):
        chunks = chunk_text(text, args.chunk_size, args.overlap)
        for ch in chunks:
            records.append({"text": ch, "source": path})

    print(f"Total chunks: {len(records)}")

    # Embed with fastembed
    from fastembed import TextEmbedding
    model_name = "BAAI/bge-small-en-v1.5"  # 384-dim
    embedder = TextEmbedding(model_name=model_name)
    texts = [r["text"] for r in records]

    # Fastembed returns a generator; collect into an array
    vecs = []
    for emb in tqdm(embedder.embed(texts), total=len(texts), desc="Embedding"):
        vecs.append(np.array(emb, dtype="float32"))
    X = np.vstack(vecs)

    # Normalize for cosine similarity (use inner product index)
    faiss.normalize_L2(X)
    dim = X.shape[1]
    index = faiss.IndexFlatIP(dim)
    index.add(X)
    faiss.write_index(index, str(out_dir / "index.faiss"))

    # Save metadata
    with open(out_dir / "meta.json", "w", encoding="utf-8") as f:
        json.dump({"records": records, "model": model_name}, f, ensure_ascii=False)

    print(f"Index written to {out_dir}")
    print("Done.")

if __name__ == "__main__":
    main()

Build the index:

python3 build_index.py --docs docs --out rag_store

Step 4: Query with a local LLM (Ollama API)

Save as ask.py:

#!/usr/bin/env python3
import argparse, json
from pathlib import Path
import numpy as np
import faiss
import requests

def load_store(store_dir: Path):
    index = faiss.read_index(str(store_dir / "index.faiss"))
    meta = json.loads((store_dir / "meta.json").read_text(encoding="utf-8"))
    records = meta["records"]
    model_name = meta.get("model", "BAAI/bge-small-en-v1.5")
    return index, records, model_name

def embed_query(q: str, model_name: str):
    from fastembed import TextEmbedding
    emb = next(TextEmbedding(model_name=model_name).embed([q]))
    x = np.array(emb, dtype="float32")[None, :]
    faiss.normalize_L2(x)
    return x

def build_prompt(context_snippets, question):
    context = "\n\n---\n\n".join(context_snippets)
    return (
        "You are a helpful assistant. Answer the question using ONLY the Context. "
        "If the answer is not contained in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
    )

def ask_ollama(prompt: str, model: str = "llama3:8b", temperature: float = 0.2):
    url = "http://localhost:11434/api/generate"
    payload = {"model": model, "prompt": prompt, "temperature": temperature, "stream": False}
    r = requests.post(url, json=payload, timeout=600)
    r.raise_for_status()
    return r.json()["response"]

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("question", help="Your question")
    ap.add_argument("--store", default="rag_store", help="Path to the FAISS store")
    ap.add_argument("--topk", type=int, default=4)
    ap.add_argument("--model", default="llama3:8b", help="Ollama model name")
    args = ap.parse_args()

    store_dir = Path(args.store).resolve()
    index, records, emb_model = load_store(store_dir)
    qvec = embed_query(args.question, emb_model)
    scores, idx = index.search(qvec, args.topk)
    idx = idx[0].tolist()

    snippets = [records[i]["text"] for i in idx]
    sources = [records[i]["source"] for i in idx]
    prompt = build_prompt(snippets, args.question)
    answer = ask_ollama(prompt, model=args.model)

    print("Answer:\n")
    print(answer.strip())
    print("\nSources:")
    for s in dict.fromkeys(sources):  # de-duplicate, preserve order
        print(f"- {s}")

if __name__ == "__main__":
    main()

Ask a question:

python3 ask.py "How do I configure Nginx to reverse proxy to a local app?"

Try it on your own docs, READMEs, wiki exports, or PDFs. The response includes cited sources, so you can verify answers.

Real-World Examples You Can Try Today

  • Your team’s docs: Drop your internal Markdown/PDF docs into docs/ and ask operational questions.

  • Man pages snapshot: Export man pages to text and index them for conversational lookup.

    • Quick idea:
    mkdir -p docs/man
for cmd in bash awk sed grep find tar ssh systemctl journalctl; do
  man $cmd | col -b > "docs/man/${cmd}.txt"
done
python3 build_index.py --docs docs --out rag_store
python3 ask.py "How do I follow logs for a specific systemd unit?"

  • Logs and runbooks: Chunk long logs or runbooks; retrieve relevant snippets alongside commands you can paste into your shell.

4 Actionable Tips To Level Up Your RAG

1) Tune chunking for your data

  • Code/docs with short sections: try --chunk_size 400 --overlap 80.

  • Long PDFs: increase chunk size for more context.

2) Swap models without changing code

  • Embeddings: change model_name in build_index.py to another fastembed model (e.g., intfloat/e5-small-v2).

  • LLM: --model mistral or --model qwen2:7b on ask.py.

3) Automate re-indexing

  • Cron example:

    crontab -e
# Rebuild at 1am daily
0 1 * * * cd ~/rag-linux && . .venv/bin/activate && python3 build_index.py --docs docs --out rag_store >/tmp/rag_rebuild.log 2>&1

  • Or a systemd user service/timer for more control and logging.

4) Scale up when you outgrow FAISS-on-disk

  • For millions of chunks, switch to a vector database (Qdrant, Milvus). You can still use fastembed and the same retrieval pattern.

Troubleshooting and Performance

  • Missing pdftotext? Install poppler utilities as above (apt/dnf/zypper).

  • Slow embedding? fastembed is CPU-friendly; still, batch by default—be patient for large corpora.

  • Memory pressure? Index by batches and write partials; or increase swap temporarily.

  • Hallucinations? Keep temperature low (0.0–0.3) and ensure the prompt stresses “use only the context.”

Conclusion and Next Steps

You now have a private, Linux-native RAG pipeline you can script, cron, and customize. Your docs stay local; your answers come with citations; and everything runs with a few commands.

Next steps:

  • Point it at your real project docs or wiki exports.

  • Add a thin Bash wrapper to ask questions from anywhere on your system.

  • Experiment with better models in Ollama or different embedding models in fastembed.

  • Put the rag_store/ under version control for reproducible research builds.

If this was useful, share it with a teammate and wire it into your daily CLI workflow. Happy hacking!

Related posts