RAG with Local Documents: Build a Private, Searchable AI Brain on Linux

Ever wished your shell could answer questions about your own PDFs, notes, and wikis—without sending anything to the cloud? Retrieval-Augmented Generation (RAG) with local documents gives you a private, high-signal assistant that understands your files. In this guide, you’ll build a minimal RAG pipeline that runs fully on Linux, uses the terminal, and keeps your data on disk.

We’ll cover what RAG is, why it matters, and walk through a working setup using Bash, Python, FAISS, and a local LLM. You’ll end with a repeatable workflow you can run on any Linux machine.

What’s RAG and why care?

• The problem: LLMs hallucinate and don’t know your internal docs. Pasting sensitive text into a web form is a compliance and privacy nightmare.

• The idea: Retrieval-Augmented Generation marries search and generation. You first retrieve the most relevant passages from your local corpus, then feed those into the LLM. The model answers using your context.

• The value:

  • Private: Your documents never leave your machine.
  • Relevant: Answers cite your PDFs, Markdown, and notes.
  • Efficient: Smaller local models perform better with good context.

What you’ll build

A simple, robust RAG stack:

• Ingest local documents (PDF/Markdown/TXT) and convert to plain text.

• Create embeddings and build a FAISS vector index.

• Query the index, assemble a context prompt, and call a local LLM (via Ollama).

• All wired together with Bash and Python scripts you can audit and extend.

You’ll get:

• ingest.sh to preprocess files

• build_index.py to embed and index

• ask.py to run retrieval + generation against your local model

1) Install prerequisites

We’ll install system tools for document conversion and a Python environment. Use the command set for your distro.

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip \
  git curl \
  poppler-utils ripgrep pandoc

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y \
  python3 python3-virtualenv python3-pip \
  git curl \
  poppler-utils ripgrep pandoc

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  python3 python3-virtualenv python3-pip \
  git curl \
  poppler-tools ripgrep pandoc

Why these packages?

• poppler-utils/poppler-tools: pdftotext for extracting text from PDFs

• pandoc: converts Markdown and DOCX to plain text

• ripgrep: fast file discovery (optional but handy)

2) Install a local LLM runtime (Ollama)

Ollama makes running local LLMs dead simple. If curl isn’t installed, use your package manager from above to install it, then:

curl -fsSL https://ollama.com/install.sh | sh

Start the Ollama server in a terminal:

ollama serve

In a new terminal, pull a small, capable model (you can swap later):

ollama pull llama3

Tip: Keep the model name (e.g., llama3) handy. You can also try mistral or other models in Ollama’s library.

3) Set up your project

Make a clean workspace and Python virtual environment:

mkdir -p ~/local-rag/{docs,corpus}
cd ~/local-rag
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install faiss-cpu sentence-transformers pypdf tqdm requests

• faiss-cpu: fast vector search

• sentence-transformers: quick, high-quality embeddings

• requests: simple HTTP calls to Ollama

• pypdf: optional fallback if you want Python-side PDF parsing

4) Ingest your local documents

Drop your files into ~/local-rag/docs (PDF/MD/TXT/DOCX). Then create ingest.sh to normalize everything to plain text.

Create ingest.sh:

#!/usr/bin/env bash
set -euo pipefail

DOCS_DIR="${1:-docs}"
OUT_DIR="${2:-corpus}"

mkdir -p "$OUT_DIR"

shopt -s nullglob
for f in "$DOCS_DIR"/*; do
  base="$(basename "$f")"
  stem="${base%.*}"
  ext="${base##*.}"
  out="$OUT_DIR/$stem.txt"

  case "${ext,,}" in
    pdf)
      echo "[pdf]  $base -> $out"
      pdftotext -layout -nopgbrk "$f" "$out"
      ;;
    md|markdown)
      echo "[md]   $base -> $out"
      pandoc -f gfm -t plain "$f" -o "$out"
      ;;
    txt|log|conf|ini)
      echo "[txt]  $base -> $out"
      # Normalize line endings and strip control chars
      sed -e 's/\r$//' "$f" | tr -cd '\11\12\15\40-\176' > "$out"
      ;;
    docx)
      echo "[docx] $base -> $out"
      pandoc "$f" -t plain -o "$out"
      ;;
    *)
      echo "[skip] $base (unsupported extension: .$ext)"
      ;;
  esac
done

# Optional: remove empty files
find "$OUT_DIR" -type f -size 0 -delete
echo "Ingest done. Text files are in: $OUT_DIR"

Make it executable and run:

chmod +x ingest.sh
./ingest.sh docs corpus

Result: All supported documents become plain text in corpus/.

5) Build a vector index

Create build_index.py to embed chunks and store them in a FAISS index.

#!/usr/bin/env python3
import os, json, glob, math
from tqdm import tqdm
import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

CORPUS_DIR = os.environ.get("CORPUS_DIR", "corpus")
INDEX_PATH = os.environ.get("INDEX_PATH", "rag.index")
META_PATH  = os.environ.get("META_PATH",  "chunks.jsonl")
EMB_MODEL  = os.environ.get("EMB_MODEL",  "sentence-transformers/all-MiniLM-L6-v2")

def read_text(path):
    with open(path, "r", encoding="utf-8", errors="ignore") as f:
        return f.read()

def chunk_text(text, chunk_size=800, overlap=120):
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = words[i:i+chunk_size]
        if not chunk:
            break
        chunks.append(" ".join(chunk))
        i += chunk_size - overlap
    return chunks

def l2_normalize(x):
    norms = np.linalg.norm(x, axis=1, keepdims=True) + 1e-12
    return x / norms

def main():
    files = sorted(glob.glob(os.path.join(CORPUS_DIR, "*.txt")))
    if not files:
        raise SystemExit(f"No .txt files found in {CORPUS_DIR}. Run ingest.sh first.")

    print(f"Loading embedding model: {EMB_MODEL}")
    model = SentenceTransformer(EMB_MODEL)

    chunks = []
    sources = []
    for path in files:
        text = read_text(path)
        if not text.strip():
            continue
        for i, c in enumerate(chunk_text(text)):
            chunks.append(c)
            sources.append({"source": os.path.basename(path), "chunk": i})

    print(f"Total chunks: {len(chunks)}")
    batch = 256
    embeddings = []
    for i in tqdm(range(0, len(chunks), batch), desc="Embedding"):
        embs = model.encode(chunks[i:i+batch], convert_to_numpy=True, show_progress_bar=False, normalize_embeddings=False)
        embeddings.append(embs)
    X = np.vstack(embeddings).astype("float32")
    X = l2_normalize(X)  # for cosine similarity via inner product

    dim = X.shape[1]
    index = faiss.IndexFlatIP(dim)
    index.add(X)

    faiss.write_index(index, INDEX_PATH)
    with open(META_PATH, "w", encoding="utf-8") as f:
        for c, s in zip(chunks, sources):
            rec = {"text": c, **s}
            f.write(json.dumps(rec, ensure_ascii=False) + "\n")

    print(f"Wrote index: {INDEX_PATH}")
    print(f"Wrote meta:  {META_PATH}")

if __name__ == "__main__":
    main()

Run it:

python build_index.py

This creates:

• rag.index: FAISS index on your embeddings

• chunks.jsonl: Text and source metadata aligned with the index

6) Ask questions with local RAG

Create ask.py to retrieve the most relevant chunks and ask your local model via Ollama.

#!/usr/bin/env python3
import os, sys, json
import numpy as np
import faiss
import requests
from sentence_transformers import SentenceTransformer

INDEX_PATH = os.environ.get("INDEX_PATH", "rag.index")
META_PATH  = os.environ.get("META_PATH",  "chunks.jsonl")
EMB_MODEL  = os.environ.get("EMB_MODEL",  "sentence-transformers/all-MiniLM-L6-v2")
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434/api/generate")
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "llama3")
TOP_K = int(os.environ.get("TOP_K", "5"))

def l2_normalize(x):
    norms = np.linalg.norm(x, axis=1, keepdims=True) + 1e-12
    return x / norms

def load_meta(path):
    meta = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            meta.append(json.loads(line))
    return meta

def build_prompt(question, contexts):
    lines = []
    lines.append("You are a helpful, concise assistant. Use the provided context to answer.")
    lines.append("If the answer is not in the context, say you don't know.")
    lines.append("")
    lines.append("Context:")
    for i, ctx in enumerate(contexts, 1):
        src = ctx.get("source", "unknown")
        txt = ctx.get("text", "").strip().replace("\n", " ")
        lines.append(f"- [{src}] {txt}")
    lines.append("")
    lines.append(f"Question: {question}")
    lines.append("Answer:")
    return "\n".join(lines)

def ask_ollama(model, prompt, url=OLLAMA_URL):
    payload = {"model": model, "prompt": prompt, "stream": False}
    r = requests.post(url, json=payload, timeout=300)
    r.raise_for_status()
    data = r.json()
    return data.get("response", "").strip()

def main():
    if len(sys.argv) < 2:
        print("Usage: ask.py \"your question here\"")
        sys.exit(1)
    question = sys.argv[1]

    index = faiss.read_index(INDEX_PATH)
    meta = load_meta(META_PATH)

    emb_model = SentenceTransformer(EMB_MODEL)
    q = emb_model.encode([question], convert_to_numpy=True).astype("float32")
    q = l2_normalize(q)
    D, I = index.search(q, TOP_K)

    contexts = [meta[i] for i in I[0]]
    prompt = build_prompt(question, contexts)
    answer = ask_ollama(OLLAMA_MODEL, prompt)
    print(answer)

    # Optional: show sources
    print("\nSources:")
    for m in contexts:
        print(f"- {m.get('source')} (chunk {m.get('chunk')})")

if __name__ == "__main__":
    main()

Ask a question:

. .venv/bin/activate
python ask.py "What ports does our VPN guide say to open?"

If your documents contain that info, the answer will cite relevant chunks and list the source files at the end.

Real-world examples

• On-call ops: Ask “How do I rotate the staging DB credentials?” and get the canonical steps from your runbooks.

• SecOps: “Which cipher suites are approved?” to search policy PDFs or Markdown.

• Dev onboarding: “How do I run the integration tests locally?” across READMEs and wiki exports.

• Customer support: “Refund policy exceptions?” across your internal knowledge base.

7) Practical tips to level up

• Keep the index fresh:

  • Add this to a nightly cron to re-run ingestion and indexing:

  0 2 * * * cd ~/local-rag && . .venv/bin/activate && ./ingest.sh docs corpus && python build_index.py
  

• Model selection:

  • Start with llama3 or mistral. For tighter hardware, try smaller variants. Higher-quality prompts plus good retrieval often beats chasing bigger models.

• Retrieval quality:

  • Tune chunk size and overlap in build_index.py. Larger chunks = more context, but too large can dilute relevance. Try chunk_size=600–1000 with overlap 80–160.

• Robust ingestion:

  • Extend ingest.sh with more converters (HTML via pandoc -f html -t plain, EPUB via pandoc).

• Safety:

  • The assistant should admit “I don’t know” if context is missing. This is a feature, not a bug.

Troubleshooting

• Ollama not responding:

  • Ensure it’s running: ollama serve in a terminal, then curl localhost:11434/api/tags.

• No results or weak answers:

  • Verify your corpus/ has non-empty .txt files.
  • Increase TOP_K to 8–10.
  • Try a stronger embedding model (e.g., sentence-transformers/all-mpnet-base-v2) and rebuild the index.

• FAISS import error:

  • Ensure you’re using the venv where faiss-cpu is installed.

Conclusion and next steps

You’ve built a private RAG assistant that understands your local documents—no cloud, no copy-paste. From here you can:

• Add reranking (e.g., sentence-transformers cross-encoders) for even better retrieval.

• Swap FAISS for a server-backed vector DB (e.g., Qdrant) if you scale up.

• Wrap this in a TUI or a simple web UI with FastAPI for your team.

Call to action:

• Point it at a real folder you care about (wikis, policies, READMEs).

• Iterate on chunking and embedding models.

• Share a sanitized version of your ingest.sh and tweaks with your team.

Your terminal just got smarter—with your own data.