Posted on
• Artificial Intelligence

RAG on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

RAG on Linux: Build a Private, Search-Backed AI Assistant from the Shell

Ever wished your terminal could answer questions about your own documentation just as confidently as a chatbot? The problem: large language models (LLMs) don’t know your private files and often hallucinate. The value: Retrieval-Augmented Generation (RAG) grounds an LLM’s answers in your real documents, so it cites what it knows and keeps your data local. Best part? You can build it on Linux with a few shell commands.

This guide shows you how to stand up a minimal, fast, and fully local RAG stack on Linux using Ollama (local LLM), Chroma (vector store), and Sentence-Transformers (embeddings). You’ll index any folder of Markdown/text/PDF files and then query it from the command line.

Why RAG on Linux is worth your time

  • Reduce hallucinations: Answers are based on retrieved passages from your own docs.

  • Keep data private: Everything runs locally—no cloud uploads required.

  • Scriptable and automatable: Cron, systemd, Bash—Linux makes RAG operations robust and repeatable.

  • Composable: Swap models, embeddings, or stores without rewriting everything.

What we’ll build

  • A small Python tool that:
    • Ingests a folder of .txt/.md/.pdf files
    • Chunks and embeds text
    • Stores vectors in a persistent Chroma index
    • Answers your question using a local LLM (via Ollama), showing sources

1) Install prerequisites

You’ll need Git, Python, build tools, and PDF tooling. Pick your distro:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y git curl python3 python3-venv python3-pip build-essential cmake pkg-config poppler-utils

  • Fedora/RHEL (dnf):

    sudo dnf -y groupinstall "Development Tools"
sudo dnf -y install git curl python3 python3-pip python3-virtualenv cmake pkgconfig poppler-utils

  • openSUSE (zypper):

    sudo zypper refresh
sudo zypper install -y git curl python3 python3-pip python3-virtualenv gcc-c++ make cmake pkg-config poppler-tools

Create and activate a Python virtual environment:

python3 -m venv rag-venv
source rag-venv/bin/activate
python -m pip install --upgrade pip

Install Python libraries:

pip install chromadb sentence-transformers ollama pypdf tqdm

2) Run a local model with Ollama

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Start the Ollama service (if not already running):

# One-shot in foreground:
ollama serve

# Or via systemd if available:
sudo systemctl enable --now ollama

Pull a model. Llama 3 (8B) is a good CPU-friendly default:

ollama pull llama3

You can use a different model (e.g., mistral, llama3:instruct, qwen2) if you prefer.

3) Prepare some documents

Create a folder and drop in text, Markdown, or PDFs you want the assistant to know:

mkdir -p ~/rag/docs
# Copy or create files like:
# ~/rag/docs/README.md
# ~/rag/docs/handbook.pdf
# ~/rag/docs/notes.txt

Tip: You can also convert PDFs to text via Poppler (optional), but the script below reads PDFs directly:

pdftotext ~/rag/docs/handbook.pdf ~/rag/docs/handbook.txt

4) A minimal RAG script (index + query)

Save the following as linux_rag.py:

#!/usr/bin/env python3
import argparse, os, sys, uuid, pathlib, time
from typing import List, Dict
from tqdm import tqdm

# Embeddings
from sentence_transformers import SentenceTransformer

# Vector store (Chroma)
import chromadb
from chromadb.config import Settings

# PDF loader
from pypdf import PdfReader

# Local LLM client (Ollama)
import ollama


def read_file(p: pathlib.Path) -> str:
    suf = p.suffix.lower()
    try:
        if suf in [".txt", ".md", ".log", ".cfg", ".conf"]:
            return p.read_text(encoding="utf-8", errors="ignore")
        if suf == ".pdf":
            text = []
            reader = PdfReader(str(p))
            for page in reader.pages:
                text.append(page.extract_text() or "")
            return "\n".join(text)
    except Exception as e:
        print(f"[warn] Failed to read {p}: {e}", file=sys.stderr)
    return ""


def chunk_text(text: str, size: int = 800, overlap: int = 150) -> List[str]:
    chunks, start = [], 0
    n = len(text)
    while start < n:
        end = min(start + size, n)
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        if end == n:
            break
        start = max(0, end - overlap)
        if start >= n:
            break
    return chunks


def build_or_load_collection(db_path: str, name: str = "docs"):
    client = chromadb.PersistentClient(path=db_path, settings=Settings(allow_reset=False))
    try:
        col = client.get_collection(name)
    except Exception:
        col = client.create_collection(name, metadata={"hnsw:space": "cosine"})
    return client, col


def index_folder(docs_dir: str, db_path: str, chunk: int, overlap: int, model_name: str):
    docs_dir = pathlib.Path(docs_dir).expanduser().resolve()
    client, col = build_or_load_collection(db_path)

    embedder = SentenceTransformer(model_name)
    files = [p for p in docs_dir.rglob("*") if p.is_file() and p.suffix.lower() in (".txt", ".md", ".pdf", ".log", ".cfg", ".conf")]

    if not files:
        print(f"[err] No supported files found in {docs_dir}")
        sys.exit(1)

    print(f"[info] Found {len(files)} files. Chunking and embedding...")
    ids, docs, metas, embs = [], [], [], []

    for f in tqdm(files, unit="file"):
        content = read_file(f)
        if not content.strip():
            continue
        chunks = chunk_text(content, size=chunk, overlap=overlap)
        if not chunks:
            continue
        # Embed in batches
        batch_embeddings = embedder.encode(chunks, normalize_embeddings=True).tolist()
        for ch, emb in zip(chunks, batch_embeddings):
            ids.append(str(uuid.uuid4()))
            docs.append(ch)
            metas.append({"source": str(f)})
            embs.append(emb)

        # Commit in reasonable batches to avoid huge payloads
        if len(ids) >= 500:
            col.add(ids=ids, documents=docs, metadatas=metas, embeddings=embs)
            ids, docs, metas, embs = [], [], [], []

    if ids:
        col.add(ids=ids, documents=docs, metadatas=metas, embeddings=embs)

    print(f"[ok] Indexed documents into {db_path} (collection: {col.name}).")


def retrieve_and_answer(db_path: str, question: str, k: int, llm_model: str, temperature: float):
    client, col = build_or_load_collection(db_path)
    # Simple retrieval
    res = col.query(query_texts=[question], n_results=k)
    if not res["documents"]:
        print("[err] No results. Did you index a folder?")
        sys.exit(1)
    docs = res["documents"][0]
    metas = res["metadatas"][0]
    context_parts = []
    for i, (d, m) in enumerate(zip(docs, metas), 1):
        src = m.get("source", "unknown")
        context_parts.append(f"[{i}] {src}\n{d}")
    context = "\n\n".join(context_parts)

    system_prompt = (
        "You are a careful assistant. Use ONLY the provided context to answer. "
        "If the answer is not in the context, say you don't know."
    )
    user_prompt = f"Context:\n{context}\n\nQuestion: {question}\nAnswer with citations like [1], [2] as needed."

    resp = ollama.chat(
        model=llm_model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        options={"temperature": temperature},
    )
    answer = resp["message"]["content"].strip()

    print("\n=== Answer ===\n")
    print(answer)
    print("\n=== Sources ===\n")
    for i, m in enumerate(metas, 1):
        print(f"[{i}] {m.get('source', 'unknown')}")


def main():
    ap = argparse.ArgumentParser(description="Minimal local RAG on Linux (Chroma + Sentence-Transformers + Ollama)")
    ap.add_argument("--docs", help="Path to folder of documents to index")
    ap.add_argument("--rebuild", action="store_true", help="Rebuild index from --docs")
    ap.add_argument("--db", default="./chroma_db", help="Chroma persistence directory")
    ap.add_argument("--embed-model", default="sentence-transformers/all-MiniLM-L6-v2", help="Sentence-Transformer model")
    ap.add_argument("--chunk", type=int, default=800, help="Chunk size (characters)")
    ap.add_argument("--overlap", type=int, default=150, help="Chunk overlap (characters)")
    ap.add_argument("--k", type=int, default=4, help="Top-k passages to retrieve")
    ap.add_argument("--llm", default="llama3", help="Ollama model to use (e.g., llama3, mistral)")
    ap.add_argument("--temperature", type=float, default=0.2, help="LLM temperature")
    ap.add_argument("--question", help="Question to ask")
    args = ap.parse_args()

    if args.rebuild:
        if not args.docs:
            print("[err] --rebuild requires --docs")
            sys.exit(1)
        t0 = time.time()
        index_folder(args.docs, args.db, args.chunk, args.overlap, args.embed_model)
        print(f"[timing] Index built in {time.time() - t0:.1f}s")

    if args.question:
        retrieve_and_answer(args.db, args.question, args.k, args.llm, args.temperature)
    elif not args.rebuild:
        print("Nothing to do. Use --rebuild --docs DIR to index and/or --question '...' to query.")


if __name__ == "__main__":
    main()

Make it executable:

chmod +x linux_rag.py

5) Try it out

  • Build the index from your docs:

    ./linux_rag.py --rebuild --docs ~/rag/docs

  • Ask a question:

    ./linux_rag.py --question "What does the onboarding process require?"

  • Change models or retrieval depth:

    ./linux_rag.py --question "Summarize our SSO setup." --llm mistral --k 6

You’ll see an answer grounded in your files, followed by source citations.

Actionable extras and real-world tips

  • Tune chunking:

    • If answers feel off, try --chunk 1200 --overlap 200 to give the LLM more contiguous context.

  • Keep sources readable:

    • The script prints file paths; organize docs so paths convey meaning (e.g., docs/policies/security.md).

  • Rebuild safely:

    • Re-index after big documentation changes: ./linux_rag.py --rebuild --docs ~/rag/docs

  • Automate refresh:

    • Use cron or systemd timers to refresh nightly if your docs change frequently.

  • Troubleshooting Ollama:

    • Check the service: systemctl status ollama
    • Test a bare prompt: ollama run llama3 "Say hello"
    • If you’re low on RAM, try a smaller model variant (search ollama list and ollama run).

  • GPU acceleration (optional):

    • Sentence-Transformers will use CUDA if available; install your GPU drivers and PyTorch with CUDA to speed up embedding.

Conclusion and next steps

You’ve just built a private, local RAG system on Linux that can answer questions with citations from your own files. From here you can:

  • Add a TUI/CLI wrapper or a simple Flask app for teammates.

  • Swap in a different vector store (e.g., FAISS, Qdrant) or a different embedding model.

  • Containerize it with Podman/Docker and schedule regular indexing.

Ready to level up? Point it at your entire internal knowledge base and turn your terminal into a reliable, source-backed assistant.

Related posts