Building RAG Applications on Linux: From Zero to Local Q&A

Ever wished your shell could answer questions about your codebase, your logs, or your company wiki—without sending data to the cloud? Retrieval-Augmented Generation (RAG) lets you ground an LLM with your own documents so answers are accurate, cite sources, and stay private. On Linux, it’s straightforward, scriptable, and fast.

This post shows you how to build a local RAG pipeline on Linux using open-source tools. You’ll index your docs, ask questions, and get sourced answers—entirely on your machine.

What problem does RAG solve?

LLMs hallucinate when they don’t know your domain. RAG retrieves relevant context from your docs and feeds it into the LLM, improving accuracy.
Keeps your data local. No copy-pasting screenshots into random chat windows.
Repeatable, automatable. Linux cron jobs, shell scripts, systemd—build it into your workflow.

At a high level:

Retrieval: Embed your documents and fetch the most relevant chunks for a question.
Generation: Pass those chunks to an LLM to produce a grounded answer with citations.

What we’ll build

A tiny RAG stack on Linux:
- Local LLM runtime: Ollama
- Embeddings + Vector store: sentence-transformers + FAISS
- Orchestration: LangChain
Two scripts:
- Index your docs into a vector database
- Ask questions and get answers with sources

You can adapt this to your codebase, knowledge base, logs, or even man pages.

1) Install prerequisites

We’ll install Python, Git, and a few basics. Use your distro’s package manager.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y python3 python3-pip python3-virtualenv git curl

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-pip python3-virtualenv git curl

Note: We’ll create a Python venv, so either python3-venv or python3-virtualenv will work depending on your distro.

2) Install a local LLM with Ollama

Ollama runs LLMs locally and exposes a simple HTTP API.

Install Ollama (official script for Linux):

curl -fsSL https://ollama.com/install.sh | sh

Start the service (if not already running):
```
ollama serve
```
Pull a model (Llama 3 8B is a solid default for CPU/GPU laptops):
```
ollama pull llama3:8b
```

Tip: You can also try mistral:7b, phi3:mini, or other models from the Ollama library. Keep an eye on RAM/VRAM requirements.

3) Create a Python environment and install libraries

We’ll use LangChain, FAISS, and sentence-transformers.

Create and activate a virtual environment:

python3 -m venv .venv
. .venv/bin/activate

Install Python packages:

pip install --upgrade pip
pip install "langchain>=0.2" "langchain-community>=0.2" "langchain-text-splitters>=0.2" \
          "faiss-cpu" "sentence-transformers" "pypdf" "ollama"

4) Prepare some documents

Create a docs folder and add files you care about. Text, Markdown, code, and PDFs are fine.

Example: Export a few man pages into plain text:

mkdir -p docs
for m in bash grep sed awk find; do
man -P cat "$m" > "docs/$m.txt"
done

Or add your project README and docs:

cp -r /path/to/your/project/docs/* ./docs/

5) Index and query: minimal RAG in two scripts

Below are two scripts:

rag_index.py: loads and chunks your documents, builds embeddings, and saves a FAISS index to disk.
rag_ask.py: retrieves the most relevant chunks and asks a local LLM (via Ollama) to answer with sources.

Save both scripts to your project directory.

rag_index.py:

#!/usr/bin/env python3
import sys
from pathlib import Path
from typing import List

from langchain_community.document_loaders import TextLoader, PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import SentenceTransformerEmbeddings

def load_docs(root: Path) -> List:
    docs = []
    for path in root.rglob("*"):
        if not path.is_file():
            continue
        suffix = path.suffix.lower()
        try:
            if suffix in [".txt", ".md", ".rst", ".log", ".cfg", ".ini", ".py", ".sh", ".yaml", ".yml"]:
                docs.extend(TextLoader(str(path), encoding="utf-8").load())
            elif suffix == ".pdf":
                docs.extend(PyPDFLoader(str(path)).load())
        except Exception as e:
            print(f"Skip {path}: {e}")
    return docs

def main():
    if len(sys.argv) < 2:
        print("Usage: rag_index.py /path/to/docs [index_dir]")
        sys.exit(1)

    docs_path = Path(sys.argv[1]).expanduser().resolve()
    index_dir = Path(sys.argv[2]).resolve() if len(sys.argv) > 2 else Path("faiss_index")

    print(f"Loading documents from: {docs_path}")
    raw_docs = load_docs(docs_path)
    if not raw_docs:
        print("No documents found.")
        sys.exit(1)

    print(f"Loaded {len(raw_docs)} documents. Chunking...")
    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
    chunks = splitter.split_documents(raw_docs)
    print(f"Created {len(chunks)} chunks.")

    print("Creating embeddings (all-MiniLM-L6-v2)...")
    embeddings = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")

    print("Building FAISS index...")
    vectordb = FAISS.from_documents(chunks, embeddings)

    index_dir.mkdir(parents=True, exist_ok=True)
    vectordb.save_local(str(index_dir))
    print(f"Index saved to: {index_dir}")

if __name__ == "__main__":
    main()

rag_ask.py:

#!/usr/bin/env python3
import sys
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import SentenceTransformerEmbeddings
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA

def main():
    if len(sys.argv) < 2:
        print("Usage: rag_ask.py \"your question\" [index_dir] [model]")
        sys.exit(1)

    question = sys.argv[1]
    index_dir = sys.argv[2] if len(sys.argv) > 2 else "faiss_index"
    model = sys.argv[3] if len(sys.argv) > 3 else "llama3:8b"

    print(f"Loading index from: {index_dir}")
    embeddings = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
    vectordb = FAISS.load_local(index_dir, embeddings, allow_dangerous_deserialization=True)

    retriever = vectordb.as_retriever(search_kwargs={"k": 4})
    llm = Ollama(model=model)

    qa = RetrievalQA.from_chain_type(
        llm=llm,
        retriever=retriever,
        chain_type="stuff",
        return_source_documents=True,
    )

    print(f"Asking: {question}\n")
    result = qa({"query": question})
    print("Answer:\n")
    print(result["result"])
    print("\nSources:")
    for i, d in enumerate(result["source_documents"], 1):
        print(f"{i}. {d.metadata.get('source', 'unknown')}")
    print()

if __name__ == "__main__":
    main()

Make them executable and run:

chmod +x rag_index.py rag_ask.py

# 1) Index your docs
./rag_index.py ./docs

# 2) Ask a question
./rag_ask.py "How do I search recursively for a string and show line numbers?"

Tip: If Ollama isn’t running, start it in another terminal:

ollama serve

And ensure you’ve pulled a model:

ollama pull llama3:8b

Real-world examples you can try today

Your codebase: Point the indexer at your repo’s docs and src to build a code-aware assistant.
System administration: Index /etc configs, runbook Markdown, and logs to quickly troubleshoot recurring issues.
Man pages + README: Combine local man pages with your project README for a power-user shell assistant.
Internal wiki exports: Export Confluence/Notion/MediaWiki pages to Markdown and index them.

Tips for good RAG results

Chunking matters: 600–1,000 characters with 10–20% overlap is a solid default. Tune for your data.
Keep embeddings small and fast: all-MiniLM-L6-v2 is lightweight and accurate for many cases.
Garbage in, garbage out: Clean your docs (remove boilerplate, banners, navigation).
Add citations to prompts: The RetrievalQA chain will include sources; keep them visible for trust and debugging.
Iterate: Evaluate a small test set of Q&A pairs and refine chunk sizes, k (number of retrieved chunks), and model choice.

Optional: Package install recap (all distros)

If you skipped the prerequisites earlier or are setting up another machine: