Posted on
• Artificial Intelligence

Build Your First RAG Pipeline

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Build Your First RAG Pipeline on Linux (with Bash-first ergonomics)

What if your terminal could answer questions about your own docs with the clarity of a seasoned teammate—and without leaking data to the internet? That’s the promise of Retrieval-Augmented Generation (RAG): pair a language model with a private, searchable index of your documents, so responses are grounded in your data instead of guesses.

In this guide, you’ll build a minimal, working RAG pipeline on Linux using simple Bash commands and a few small Python scripts. You’ll ingest docs, embed and index them locally, retrieve relevant chunks for a question, and generate answers using either a local LLM (via Ollama) or a hosted API.

You’ll end with a repeatable workflow you can aim at logs, wikis, PDFs, READMEs—whatever text you care about.

Why RAG (and why now)?

  • Hallucinations cost time. RAG grounds answers in your data to reduce guesswork.

  • Privacy by default. Keep proprietary documentation and notes on your machine.

  • Lightweight and flexible. You don’t need a massive stack: a vector store (Chroma), an embedding model, and a generator (local or API) is enough.

  • Linux-native flow. Use Bash to wire everything together; keep scripts small, auditable, and portable.

1) Install prerequisites

We’ll use:

  • Python 3 + venv, pip

  • Basic build tools

  • Optional PDF-to-text (pdftotext via Poppler)

  • Git, curl, jq, unzip

  • Optional: Ollama for a local model

System packages (choose your distro):

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl jq unzip gcc build-essential pkg-config
# Optional: PDF to text
sudo apt install -y poppler-utils

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y python3 python3-virtualenv python3-pip git curl jq unzip gcc gcc-c++ make pkgconfig
# Optional: PDF to text
sudo dnf install -y poppler-utils

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-pip python3-venv git curl jq unzip gcc gcc-c++ make pkg-config
# Optional: PDF to text (note the package name)
sudo zypper install -y poppler-tools

Create a project and virtual environment:

mkdir -p ~/rag-lab && cd ~/rag-lab
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

Python packages:

pip install chromadb sentence-transformers requests tqdm

Optional: Install Ollama (local LLM runtime):

curl -fsSL https://ollama.com/install.sh | sh
# Pull a small, capable model (choose one):
ollama pull llama3       # good general model
# or
ollama pull mistral

Tip: You can also use a hosted API (e.g., OpenAI). If so, export your key:

export OPENAI_API_KEY="sk-..."

If you’ll use Ollama, export a model name (or keep the default in the script):

export OLLAMA_MODEL="llama3"

2) Prepare documents and index them

Create a docs directory and drop in some files (Markdown, TXT, and optional PDF). For a quick test:

mkdir -p docs
curl -L https://raw.githubusercontent.com/docker/docs/main/get-started/overview.md -o docs/docker_overview.md
curl -L https://raw.githubusercontent.com/redis/redis/unstable/README.md -o docs/redis_readme.md

Optional PDF example (if you installed pdftotext):

  • Place any PDF in docs/, e.g., docs/example.pdf

Now create a script to ingest and embed documents into a local Chroma DB.

Save as ingest.py:

#!/usr/bin/env python3
import os, sys, subprocess, uuid
from pathlib import Path
from typing import List
from tqdm import tqdm

import chromadb
from sentence_transformers import SentenceTransformer

DOC_DIR = Path("docs")
DB_DIR = Path("rag_store")
COLLECTION_NAME = "docs"

def read_text_from_file(path: Path) -> str:
    suffix = path.suffix.lower()
    if suffix in [".md", ".txt", ".rst"]:
        try:
            return path.read_text(encoding="utf-8", errors="ignore")
        except Exception:
            return path.read_text(errors="ignore")
    elif suffix == ".pdf":
        # Requires pdftotext (poppler-utils/poppler-tools)
        try:
            out = subprocess.check_output(
                ["pdftotext", "-layout", "-nopgbrk", str(path), "-"],
                stderr=subprocess.DEVNULL
            )
            return out.decode("utf-8", errors="ignore")
        except Exception as e:
            print(f"Warning: could not convert PDF {path}: {e}", file=sys.stderr)
            return ""
    else:
        return ""

def chunk_by_words(text: str, chunk_size: int = 200, overlap: int = 40) -> List[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunk = " ".join(words[start:end])
        if chunk.strip():
            chunks.append(chunk)
        if end == len(words):
            break
        start = end - overlap
        if start < 0:
            start = 0
    return chunks

def main():
    if not DOC_DIR.exists():
        print(f"Missing {DOC_DIR}/", file=sys.stderr)
        sys.exit(1)

    client = chromadb.PersistentClient(path=str(DB_DIR))
    collection = client.get_or_create_collection(name=COLLECTION_NAME)

    model_name = os.environ.get("EMBED_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
    embedder = SentenceTransformer(model_name)

    files = [p for p in DOC_DIR.rglob("*") if p.is_file()]
    if not files:
        print(f"No files found in {DOC_DIR}/", file=sys.stderr)
        sys.exit(1)

    for path in tqdm(files, desc="Indexing"):
        text = read_text_from_file(path)
        if not text.strip():
            continue
        chunks = chunk_by_words(text, chunk_size=220, overlap=50)
        ids = []
        metas = []
        docs = []
        for i, chunk in enumerate(chunks):
            ids.append(f"{path.relative_to(DOC_DIR)}::chunk{i}::{uuid.uuid4().hex[:8]}")
            metas.append({"source": str(path.relative_to(DOC_DIR)), "chunk": i})
            docs.append(chunk)
        if docs:
            embs = embedder.encode(docs, normalize_embeddings=True).tolist()
            collection.add(ids=ids, documents=docs, metadatas=metas, embeddings=embs)

    print(f"Indexed {len(files)} files into {DB_DIR}/")

if __name__ == "__main__":
    main()

Make it executable and run:

chmod +x ingest.py
./ingest.py

What you just did:

  • Converted files to plain text (PDFs via pdftotext, others directly).

  • Chunked documents into ~220-word slices with slight overlap.

  • Generated embeddings with a small, fast model.

  • Stored embeddings and metadata in a persistent Chroma DB at ./rag_store.

3) Retrieve relevant chunks and generate an answer

Create query.py that:

  • Embeds your question

  • Retrieves top-N relevant chunks

  • Builds a prompt and sends it to either Ollama or a hosted API

  • Prints the final answer (and optionally the sources)

Save as query.py:

#!/usr/bin/env python3
import os, sys, json
from typing import List, Dict
import chromadb
from sentence_transformers import SentenceTransformer
import requests

DB_DIR = "rag_store"
COLLECTION_NAME = "docs"

def get_collection():
    client = chromadb.PersistentClient(path=DB_DIR)
    return client.get_or_create_collection(name=COLLECTION_NAME)

def retrieve(question: str, k: int = 5):
    collection = get_collection()
    embed_model = os.environ.get("EMBED_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
    embedder = SentenceTransformer(embed_model)

    q_emb = embedder.encode([question], normalize_embeddings=True).tolist()
    res = collection.query(query_embeddings=q_emb, n_results=k, include=["documents","metadatas","distances"])
    docs = res.get("documents", [[]])[0]
    metas = res.get("metadatas", [[]])[0]
    dists = res.get("distances", [[]])[0]
    return list(zip(docs, metas, dists))

def build_prompt(question: str, contexts: List[Dict]) -> str:
    parts = []
    for i, (doc, meta, dist) in enumerate(contexts, 1):
        src = meta.get("source", "unknown")
        parts.append(f"[Source {i}: {src}] {doc}")
    context_block = "\n\n---\n".join(parts)
    prompt = (
        "You are a helpful assistant. Use ONLY the context below to answer. "
        "If the answer is not contained in the context, say you don't know.\n\n"
        f"Context:\n{context_block}\n\n"
        f"Question: {question}\n"
        "Answer concisely:"
    )
    return prompt

def call_ollama(prompt: str, model: str = "llama3") -> str:
    url = "http://localhost:11434/api/generate"
    payload = {"model": model, "prompt": prompt, "stream": False}
    r = requests.post(url, json=payload, timeout=600)
    r.raise_for_status()
    data = r.json()
    return data.get("response", "").strip()

def call_openai(prompt: str, model: str = "gpt-4o-mini") -> str:
    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        raise RuntimeError("OPENAI_API_KEY not set")
    url = "https://api.openai.com/v1/chat/completions"
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    body = {
        "model": model,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant that answers using provided context only."},
            {"role": "user", "content": prompt},
        ],
        "temperature": 0.2,
    }
    r = requests.post(url, headers=headers, data=json.dumps(body), timeout=600)
    r.raise_for_status()
    data = r.json()
    return data["choices"][0]["message"]["content"].strip()

def main():
    if len(sys.argv) < 2:
        print("Usage: ./query.py \"your question\" [top_k]", file=sys.stderr)
        sys.exit(1)
    question = sys.argv[1]
    top_k = int(sys.argv[2]) if len(sys.argv) > 2 else 5
    hits = retrieve(question, k=top_k)
    if not hits:
        print("No results. Did you run ./ingest.py and add docs/?", file=sys.stderr)
        sys.exit(1)

    prompt = build_prompt(question, hits)
    answer = ""
    try:
        if os.environ.get("OLLAMA_MODEL"):
            answer = call_ollama(prompt, os.environ["OLLAMA_MODEL"])
        elif os.environ.get("OPENAI_API_KEY"):
            answer = call_openai(prompt)
        else:
            # Fallback: show the prompt so you can pipe it to an LLM manually
            print("No generator configured (set OLLAMA_MODEL or OPENAI_API_KEY).")
            print("\n----- PROMPT BELOW -----\n")
            print(prompt)
            sys.exit(0)
    except Exception as e:
        print(f"Generation error: {e}", file=sys.stderr)
        sys.exit(2)

    # Print answer and sources
    print("Answer:\n")
    print(answer)
    print("\nSources:")
    for i, (_, meta, dist) in enumerate(hits, 1):
        print(f"- [{i}] {meta.get('source')} (similarity ~ {1 - dist:.3f})")

if __name__ == "__main__":
    main()

Make it executable:

chmod +x query.py

Ask your first question:

export OLLAMA_MODEL="llama3"     # or ensure OPENAI_API_KEY is set
./query.py "What is Docker used for?"

You should see a concise answer with a list of sources pointing to the chunks retrieved from your docs.

4) Real-world usage patterns

Here are a few concrete ways to point this at real data:

  • Team knowledge base

    • Export Confluence/Notion pages as Markdown and drop them in docs/.
    • Re-run ./ingest.py as content changes, or put it in a cron job.

  • Incident timelines and logs

    • Convert key timelines to .txt and store in docs/incidents/.
    • Query: ./query.py "What remediations did we try for the TLS outage?"

  • Developer onboarding

    • Include READMEs from multiple services.
    • Query: ./query.py "How do I run the API service locally with a seeded DB?"

  • Mixed PDFs

    • Add postmortems or vendor docs as PDFs. Ensure pdftotext is installed.
    • Query: ./query.py "Which Redis config is recommended for persistence?"

Tip: For repeat indexing, you can start by deleting the store:

rm -rf rag_store && ./ingest.py

5) Common tweaks and troubleshooting

  • Speed vs accuracy

    • Increase chunk size for more context per chunk; increase top_k to retrieve more chunks.
    • Try a different embedding model via export EMBED_MODEL="sentence-transformers/all-mpnet-base-v2".

  • GPU acceleration

    • If you have a GPU, sentence-transformers can leverage it when installed with the right PyTorch build; consult PyTorch’s install matrix.

  • PDFs not indexing

    • Make sure pdftotext is installed:
    • apt: sudo apt install -y poppler-utils
    • dnf: sudo dnf install -y poppler-utils
    • zypper: sudo zypper install -y poppler-tools

  • Local model not responding

    • Ensure Ollama is running: sudo systemctl status ollama (or restart sudo systemctl restart ollama) and that your model is pulled: ollama list.

  • Verify the retrieval step alone

    • Temporarily comment out the generation calls and print the prompt to inspect the context that’s fed to the model.

Summary and next steps

You just built a minimal, end-to-end RAG pipeline on Linux:

  • Ingest and chunk your docs

  • Embed and index locally with Chroma

  • Retrieve relevant chunks

  • Generate grounded answers with a local or hosted LLM

From here, you can:

  • Add a simple REST API or TUI wrapper around query.py

  • Store source URLs and add clickable references

  • Swap in different embedding/LLM models

  • Schedule incremental re-indexing with a Makefile or systemd timer

Call to action:

  • Point this at a folder you actually use at work—playbooks, READMEs, or SOPs—then ask 3 questions that normally take you 10+ minutes to answer. Measure the time saved, and iterate.

Related posts