RAG with Ollama: Build a Private AI Searcher on Linux Using Just Bash

If you’ve ever copied half your docs into a prompt and prayed the model wouldn’t hallucinate, you’ve met the limits of plain LLMs. Retrieval-Augmented Generation (RAG) fixes this by letting your model “look up” the right context before it answers. Even better: you can run it locally, privately, and script it with Bash using Ollama.

In this post, you’ll:

• Understand why RAG + Ollama is a great fit on Linux

• Install everything you need with apt, dnf, or zypper

• Build a minimal RAG stack (index + query) using only Bash, curl, and jq

• See a real example you can adapt for your own docs

No Python. No cloud. All local.

---

Why RAG on Linux with Ollama?

• Relevance and accuracy: RAG reduces hallucinations by grounding answers in your own documents.

• Privacy: Your content and queries never leave your machine.

• Control and cost: Use local models you choose, no API bills or rate limits.

• Scriptability: Ollama exposes a clean local HTTP API, perfect for Bash pipelines and cron jobs.

What we’ll build:

• An indexing script that chunks your text files, creates embeddings with Ollama, and stores them as JSONL.

• A query script that finds the most relevant chunks using cosine similarity (in jq!) and asks a local LLM to answer using that context.

---

1) Install prerequisites

We’ll need: curl, jq, and Ollama. Optional tools for converting documents are included below.

• On Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq

• On Fedora (dnf):

sudo dnf install -y curl jq

• On openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq

Install Ollama (Linux):

curl -fsSL https://ollama.com/install.sh | sh

Start the server (choose one):

# Foreground (good for testing)
ollama serve

# Or as a user service (if installed that way):
systemctl --user enable --now ollama

Verify:

ollama --version
curl -s http://localhost:11434/api/tags | jq

Pull a chat model and an embedding model:

ollama pull llama3.1
ollama pull nomic-embed-text

Optional: tools to convert PDFs/Docs to text

• Debian/Ubuntu (apt):

sudo apt install -y poppler-utils pandoc

• Fedora (dnf):

sudo dnf install -y poppler-utils pandoc

• openSUSE (zypper):

sudo zypper install -y poppler-tools pandoc

---

2) Prepare your knowledge base

Put your source files under a directory such as data/. Plain text and Markdown work best out of the box. If you have PDFs, convert them first:

# PDF to text
pdftotext input.pdf data/input.txt

# Docx/Markdown to text
pandoc -s input.md -t plain -o data/input.txt

---

3) Build the index (chunk + embed)

The script below:

• Walks data/ for .txt and .md files

• Chunks text by word count (with overlap)

• Calls Ollama embeddings API (nomic-embed-text)

• Writes a JSONL index with text, file, id, and embedding

Save as build_index.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

SRC_DIR="${1:-data}"
OUT="${2:-index.jsonl}"
EMBED_MODEL="${3:-nomic-embed-text}"

if ! curl -s http://localhost:11434/api/tags >/dev/null; then
  echo "Ollama server not reachable at http://localhost:11434. Start it with: ollama serve"
  exit 1
fi

echo "Building RAG index from: $SRC_DIR"
: > "$OUT"  # truncate

chunk_words() {
  # Chunk input (stdin) into ~200-word chunks with 40-word overlap, NUL-separated
  awk -v size=200 -v overlap=40 'BEGIN{ORS="\0"}
  {
    for (i=1; i<=NF; i++) { w[++n] = $i }
  }
  END{
    if (n == 0) exit
    for (s=1; s<=n; s+= (size - overlap)) {
      e = (s + size - 1 <= n) ? s + size - 1 : n
      chunk=""
      for (i=s; i<=e; i++) { chunk = chunk w[i] " " }
      print chunk
      if (e == n) break
    }
  }'
}

id=0
while IFS= read -r -d '' file; do
  echo "Indexing: $file"
  # Normalize whitespace and chunk
  tr -s '[:space:]' ' ' < "$file" | chunk_words | \
  while IFS= read -r -d '' chunk; do
    # Create embedding
    payload=$(jq -n --arg m "$EMBED_MODEL" --arg prompt "$chunk" '{model:$m, prompt:$prompt}')
    emb=$(curl -s http://localhost:11434/api/embeddings \
      -H "Content-Type: application/json" \
      -d "$payload" | jq -c '.embedding')

    # Append to JSONL
    jq -n --arg file "$file" --arg text "$chunk" --argjson emb "$emb" --argjson id "$id" \
      '{id:$id, file:$file, text:$text, embedding:$emb}' >> "$OUT"
    id=$((id+1))
  done
done < <(find "$SRC_DIR" -type f \( -iname "*.txt" -o -iname "*.md" \) -print0)

echo "Wrote index: $OUT"

Run it:

chmod +x build_index.sh
./build_index.sh data index.jsonl nomic-embed-text

---

4) Ask questions (retrieve + generate)

This script:

• Embeds your query

• Scores each chunk by cosine similarity using jq

• Picks the top 5 chunks

• Builds a prompt and asks the LLM (llama3.1) with the retrieved context

Save as ask.sh and make executable.

#!/usr/bin/env bash
set -euo pipefail

INDEX="${INDEX:-index.jsonl}"
LLM_MODEL="${LLM_MODEL:-llama3.1}"
EMBED_MODEL="${EMBED_MODEL:-nomic-embed-text}"

if [ ! -f "$INDEX" ]; then
  echo "Index not found: $INDEX (build it with build_index.sh)"
  exit 1
fi

if [ $# -lt 1 ]; then
  echo "Usage: $0 'your question here...'"
  exit 1
fi

QUESTION="$*"

if ! curl -s http://localhost:11434/api/tags >/dev/null; then
  echo "Ollama server not reachable at http://localhost:11434. Start it with: ollama serve"
  exit 1
fi

# Embed the query
q_emb=$(jq -n --arg m "$EMBED_MODEL" --arg prompt "$QUESTION" '{model:$m, prompt:$prompt}' | \
  curl -s http://localhost:11434/api/embeddings \
    -H "Content-Type: application/json" \
    -d @- | jq -c '.embedding')

# Score and pick top-k with jq (cosine similarity)
CONTEXT=$(
  jq --argjson q "$q_emb" -c '
    def dot(a;b): reduce range(0; a|length) as $i (0; . + (a[$i] * b[$i]));
    def norm(a): sqrt(dot(a;a));
    def cos(a;b): (dot(a;b)) / ((norm(a) * norm(b)) + 1e-9);

    . as $row | $row + {score: cos($row.embedding; $q)}
  ' "$INDEX" | \
  jq -s -r '
    sort_by(-.score)[:5]
    | map("* " + .file + " #" + (.id|tostring) + "\n" + .text)
    | join("\n\n---\n\n")
  '
)

PROMPT=$(cat <<EOF
You are a helpful assistant. Use the provided context to answer the question. If information is missing, say you don't know.

Question:
$QUESTION

Context:
$CONTEXT
EOF
)

# Ask the model (non-streaming for simplicity)
req=$(jq -n --arg model "$LLM_MODEL" --arg prompt "$PROMPT" '{model:$model, prompt:$prompt, stream:false}')
curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "$req" | jq -r '.response'

Run it:

chmod +x ask.sh
./ask.sh "How do I set a systemd user service to start automatically?"

You should get a focused answer backed by your indexed chunks.

---

5) Real-world example ideas

• Private project docs: Index your README, CHANGELOG, architecture notes, and ADRs. Query “How does service X authenticate?” and get code-level context.

• Homelab runbooks: Index your Ansible playbooks, backups notes, and incident logs. Ask “What’s the restore procedure for host Y?”

• Linux knowledge base: Dump select man pages to text and index them. Ask “What does ip rule do vs ip route?”

Example: indexing man pages

# Convert selected man pages to text
man bash | col -b > data/man_bash.txt
man systemctl | col -b > data/man_systemctl.txt

# Rebuild index and ask
./build_index.sh data index.jsonl nomic-embed-text
./ask.sh "How do I restart a user-level systemd service?"

---

Tips, performance, and model choices

• Model choices:

  • LLM: llama3.1 (balanced), mistral, qwen2, etc. Try smaller 7–8B models first for speed.
  • Embeddings: nomic-embed-text is a solid default for general text.

• Hardware: Ollama can use your GPU automatically if compatible; otherwise runs on CPU. Expect indexing to be the heavier step.

• Chunking: Adjust word size/overlap in the indexer to fit your content. Larger chunks = more context but slower retrieval.

• Hygiene: Keep chunks clean and de-duplicated. Extract text from PDFs/HTML cleanly to reduce noise.

• Reproducibility: Bake scripts into CI to rebuild the index whenever docs change.

---

Conclusion and next steps

You now have a minimal, entirely local RAG pipeline powered by Ollama and a few Bash utilities:

• Index your docs with build_index.sh

• Ask grounded questions with ask.sh

• Evolve from here: swap models, tune chunk sizes, or plug in a real vector DB if your corpus grows large.

Call to action:

• Point it at your own docs and try it today.

• Share a gist of your modified scripts and what models worked best for your use case.

• When you’re ready, add a small web UI or a TUI around ask.sh and make your private AI search a daily tool.