AI-Ready Knowledge Management on Linux (Bash-first, Private, Fast)

If your terminal could talk, it would probably say: “You already solved this last month.” Most teams and solo devs drown in docs, tickets, wikis, READMEs, and chat logs. The problem isn’t that we lack information; it’s that we can’t quickly find and reuse it at the moment of need.

This post shows how to turn a plain Git folder into an AI-augmented knowledge base you can search semantically and query with a local LLM—using simple Bash and lightweight, local-first tools. You get faster answers, better reuse of hard-won knowledge, and you keep your data on your machine.

Highlights:

• Organize knowledge with simple conventions (Git + folders)

• Lightning-fast keyword search with ripgrep/fzf

• Semantic search using embeddings (local, no cloud)

• Retrieval-Augmented Generation (RAG) with a local LLM via Ollama

• Everything scriptable in Bash

Why AI knowledge management on Linux is worth it

• Time-to-answer matters: Context switching and re‑discovering past fixes kills flow. Search should be <5 seconds.

• Semantic vs. keyword: Grep is great for exact strings; embeddings find “what you mean,” not just “what you typed.”

• Privacy and control: Keep source docs and indexes local. No SaaS lock-in or data exfiltration.

• Composable Unix style: Glue together small tools. Replace pieces later without re-platforming.

Prerequisites (install via your package manager)

Run the block for your distro to get the basics.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git curl jq ripgrep fzf python3 python3-venv python3-pip sqlite3 pandoc

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y git curl jq ripgrep fzf python3 python3-pip python3-virtualenv sqlite pandoc

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y git curl jq ripgrep fzf python3 python3-pip sqlite3 pandoc

Optional local LLM runner (Ollama):

curl -fsSL https://ollama.com/install.sh | sh
# Then pull a model, e.g.:
ollama pull llama3.1:8b

Note: We’ll use Python inside a local virtual environment for the embedding and vector index.

Create and activate a venv:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install "chromadb>=0.5" "fastembed>=0.3"

Step 1 — Structure a Git-backed knowledge base

Keep it boring and durable. Text files in Git are easy to search, diff, and back up.

mkdir -p ~/kb/{inbox,docs,notes,snippets}
cd ~/kb && git init
printf ".venv/\n.kb_index/\n__pycache__/\n" >> .gitignore

Adopt simple naming rules:

• notes/YYYY-MM-DD-topic.md

• docs//.md

• snippets//.md

Optional front matter at the top of files:

---
tags: [linux, networking]
owner: ops
updated: 2026-07-01
---

Template helper to start a new note:

note() {
  local f=~/kb/notes/$(date +%F)-${1:-idea}.md
  [ -e "$f" ] || cat > "$f" <<'EOF'
---
tags: []
owner: me
updated:
---
# Title

Context:

Steps:

References:
EOF
  ${EDITOR:-vi} "$f"
}

Step 2 — Instant keyword search with ripgrep + fzf

Exact string search remains the first-line tool. Make it feel like muscle memory.

Install was covered above. Try:

rg -n --hidden -S "timeout|retry" ~/kb

Fuzzy “open match in $EDITOR” helper:

kb() {
  local root=~/kb
  local target
  target=$(rg -n --hidden --line-number --no-heading "${*:-.}" "$root" \
    | fzf --ansi --height 80% --reverse --preview 'bat --style=numbers --color=always --line-range :200 {1} --highlight-line {2}' \
           --preview-window=right,70% \
    | awk -F: '{print $1":"$2}')
  [ -z "$target" ] && return
  ${EDITOR:-vi} "+$(echo "$target" | cut -d: -f2)" "$(echo "$target" | cut -d: -f1)"
}

Usage:

kb "ssh key"

Step 3 — Add semantic search with a local vector index

We’ll embed your notes and store vectors in a local ChromaDB index using FastEmbed (small, CPU-friendly). No external services.

Create a Python script to index and query:

Save as kb_index.py:

#!/usr/bin/env python3
import argparse, os, sys, uuid, pathlib, re
from typing import List, Iterable, Tuple
import chromadb
from fastembed import TextEmbedding

KB_INDEX = os.environ.get("KB_INDEX", os.path.join(os.getcwd(), ".kb_index"))
ALLOWED_EXT = {".md", ".txt", ".rst", ".sh", ".py", ".conf", ".ini"}

def read_text(path: pathlib.Path) -> str:
    try:
        return path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return ""

def chunk_text(text: str, max_chars: int = 800, overlap: int = 100) -> List[str]:
    text = re.sub(r"\n{3,}", "\n\n", text.strip())
    chunks, start = [], 0
    while start < len(text):
        end = min(start + max_chars, len(text))
        chunks.append(text[start:end])
        if end == len(text): break
        start = end - overlap
    return [c for c in chunks if c.strip()]

class FastEmbedder:
    def __init__(self, model="BAAI/bge-small-en-v1.5"):
        self.model = model
        self._embedder = TextEmbedding(model_name=model)
    def __call__(self, texts: List[str]) -> List[List[float]]:
        # fastembed returns a generator of numpy arrays
        return [vec.tolist() for vec in self._embedder.embed(texts)]

def get_collection():
    client = chromadb.PersistentClient(path=KB_INDEX)
    emb = FastEmbedder()
    return client.get_or_create_collection(
        name="kb",
        metadata={"hnsw:space": "cosine"},
        embedding_function=emb
    )

def iter_files(root: pathlib.Path) -> Iterable[pathlib.Path]:
    for p in root.rglob("*"):
        if p.is_file() and p.suffix.lower() in ALLOWED_EXT:
            yield p

def do_index(paths: List[str], clear: bool):
    col = get_collection()
    if clear:
        try:
            col.delete(where={})
        except Exception:
            pass
    for base in paths:
        root = pathlib.Path(base).expanduser().resolve()
        for f in iter_files(root):
            text = read_text(f)
            if not text.strip():
                continue
            chunks = chunk_text(text)
            ids = [f"{f.as_posix()}::{i}" for i in range(len(chunks))]
            metadatas = [{"source": f.as_posix(), "chunk": i} for i in range(len(chunks))]
            col.upsert(ids=ids, metadatas=metadatas, documents=chunks)
    print("Indexing complete.", file=sys.stderr)

def do_query(q: str, k: int, format: str):
    col = get_collection()
    res = col.query(query_texts=[q], n_results=k, include=["documents","metadatas","distances"])
    docs = res["documents"][0]
    metas = res["metadatas"][0]
    dists = res["distances"][0]
    if format == "json":
        import json
        out = []
        for doc, meta, dist in zip(docs, metas, dists):
            out.append({"source": meta["source"], "chunk": meta["chunk"], "distance": dist, "text": doc})
        print(json.dumps(out, ensure_ascii=False, indent=2))
    else:
        for i, (doc, meta, dist) in enumerate(zip(docs, metas, dists), 1):
            print(f"### {i}. {meta['source']} (chunk {meta['chunk']}, score {dist:.4f})")
            print(doc.strip())
            print()

def main():
    ap = argparse.ArgumentParser(description="KB index/query")
    sub = ap.add_subparsers(dest="cmd", required=True)

    ap_i = sub.add_parser("index", help="Index directories/files")
    ap_i.add_argument("paths", nargs="+", help="Paths to index")
    ap_i.add_argument("--clear", action="store_true", help="Clear existing entries first")

    ap_q = sub.add_parser("query", help="Query the KB")
    ap_q.add_argument("question", help="Natural-language question or keywords")
    ap_q.add_argument("-k", type=int, default=5, help="Number of results")
    ap_q.add_argument("--format", choices=["md","json","plain"], default="md")

    args = ap.parse_args()
    if args.cmd == "index":
        do_index(args.paths, args.clear)
    elif args.cmd == "query":
        do_query(args.question, args.k, args.format)

if __name__ == "__main__":
    main()

Make it executable:

chmod +x kb_index.py

Build your first index:

./kb_index.py index ~/kb

Try a query:

./kb_index.py query "rotate SSH keys and update known_hosts" -k 5

Tip: Re-run indexing when you add/update notes. For large KBs, you can add basic file-change detection later; start simple now.

Step 4 — Ask natural-language questions with local RAG (Ollama)

We’ll glue semantic retrieval to a small local LLM via Ollama. The script pulls top chunks from the vector index and feeds them into the model with a grounded prompt.

Ensure Ollama is installed and you have a model:

ollama pull llama3.1:8b

Create a Bash helper called ask:

ask() {
  local q="$*"
  if [ -z "$q" ]; then
    echo "Usage: ask <question>"
    return 1
  fi
  local ctx
  ctx=$(./kb_index.py query "$q" -k 6 --format md)

  local prompt
  prompt=$(cat <<EOF
You are a concise assistant. Answer the user's question using ONLY the context below. 
If the answer isn't in the context, say "I don't know from the KB." 
Cite file paths when applicable.

Context:
$ctx

Question: $q

Answer:
EOF
)
  ollama run llama3.1:8b -p "$prompt"
}

Example:

ask "How do I set up a systemd unit to auto-restart a service on failure?"

This returns an answer based on your indexed notes/snippets. If nothing relevant is found, you’ll get “I don’t know from the KB,” which is exactly what you want—no hallucinations.

Real-world examples

• On-call runbooks: Index your incident guides and one-liners. In the middle of an outage, “ask” beats hunting through bookmarks.

• Config snippets: Keep small, commented examples for Nginx, iptables, SSH, and systemd; retrieve them by intent, not just filename.

• Project knowledge: Pair RFCs and decision logs with code snippets and commands. Ask “Why did we disable NUMA on staging?” and get context with citations.

Operational tips

• Small, frequent commits to your KB; treat it like code.

• Prefer text (Markdown, code) over binary formats. For PDFs, consider pre-converting to text with pandoc if needed:

  • apt: sudo apt install -y pandoc
  • dnf: sudo dnf install -y pandoc
  • zypper: sudo zypper install -y pandoc Then: pandoc file.pdf -t plain -o file.txt (works best for text-based PDFs).

• Start with CPU-friendly models; upgrade later if you need more reasoning power.

• Script everything. Your future self will thank you.

Conclusion and next steps

You don’t need a giant platform to get real value from AI in day-to-day engineering. A Git folder, a fast index, and a small local model will recover lost knowledge, speed up recurring tasks, and reduce context-switching.

Your next steps: 1) Initialize your KB folder and commit 10–20 of your most-used notes/snippets. 2) Install the prerequisites and run the indexer. 3) Wire up the ask function and answer 3 real problems today. 4) Share the workflow with your team and standardize note templates.

Have questions or want a follow-up post on incremental indexing, citation formatting, or team-sharing patterns? Ask—and include your distro so I can tailor the commands.