Artificial Intelligence Homelab Case Studies (A Bash‑First Guide)

Ever wished you could run private, low‑latency AI at home—without sending your data to the cloud or paying per‑token fees? Good news: modern open models and lightweight tooling make this not only possible, but practical on modest hardware. In this article, you’ll get three real homelab case studies—with copy/paste‑ready Bash and minimal moving parts—that deliver immediate value: a local AI chat server, an “ask my docs” retrieval system, and experiment tracking with MLflow.

What you’ll get:

• Clear problem → value framing for each case study

• Why a homelab for AI makes sense (privacy, control, cost)

• 3 actionable builds with commands for apt, dnf, and zypper

• Reusable snippets you can adapt to your own setup

---

Why AI in a homelab?

• Privacy and control: Keep sensitive documents, camera feeds, and prompts on hardware you own.

• Latency and reliability: Local models answer instantly—even offline.

• Cost predictability: One‑time hardware + open models beats recurring API bills.

• Skill building: Infra, automation, containers, Python, and MLOps—all hands‑on.

Design principles used here:

• Prefer rootless containers (Podman) and system packages.

• Keep data on disk in simple, inspectable formats (text, JSON, SQLite).

• Start CPU‑first; add GPU acceleration later if you need it.

---

Case Study 1: Local LLM Chat (Ollama + Open WebUI)

Goal: A private, web‑based chat UI backed by a local LLM.

What you’ll build:

• Ollama model server on the host

• Open WebUI in a container connecting to Ollama

1.1 Install prerequisites

• apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y podman curl

• dnf (Fedora/RHEL/CentOS Stream):

sudo dnf install -y podman curl

• zypper (openSUSE):

sudo zypper install -y podman curl

1.2 Install and run Ollama

curl -fsSL https://ollama.com/install.sh | sh
# Start the service (if the installer didn’t already):
sudo systemctl enable --now ollama || ollama serve &

Pull a small, capable model (adjust to your RAM/CPU):

ollama pull llama3.2:3b

Quick sanity test:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Say hello in one short sentence."}'

1.3 Launch Open WebUI via Podman

Use host networking so the container can reach Ollama on 11434:

podman run -d --name openwebui --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  ghcr.io/open-webui/open-webui:latest

Open http://localhost:8080 in your browser. Select the model (e.g., llama3.2:3b) and chat.

Pro tip:

• CPU‑only works fine for smaller models. Add a GPU later if desired (check Ollama docs for CUDA/ROCm requirements).

---

Case Study 2: Ask‑My‑Docs (RAG) with Local Embeddings + Ollama

Goal: Ask natural‑language questions about your PDFs/TXT/MD—fully offline.

Approach:

• Convert documents to plain text

• Embed and index chunks locally (NumPy + sentence-transformers)

• Retrieve top matches and ask Ollama with that context

2.1 Install prerequisites

• apt:

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git poppler-utils

• dnf:

sudo dnf install -y python3 python3-venv python3-pip git poppler-utils

• zypper:

sudo zypper install -y python3 python3-pip git poppler-tools

2.2 Project layout and environment

mkdir -p ~/homelab-rag/{docs,corpus,index}
cd ~/homelab-rag
python3 -m venv .venv
. .venv/bin/activate
pip install -U pip
pip install sentence-transformers numpy tqdm httpx

2.3 Convert documents to text (Bash)

Place PDFs/TXT/MD into ~/homelab-rag/docs, then run:

cat > to_text.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
in_dir="${1:-docs}"
out_dir="${2:-corpus}"
mkdir -p "$out_dir"
shopt -s nullglob
for f in "$in_dir"/*; do
  base="$(basename "$f")"
  name="${base%.*}"
  ext="${base##*.}"
  case "$ext" in
    pdf)
      pdftotext -layout "$f" "$out_dir/$name.txt"
      ;;
    md|txt)
      cp "$f" "$out_dir/$name.txt"
      ;;
    *)
      echo "Skipping unsupported file: $f" >&2
      ;;
  esac
done
echo "Text corpus in: $out_dir"
EOF
chmod +x to_text.sh

./to_text.sh docs corpus

2.4 Embed corpus and build a lightweight index (Python)

cat > embed.py << 'PY'
import os, json, glob
import numpy as np
from tqdm import tqdm
from sentence_transformers import SentenceTransformer

MODEL = "sentence-transformers/all-MiniLM-L6-v2"
CHUNK = 1000
OVERLAP = 150

def chunk_text(t, size=CHUNK, overlap=OVERLAP):
    out, i = [], 0
    while i < len(t):
        out.append(t[i:i+size])
        i += size - overlap
    return [c for c in out if c.strip()]

def main(corpus_dir="corpus", index_dir="index"):
    os.makedirs(index_dir, exist_ok=True)
    model = SentenceTransformer(MODEL)
    texts, meta = [], []
    for path in tqdm(sorted(glob.glob(os.path.join(corpus_dir, "*.txt")))):
        with open(path, "r", encoding="utf-8", errors="ignore") as f:
            text = f.read()
        for j, chunk in enumerate(chunk_text(text)):
            texts.append(chunk)
            meta.append({"file": os.path.basename(path), "chunk_id": j, "preview": chunk[:180].replace("\n"," ")})
    if not texts:
        print("No texts found. Run the to_text.sh step first.")
        return
    print(f"Embedding {len(texts)} chunks...")
    emb = model.encode(texts, normalize_embeddings=True, convert_to_numpy=True)
    np.save(os.path.join(index_dir, "embeddings.npy"), emb)
    with open(os.path.join(index_dir, "meta.jsonl"), "w", encoding="utf-8") as out:
        for m in meta:
            out.write(json.dumps(m, ensure_ascii=False) + "\n")
    with open(os.path.join(index_dir, "texts.jsonl"), "w", encoding="utf-8") as out:
        for t in texts:
            out.write(json.dumps({"text": t}, ensure_ascii=False) + "\n")
    print(f"Index written to: {index_dir}")
if __name__ == "__main__":
    main()
PY

python embed.py

2.5 Query with local retrieval + Ollama (Python)

cat > ask.py << 'PY'
import os, sys, json, numpy as np, httpx
from sentence_transformers import SentenceTransformer

OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434")
MODEL = os.environ.get("MODEL", "llama3.2:3b")
TOPK = int(os.environ.get("TOPK", "5"))

def load_index(index_dir="index"):
    emb = np.load(os.path.join(index_dir, "embeddings.npy"))
    meta = [json.loads(l) for l in open(os.path.join(index_dir, "meta.jsonl"), encoding="utf-8")]
    texts = [json.loads(l)["text"] for l in open(os.path.join(index_dir, "texts.jsonl"), encoding="utf-8")]
    return emb, meta, texts

def main():
    if len(sys.argv) < 2:
        print("Usage: python ask.py \"your question here\"")
        sys.exit(1)
    question = sys.argv[1]
    emb, meta, texts = load_index()
    enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    q = enc.encode([question], normalize_embeddings=True, convert_to_numpy=True)[0]
    sims = emb @ q
    top_idx = sims.argsort()[-TOPK:][::-1]
    ctx = "\n\n---\n\n".join([texts[i] for i in top_idx])
    prompt = f"Use the context to answer concisely.\n\nContext:\n{ctx}\n\nQuestion: {question}\nAnswer:"
    payload = {"model": MODEL, "prompt": prompt, "stream": False}
    try:
        r = httpx.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=120)
        r.raise_for_status()
        print(json.loads(r.text)["response"].strip())
    except Exception as e:
        print(f"[WARN] Ollama call failed: {e}")
        print("Top contexts:\n", ctx[:2000])
if __name__ == "__main__":
    main()
PY

python ask.py "Summarize the key points across my documents."

You now have an offline, private Q&A tool over your docs. Tune CHUNK/OVERLAP/TOPK for your content.

---

Case Study 3: Track Experiments with MLflow (Local UI + SQLite)

Goal: Keep your training runs organized, compare metrics, and version artifacts—no cloud needed.

3.1 Install prerequisites

If you completed Case Study 2, you already have Python and venv. Otherwise:

• apt:

sudo apt update
sudo apt install -y python3 python3-venv python3-pip

• dnf:

sudo dnf install -y python3 python3-venv python3-pip

• zypper:

sudo zypper install -y python3 python3-pip

3.2 Create an MLflow workspace

mkdir -p ~/ai-mlflow
cd ~/ai-mlflow
python3 -m venv .venv
. .venv/bin/activate
pip install -U pip
pip install mlflow scikit-learn pandas numpy

Run the server (SQLite backend, local artifacts):

mlflow server \
  --backend-store-uri sqlite:///mlflow.db \
  --default-artifact-root "$(pwd)/artifacts" \
  --host 0.0.0.0 --port 5000

Open http://localhost:5000

3.3 Log a simple training run

cat > train.py << 'PY'
import mlflow, mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True, as_frame=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)

n_estimators, depth = 200, 5
with mlflow.start_run():
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", depth)
    clf = RandomForestClassifier(n_estimators=n_estimators, max_depth=depth, random_state=42)
    clf.fit(Xtr, ytr)
    acc = accuracy_score(yte, clf.predict(Xte))
    mlflow.log_metric("accuracy", acc)
    mlflow.sklearn.log_model(clf, "model")
    print("Accuracy:", acc)
PY

python train.py

Refresh the MLflow UI to compare runs.

3.4 Optional: run MLflow as a user service (systemd)

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/mlflow.service << 'EOF'
[Unit]
Description=MLflow Tracking Server (user)
After=network-online.target

[Service]
Type=simple
WorkingDirectory=%h/ai-mlflow
ExecStart=%h/ai-mlflow/.venv/bin/mlflow server --backend-store-uri sqlite:///%h/ai-mlflow/mlflow.db --default-artifact-root %h/ai-mlflow/artifacts --host 0.0.0.0 --port 5000
Restart=on-failure

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now mlflow

---

Homelab Tips That Pay Off

• Backups: Snapshot your ~/homelab-rag and ~/ai-mlflow folders. Your embeddings, indexes, and experiment history are assets.

• Security: Bind to localhost or firewall ports if you open UIs (Open WebUI 8080, MLflow 5000).

• Resources: Start with small models (e.g., 3B parameter LLMs). Scale only when you need better quality.

• Automation: Wrap commands into Makefiles or systemd units for repeatable starts.

---

Conclusion and Next Steps

You just stood up:

• A private local LLM with a friendly web UI

• An offline “ask my docs” RAG pipeline

• A proper MLOps tracking UI with MLflow

Next steps:

• Add a GPU and switch Ollama to a larger model for richer answers.

• Point the RAG pipeline at wikis, logs, and code to supercharge your ops.

• Containerize MLflow or reverse‑proxy behind Caddy/NGINX for multi‑user access.

• Tie it all together with Ansible so you can rebuild in minutes.

Have a favorite homelab AI use‑case you want to see? Reply with your hardware and goals—I’ll tailor the next guide with Bash you can paste and run.