Build an Artificial Intelligence Linux Homelab

Own your models. Keep your data. Learn the stack end to end. An AI homelab on Linux lets you experiment with large language models (LLMs), speech-to-text, and search—all on hardware you control. No API keys, no rate limits, no data leaving your network. In this guide you’ll build a minimal, production-ish AI lab that runs on a single Linux box using containers and a Python toolbox.

What you’ll get:

• A reproducible base with containers and a Python environment

• A local LLM service you can chat with (Ollama)

• A lightweight vector database for RAG (Qdrant) and a small Python RAG script

• A fast local transcription pipeline (Whisper via faster-whisper)

• Install instructions for apt, dnf, and zypper where packages are used

Note: GPU acceleration is optional. Everything here also runs on CPU (slower but simpler). You can add NVIDIA/AMD acceleration later.

Why a homelab for AI?

• Privacy and cost control: Run models locally, avoid usage-based billing and data exfiltration.

• Reproducibility: Containers + pinned dependencies = fewer surprises.

• Performance you can feel: Optimize your own stack (I/O, CPU/GPU, RAM) for your workloads.

• Skills that transfer: Learn Linux, containers, systemd, Python packaging, and model ops.

Hardware quick guide

• CPU: 6–12 cores modern x86_64 is a great start.

• RAM: 32 GB is comfortable; 16 GB works for smaller models.

• GPU (optional): 8–24 GB VRAM helps with bigger or faster models (NVIDIA easiest today).

• Storage: 1 TB NVMe recommended (datasets/models are large).

• Form factor: Mini PC, used workstation, or SFF server. Keep noise and power in mind.

Step 1 — Prepare the OS: essentials and updates

Update your system and install build tools, Git, curl, Python, and common utilities.

Debian/Ubuntu (apt):

sudo apt update
sudo apt upgrade -y
sudo apt install -y build-essential git curl wget unzip pkg-config cmake \
  python3 python3-venv python3-pip pipx
# If pipx package is missing, fallback:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf upgrade -y
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git curl wget unzip pkg-config cmake \
  python3 python3-pip python3-venv pipx
# If pipx package is missing, fallback:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath

openSUSE (zypper):

sudo zypper refresh
sudo zypper update -y
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y git curl wget unzip pkgconfig cmake \
  python3 python3-pip python3-virtualenv pipx
# If pipx package is missing, fallback:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath

Create a workspace and a Python virtual environment:

mkdir -p ~/ai-lab && cd ~/ai-lab
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel

Tip: Add source ~/ai-lab/.venv/bin/activate to your shell rc for convenience.

Step 2 — Containers you can trust (Docker or Podman)

Containers make your AI services portable and easy to update.

Option A: Docker

Debian/Ubuntu (apt):

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release; echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
# Log out/in or run: newgrp docker

Fedora (dnf):

sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Log out/in or run: newgrp docker

openSUSE (zypper):

sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Log out/in or run: newgrp docker

Option B: Podman (rootless by default)

# apt
sudo apt install -y podman
# dnf
sudo dnf install -y podman
# zypper
sudo zypper install -y podman

Quick test:

docker run --rm hello-world  # or: podman run --rm hello-world

GPU users (optional, NVIDIA):

• Install a recent NVIDIA driver using your distro’s recommended method.

• For Docker GPU access, install NVIDIA Container Toolkit:

# Debian/Ubuntu example
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then run containers with --gpus all. AMD users can explore ROCm-enabled images for specific projects.

Step 3 — Your first LLM service with Ollama (containerized)

Ollama is a simple way to run and swap local LLMs.

Start Ollama (CPU-only):

docker run -d --name ollama -p 11434:11434 \
  -v ollama:/root/.ollama \
  --restart unless-stopped \
  ollama/ollama:latest

Start with NVIDIA GPU (optional):

docker run -d --name ollama -p 11434:11434 \
  -v ollama:/root/.ollama \
  --gpus all \
  --restart unless-stopped \
  ollama/ollama:latest

Pull a model and chat via API:

# Pull a small, capable model (adjust name as desired)
curl http://localhost:11434/api/pull -d '{"name":"qwen2.5:3b"}'
# Simple prompt
curl -s http://localhost:11434/api/generate -d '{
  "model":"qwen2.5:3b",
  "prompt":"Explain containers to a new Linux user in 3 sentences."
}' | jq -r '.response'

Real-world uses:

• Private chat and summarization of notes

• Generate bash one-liners (and learn by inspecting them)

• Draft emails or project docs offline

Tip: Persist the ollama volume to fast storage; models are large.

Step 4 — Add search with a vector database (Qdrant) + a tiny RAG script

Spin up Qdrant:

docker run -d --name qdrant -p 6333:6333 \
  -v qdrant:/qdrant/storage \
  --restart unless-stopped \
  qdrant/qdrant:latest

Install minimal Python deps in your venv:

source ~/ai-lab/.venv/bin/activate
python -m pip install "sentence-transformers>=3" "qdrant-client>=1.9"

Index a folder of Markdown/text files and query it:

python << 'PY'
import os, glob
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct
from sentence_transformers import SentenceTransformer

docs_dir = os.path.expanduser("~/ai-lab/docs")
os.makedirs(docs_dir, exist_ok=True)
# Drop a few .md or .txt files into docs_dir before running

client = QdrantClient(host="localhost", port=6333)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
dim = model.get_sentence_embedding_dimension()
coll = "lab_docs"

try:
    client.delete_collection(coll)
except:
    pass
client.recreate_collection(coll, vectors_config=VectorParams(size=dim, distance=Distance.COSINE))

points = []
pid = 0
for path in glob.glob(os.path.join(docs_dir, "**/*.*"), recursive=True):
    if not path.lower().endswith((".md",".txt")): 
        continue
    text = open(path, "r", errors="ignore").read()
    if not text.strip():
        continue
    vec = model.encode([text])[0].tolist()
    points.append(PointStruct(id=pid, vector=vec, payload={"path": path}))
    pid += 1

if points:
    client.upsert(collection_name=coll, points=points)
    print(f"Indexed {len(points)} docs.")

query = "How do I back up my homelab?"
qvec = model.encode([query])[0].tolist()
hits = client.search(collection_name=coll, query_vector=qvec, limit=3)
for h in hits:
    print(round(h.score,3), h.payload["path"])
PY

Push the top hit(s) into your LLM prompt for Retrieval-Augmented Generation (RAG). You can wire this up with a small Flask/FastAPI service next.

Real-world uses:

• Search your wiki, runbooks, and code snippets

• RAG-boosted chat for your own knowledge base

Step 5 — Local speech-to-text with faster-whisper

Install and transcribe:

source ~/ai-lab/.venv/bin/activate
python -m pip install "faster-whisper>=1.0"
python << 'PY'
from faster_whisper import WhisperModel
# Choices: tiny, base, small, medium, large-v3; start small
model = WhisperModel("small", compute_type="int8")  # auto-selects CPU; use "int8_float16" for GPU
segments, info = model.transcribe("sample.wav")
print(f"Language: {info.language}, Prob: {info.language_probability:.2f}")
for s in segments:
    print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")
PY

Real-world uses:

• Transcribe meeting notes or voicemails locally

• Build searchable archives of audio lectures or podcasts

GPU note: With a supported GPU and drivers, faster-whisper can run much faster by selecting a GPU compute type, for example compute_type="float16".

Bonus — Compose, persistence, and updates

Use Docker Compose to start your AI services together:

cat > ~/ai-lab/docker-compose.yml << 'YML'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    restart: unless-stopped
    # For NVIDIA GPU, uncomment:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - capabilities: ["gpu"]

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    ports: ["6333:6333"]
    volumes: ["qdrant:/qdrant/storage"]
    restart: unless-stopped

volumes:
  ollama:
  qdrant:
YML

docker compose -f ~/ai-lab/docker-compose.yml up -d

Keep things updated:

docker pull ollama/ollama:latest && docker pull qdrant/qdrant:latest
docker compose -f ~/ai-lab/docker-compose.yml up -d

Optional monitoring:

# apt
sudo apt install -y netdata
# dnf
sudo dnf install -y netdata
# zypper
sudo zypper install -y netdata
sudo systemctl enable --now netdata

Troubleshooting tips

• Memory pressure: Use swap and smaller models (e.g., 3B–7B quantized).

• Disk space: Put model volumes on NVMe; prune old images: docker image prune -a.

• CPU-only builds too slow? Try GPU acceleration or quantized models.

• Build errors in Python: Ensure build-essential/Development Tools and cmake are installed.

Conclusion and next steps (CTA)

You now have a working AI homelab: a local LLM service (Ollama), a vector DB (Qdrant) with a tiny RAG pipeline, and fast transcription (faster-whisper), all on Linux. From here:

• Add a thin web UI for RAG and chat (FastAPI + HTMX or a simple React/Vue frontend).

• Experiment with different models (code assistants, vision, embeddings).

• Automate backups of volumes and configs (restic, borg, or ZFS snapshots).

• Harden your services if you expose them (firewall, reverse proxy, TLS, auth).

Pick one improvement, implement it today, and you’ll be well on your way to a private, powerful AI stack you truly own.