Artificial Intelligence Home Kubernetes Lab: Private, Reproducible, and Fast

What if you could run modern AI models at home, on your own hardware, with production-grade orchestration and zero monthly cloud bills? That’s the promise of a home Kubernetes lab for AI: privacy, control, and skills you can take to work.

In this guide, you’ll build a practical, single-node Kubernetes setup that runs local LLM inference with Ollama and an optional web UI. You’ll learn how to do it CPU-only (works on almost any machine) and how to add GPU acceleration if you have an NVIDIA card. Along the way you’ll install only what you need, ship with YAML, and keep it all reproducible.

Why this matters:

• Privacy and control: keep your data on your LAN.

• Cost efficiency: reuse your workstation or homelab box.

• Real-world skills: learn the same Kubernetes patterns used in production.

• Reproducibility: declarative configs beat “works on my machine.”

Note: All commands are for Linux. When installing via a package manager, you’ll find apt, dnf, and zypper instructions.

---

What You’ll Build

• A local Kubernetes cluster (simple path: kind for CPU-only; GPU path: k3s)

• Kubernetes-native LLM inference using Ollama

• Optional: Open WebUI front-end for chat

• Optional: NVIDIA GPU support for faster inference

Minimum host specs (CPU-only):

• 4+ cores, 16 GB RAM, 50+ GB SSD

• Linux (Ubuntu/Debian, Fedora/RHEL, openSUSE/SLES)

• Internet access to pull container images

Recommended for GPU:

• NVIDIA GPU (Turing or newer)

• Recent NVIDIA driver

• NVIDIA Container Toolkit

---

1) Install prerequisites

You’ll need curl, git, a container runtime (Docker), kubectl, and Helm.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl git docker.io ca-certificates gnupg lsb-release
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker

# kubectl
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl

# Helm
curl -fsSL https://baltocdn.com/helm/signing.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" \
  | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm

Fedora/RHEL/CentOS (dnf):

sudo dnf -y install curl git ca-certificates gnupg2

# Docker (Moby)
sudo dnf -y install moby-engine moby-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

# kubectl
cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
EOF
sudo dnf -y install kubectl

# Helm
cat <<'EOF' | sudo tee /etc/yum.repos.d/helm.repo
[helm]
name=Helm
baseurl=https://baltocdn.com/helm/stable/rpm
enabled=1
gpgcheck=1
gpgkey=https://baltocdn.com/helm/signing.asc
EOF
sudo dnf -y install helm

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y curl git ca-certificates gpg2

# Docker
sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

# kubectl (RPM repo)
sudo zypper addrepo --refresh https://pkgs.k8s.io/core:/stable:/v1.30/rpm/ kubernetes
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
sudo zypper install -y kubectl

# Helm (RPM repo)
sudo zypper addrepo --refresh https://baltocdn.com/helm/stable/rpm helm
sudo rpm --import https://baltocdn.com/helm/signing.asc
sudo zypper install -y helm

---

2) Create a local Kubernetes cluster

CPU-only, quick start with kind (runs Kubernetes in Docker). This is perfect for trying things out.

Install kind:

curl -fsSLo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/
# For ARM64 hosts, replace -amd64 with -arm64 in the URL.

Create a cluster and pre-map two NodePorts for apps:

cat > kind-ai.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:

- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080
    protocol: TCP
  - containerPort: 30081
    hostPort: 30081
    protocol: TCP
EOF

kind create cluster --name ai --config kind-ai.yaml
kubectl cluster-info --context kind-ai
kubectl get nodes -o wide

Tip: If you plan to keep this running 24/7, consider k3s instead of kind (kind is developer-focused and resets on host reboot unless you script it). See the GPU path below for k3s.

---

3) Optional: Add NVIDIA GPU support (k3s path)

GPU support with kind is possible but fiddly. For smoother GPU, use k3s on the host (containerd + DaemonSets work well).

Install k3s (single-node):

curl -sfL https://get.k3s.io | sh -
sudo kubectl get nodes

Install NVIDIA Container Toolkit.

Debian/Ubuntu (apt):

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
curl -fsSL https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
sudo apt update
sudo apt install -y nvidia-container-toolkit

Fedora/RHEL/CentOS (dnf):

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /etc/pki/rpm-gpg/libnvidia-container.gpg
distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
curl -fsSL https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.repo \
  | sudo tee /etc/yum.repos.d/libnvidia-container.repo
sudo dnf install -y nvidia-container-toolkit

openSUSE/SLES (zypper):

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /etc/pki/trust/anchors/libnvidia-container.gpg
sudo update-ca-certificates || sudo update-ca-certificates --fresh || true
distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
sudo zypper addrepo --refresh https://nvidia.github.io/libnvidia-container/${distribution}/libnvidia-container.repo
sudo zypper install -y nvidia-container-toolkit

Enable CDI and restart k3s (CDI is the modern, simpler way for K8s to discover GPUs):

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo systemctl restart k3s

Deploy the NVIDIA device plugin (detects GPUs and exposes them to Pods):

helm repo add nvidia https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade --install nvidia-device-plugin nvidia/k8s-device-plugin
kubectl -n default get ds nvidia-device-plugin -o wide

Verify GPU node resources:

kubectl describe node | grep -i nvidia -A2 -B2

You should see a resource like nvidia.com/gpu.

---

4) Deploy AI inference with Ollama (+ optional Web UI)

We’ll deploy Ollama to run LLMs locally. This works great on CPU and can use GPU automatically when present.

Create a namespace and storage:

kubectl create ns ai || true

# Persistent volume claim for models (adjust size as needed)
cat > ollama-pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
  namespace: ai
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
EOF
kubectl apply -f ollama-pvc.yaml

Ollama Deployment and Service:

CPU-first manifest (works on kind and k3s). If you have GPU, see the commented GPU section in the same file.

cat > ollama-deploy.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai
spec:
  replicas: 1
  selector:
    matchLabels: { app: ollama }
  template:
    metadata:
      labels: { app: ollama }
    spec:
      # Uncomment this section if you have an NVIDIA GPU and the device plugin installed:
      # containers:
      # - name: ollama
      #   image: ollama/ollama:latest
      #   ports: [{containerPort: 11434}]
      #   volumeMounts:
      #   - name: models
      #     mountPath: /root/.ollama
      #   resources:
      #     limits:
      #       nvidia.com/gpu: "1"
      #   env:
      #   - name: OLLAMA_KEEP_ALIVE
      #     value: "5m"
      #   - name: OLLAMA_NUM_PARALLEL
      #     value: "1"
      #   securityContext:
      #     runAsUser: 0
      #     runAsGroup: 0
      #     allowPrivilegeEscalation: false
      #     capabilities:
      #       drop: ["ALL"]
      #   volumeDevices: []
      # volumes:
      # - name: models
      #   persistentVolumeClaim:
      #     claimName: ollama-models
      # restartPolicy: Always

      # CPU-only container (default)
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports: [{containerPort: 11434}]
        volumeMounts:
        - name: models
          mountPath: /root/.ollama
        env:
        - name: OLLAMA_KEEP_ALIVE
          value: "5m"
        - name: OLLAMA_NUM_PARALLEL
          value: "1"
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: ollama-models
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ai
spec:
  selector: { app: ollama }
  ports:
  - name: http
    port: 11434
    targetPort: 11434
  type: NodePort
EOF

kubectl apply -f ollama-deploy.yaml

Expose it on your host:

• If you used kind with the earlier port mappings, patch NodePort to 30080:

kubectl -n ai patch svc ollama -p '{"spec":{"ports":[{"name":"http","port":11434,"targetPort":11434,"nodePort":30080}],"type":"NodePort"}}'

• On k3s, NodePort is directly reachable at hostIP:nodePort (check the assigned NodePort: kubectl -n ai get svc ollama -o wide).

Pull a small model and run a test:

# Pull a small, fast model (CPU-friendly)
curl -X POST http://localhost:30080/api/pull -d '{"name":"qwen2.5:0.5b"}'

# Generate text
curl -s http://localhost:30080/api/generate -d '{"model":"qwen2.5:0.5b","prompt":"Explain Kubernetes like I am five."}' | jq -r .response

Optional: Add Open WebUI for a friendly chat interface:

cat > openwebui.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openwebui
  namespace: ai
spec:
  replicas: 1
  selector:
    matchLabels: { app: openwebui }
  template:
    metadata:
      labels: { app: openwebui }
    spec:
      containers:
      - name: web
        image: ghcr.io/open-webui/open-webui:latest
        env:
        - name: OLLAMA_BASE_URL
          value: http://ollama.ai.svc.cluster.local:11434
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: openwebui
  namespace: ai
spec:
  selector: { app: openwebui }
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  type: NodePort
EOF

kubectl apply -f openwebui.yaml

# Map to port 30081 on kind (or note the NodePort on k3s)
kubectl -n ai patch svc openwebui -p '{"spec":{"ports":[{"name":"http","port":8080,"targetPort":8080,"nodePort":30081}],"type":"NodePort"}}'

Visit:

• kind: http://localhost:30081

• k3s: http://:

Pick the “qwen2.5:0.5b” model (or another you’ve pulled) and start chatting.

---

5) Operate it like a pro (ingress, updates, monitoring)

• Ingress and TLS:

  • On k3s, Traefik is included. Create an Ingress to route a local domain (e.g., ai.local) to your services.
  • For TLS, add cert-manager via Helm and use a local CA or DNS challenge for public domains.

• Updates and rollbacks:

  • Update containers by changing the image tag and applying YAML.
  • Use kubectl rollout status deployment/ollama -n ai to monitor updates.
  • Roll back with kubectl rollout undo.

• Resource control:

  • For CPU-only, set requests/limits to avoid starving your desktop.
  • For GPU, set resources.limits.nvidia.com/gpu: "1" and optionally taint/label GPU nodes.

• Backups:

  • Your models live in the PVC. Snapshot/backup the underlying storage or use a tool like Velero.

• Persistence and boot:

  • kind is great for experiments; for always-on homelab nodes, k3s is easier to keep running across reboots.

---

Real-World Uses You Can Try This Weekend

• Private note assistant: run a small LLM locally to summarize notes.

• Code helper: use a 7B model on GPU for fast code completion offline.

• Family Q&A kiosk: Open WebUI on a tablet pointed at your cluster.

• Photo captioning: pair with CLIP/BLIP containers and batch jobs in Kubernetes.

---

Conclusion and Next Steps (CTA)

You now have a working AI home lab: a Kubernetes cluster, private LLM inference with Ollama, and an optional web UI—running entirely on your own hardware. From here you can:

• Swap in bigger models and tune resources.

• Add GPU acceleration (k3s + NVIDIA toolkit).

• Put an Ingress and TLS in front for safe LAN access.

• Monitor with Prometheus/Grafana and alert on resource usage.

Your next step: pull a model you’ll actually use, wire up Open WebUI, and live with it for a week. See what breaks, then iterate. That’s how real platforms are built—one declarative YAML at a time.

If you want a follow-up, ask for:

• A Helm-charted version of this setup

• An Ingress + TLS walkthrough for a local domain

• GPU benchmarking tips and model sizing guidance

Happy hacking!