Posted on
• Artificial Intelligence

Artificial Intelligence K3s Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence K3s Projects: Practical AI on Lightweight Kubernetes

Want to run real AI workloads on a Raspberry Pi 4, an Intel NUC, or a spare VM without turning your homelab into a power plant? K3s, the lightweight Kubernetes from Rancher, makes it easy to deploy AI services that are reproducible, portable, and surprisingly fast—even on edge hardware.

The problem: AI stacks are often heavy, complex, and demand pricey GPUs. The value: with K3s, you get a tiny, production-grade Kubernetes that can schedule containerized AI apps (LLMs, image classifiers, speech recognizers) anywhere—from a single node to a fleet.

Below is a practical, bash-friendly guide with 3–5 actionable projects to get you from zero to AI on K3s, including concrete manifests, curl tests, and minimal moving parts.

Why K3s for AI at the edge

  • Tiny footprint: single binary, bundled containerd, lower memory overhead than full Kubernetes.

  • Multi-arch: great for x86_64 and ARM (Raspberry Pi, Jetson).

  • Reproducible deployments: YAML/Helm installs beat “works on my laptop.”

  • Portable scaling: start with one node, add more later, same manifests.

  • Network/load balancing: with MetalLB, you get LoadBalancer IPs on bare metal/LAN.

Prerequisites

  • Linux host(s) with root/sudo.

  • Recommended: 2+ CPU cores, 4+ GB RAM for basic inference; more for LLMs.

  • Outbound internet to pull images/charts.

Install baseline CLI tools (curl, git, tar, gzip, iptables, conntrack):

  • apt (Debian/Ubuntu):
sudo apt update
sudo apt install -y curl git tar gzip iptables conntrack ca-certificates
  • dnf (Fedora/RHEL/CentOS Stream):
sudo dnf -y install curl git tar gzip iptables conntrack-tools ca-certificates
  • zypper (openSUSE/SLE):
sudo zypper refresh
sudo zypper install -y curl git tar gzip iptables conntrack ca-certificates

1) Foundation: Install K3s + Helm + MetalLB

Install K3s (single-node):

curl -sfL https://get.k3s.io | sh -
sudo systemctl status k3s --no-pager

Get kubectl access (K3s ships with kubectl and a kubeconfig):

mkdir -p "$HOME/.kube"
sudo cp /etc/rancher/k3s/k3s.yaml "$HOME/.kube/config"
sudo chown "$(id -u)":"$(id -g)" "$HOME/.kube/config"
kubectl get nodes

Install Helm:

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

Install MetalLB (to get LoadBalancer IPs on bare-metal/LAN):

helm repo add metallb https://metallb.github.io/metallb
helm repo update
helm install metallb metallb/metallb -n metallb-system --create-namespace

Pick a free IP range on your LAN (outside DHCP). Then apply:

cat <<'EOF' | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240-192.168.1.250  # change to your LAN range
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-advertisement
  namespace: metallb-system
EOF

2) Project: Triton Inference Server (CPU) + ONNX model

Goal: Serve a real image classification model via HTTP on your LAN using NVIDIA Triton (CPU mode is fine).

Create a namespace and deploy Triton with an initContainer that downloads an ONNX model into an in-cluster model repository:

kubectl create namespace ai

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton
  namespace: ai
  labels:
    app: triton
spec:
  replicas: 1
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
    spec:
      volumes:
        - name: model-repo
          emptyDir: {}
      initContainers:
        - name: fetch-model
          image: alpine:3.19
          command: ["/bin/sh","-lc"]
          args:
            - |
              set -eux
              mkdir -p /models/squeezenet/1
              wget -O /models/squeezenet/1/model.onnx \
                https://github.com/onnx/models/raw/main/vision/classification/squeezenet/model/squeezenet1.0-12.onnx
          volumeMounts:
            - name: model-repo
              mountPath: /models
      containers:
        - name: triton
          image: ghcr.io/triton-inference-server/server:23.10-py3
          args: ["tritonserver","--model-repository=/models","--http-port=8000","--grpc-port=8001","--metrics-port=8002"]
          ports:
            - name: http
              containerPort: 8000
            - name: grpc
              containerPort: 8001
            - name: metrics
              containerPort: 8002
          readinessProbe:
            httpGet:
              path: /v2/health/ready
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
            - name: model-repo
              mountPath: /models
---
apiVersion: v1
kind: Service
metadata:
  name: triton
  namespace: ai
  labels:
    app: triton
spec:
  type: LoadBalancer
  selector:
    app: triton
  ports:
    - name: http
      port: 8000
      targetPort: 8000
    - name: grpc
      port: 8001
      targetPort: 8001
    - name: metrics
      port: 8002
      targetPort: 8002
EOF

Find the service IP and test:

kubectl -n ai get svc triton
TRITON_IP=$(kubectl -n ai get svc triton -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -fsS "http://$TRITON_IP:8000/v2/health/ready" && echo "Triton ready"
curl -fsS "http://$TRITON_IP:8000/v2/models/squeezenet" | jq .

Tip: Real inference with Triton uses JSON + binary tensors over HTTP/gRPC. Start here with health checks, then integrate a client when ready.

3) Project: Run a small local LLM with llama.cpp server

Goal: Expose a tiny LLM API on your LAN using CPU-only inference (works on modest boxes). You supply a GGUF model file.

On the host, place a GGUF model at /opt/llama-models/model.gguf (for example, a small TinyLlama or Phi-2-quantized GGUF you’ve downloaded):

sudo mkdir -p /opt/llama-models
# Copy your GGUF file to /opt/llama-models/model.gguf
ls -lh /opt/llama-models/

Deploy a Pod and LoadBalancer Service:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llama-models-pv
spec:
  capacity:
    storage: 8Gi
  accessModes: ["ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /opt/llama-models
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llama-models-pvc
  namespace: ai
spec:
  accessModes: ["ReadOnlyMany"]
  resources:
    requests:
      storage: 8Gi
  volumeName: llama-models-pv
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama
  namespace: ai
  labels:
    app: llama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama
  template:
    metadata:
      labels:
        app: llama
    spec:
      containers:
        - name: server
          image: ghcr.io/ggerganov/llama.cpp:server
          command: ["/bin/sh","-lc"]
          args:
            - |
              set -eux
              llama-server -m /models/model.gguf --host 0.0.0.0 --port 8080 --ctx-size 2048 --parallel 2
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: models
              mountPath: /models
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: llama-models-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: llama
  namespace: ai
  labels:
    app: llama
spec:
  type: LoadBalancer
  selector:
    app: llama
  ports:
    - name: http
      port: 8080
      targetPort: 8080
EOF

Test a basic completion:

LLAMA_IP=$(kubectl -n ai get svc llama -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -s "http://$LLAMA_IP:8080/completion" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Write a haiku about K3s.","n_predict":64,"temperature":0.7}'

Note: For faster inference, use more quantized models or GPU acceleration (see optional notes below).

4) Project: Observe your AI with Prometheus + Grafana

Goal: Instrumentation you can trust. Triton exposes Prometheus metrics (port 8002). We’ll install a monitoring stack and scrape those metrics.

Install the kube-prometheus-stack chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
kubectl -n monitoring get pods

Tell Prometheus to scrape Triton via a ServiceMonitor:

cat <<'EOF' | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: triton-sm
  namespace: monitoring
spec:
  namespaceSelector:
    matchNames: ["ai"]
  selector:
    matchLabels:
      app: triton
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s
EOF

Access Grafana:

  • Get the admin password:
kubectl -n monitoring get secret monitoring-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
  • Port-forward Grafana to your workstation (or expose with a LoadBalancer if you prefer):
kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
# Then browse http://localhost:3000 (user: admin, password: shown above)

Now watch request rates, latencies, and resource usage while you hit Triton or the LLM server.

Optional: Notes on GPUs and multi-node

  • GPUs: If you have NVIDIA hardware, install the NVIDIA driver and NVIDIA Container Toolkit on the host, then the Kubernetes NVIDIA device plugin in the cluster. K3s works well with it. Exact package names differ by distro/version—follow your GPU vendor’s docs. Afterward, add tolerations/limits like nvidia.com/gpu: 1 to your Pods.

  • Multi-node: Join more agents to your K3s server with a single token and reuse the same manifests. Add nodeSelectors or affinities to steer LLMs to beefier nodes.

Troubleshooting quick hits

  • Services not getting IPs: ensure your MetalLB IP pool matches your LAN and doesn’t conflict with DHCP.

  • Timeouts pulling images: check DNS/firewall; try crictl info and crictl pull on the node.

  • LLM too slow: choose a smaller GGUF (q4_k_m or q5_k_m), reduce context size, or add GPU.

Conclusion and Call to Action

You don’t need a datacenter to build and ship AI. With K3s plus a few well-chosen containers, you can:

  • Serve computer vision models via Triton.

  • Host a local LLM API with llama.cpp.

  • Observe everything with Prometheus and Grafana.

Pick one project above, paste in the manifests, and watch it go. From there, wire up a simple CLI, a cron job, or a web UI—then iterate. When you’re ready, push the same YAML to a bigger box or a small cluster and scale out with confidence.

If you found this useful, try turning one of these into a reusable Helm chart for your lab—or share your results and tweaks with the community. Happy hacking!

Related posts