Posted on
• Artificial Intelligence

Artificial Intelligence Helm Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Helm Best Practices: Ship Reliable AI on Kubernetes (from Your Bash Shell)

AI apps are hungry for GPUs, data, and fast iteration. Kubernetes can give you scale and resilience—but YAML sprawl and environment drift can turn your day into fire-fighting. Helm changes that. With a few sane patterns, you can package, version, and promote AI workloads across dev, staging, and prod without guesswork.

This guide explains why Helm is the right tool for AI on Kubernetes and gives you actionable best practices you can apply today, from GPU-aware scheduling to safe rollouts. You’ll also get ready-to-run Bash commands and distro‑specific install steps.

Why Helm for AI?

  • Reproducibility: Charts freeze app, infra, and configuration into versioned, auditable bundles.

  • Environment parity: Values files let you cleanly separate dev/stage/prod without forking templates.

  • Upgrades and rollbacks: One command to migrate, diff, and roll back when something misbehaves.

  • Team velocity: Share best-practice templates for GPUs, storage, and observability across teams.

Prerequisites (Bash-friendly)

You need kubectl and Helm on your workstation, plus a Kubernetes cluster with node access. Replace v1.30 below to match your cluster’s minor version.

Install kubectl:

  • Debian/Ubuntu (apt)
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
  • Fedora/RHEL/CentOS (dnf)
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
cat <<'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
EOF
sudo dnf install -y kubectl
  • openSUSE/SLES (zypper)
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
sudo zypper addrepo https://pkgs.k8s.io/core:/stable:/v1.30/rpm/ kubernetes
sudo zypper refresh
sudo zypper install -y kubectl

Install Helm:

  • Debian/Ubuntu (apt)
sudo apt-get update
sudo apt-get install -y apt-transport-https curl gnupg
curl https://baltocdn.com/helm/stable/debian/helm-signing.gpg | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install -y helm
  • Fedora/RHEL/CentOS (dnf)
sudo rpm --import https://baltocdn.com/helm/stable/rpm/repodata/repomd.xml.key
cat <<'EOF' | sudo tee /etc/yum.repos.d/helm.repo
[helm]
name=Helm Stable
baseurl=https://baltocdn.com/helm/stable/rpm
enabled=1
gpgcheck=1
gpgkey=https://baltocdn.com/helm/stable/rpm/repodata/repomd.xml.key
EOF
sudo dnf install -y helm
  • openSUSE/SLES (zypper)
sudo rpm --import https://baltocdn.com/helm/stable/rpm/repodata/repomd.xml.key
sudo zypper addrepo https://baltocdn.com/helm/stable/rpm/ helm-stable
sudo zypper refresh
sudo zypper install -y helm

Quick checks:

helm version
kubectl version --client
kubectl get nodes -o wide

GPU clusters need the NVIDIA device plugin. If you use GPUs:

helm repo add nvidia https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install nvidia-device-plugin nvidia/k8s-device-plugin -n kube-system

5 Helm Best Practices for AI Workloads

1) Treat charts as products: version, sign, and store in an OCI registry

  • Pin app and chart versions. Set appVersion in Chart.yaml to the container tag you actually deploy.

  • Push charts to OCI registries for immutable, scoped access control.

Example:

# Package and push a chart to GHCR
helm package charts/ai-inference
helm registry login ghcr.io -u YOUR_GH_USER
helm push ai-inference-0.1.0.tgz oci://ghcr.io/YOUR_ORG/charts

# Pull with exact version in CI/CD
helm pull oci://ghcr.io/YOUR_ORG/charts/ai-inference --version 0.1.0

Why it matters: Teams can promote the exact artifact from staging to prod, preventing “works on my machine” drift.

2) Make GPU scheduling first‑class in values.yaml

AI services need predictable GPU, CPU, and memory reservations, plus tolerations and selectors.

values.gpu.yaml:

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
    nvidia.com/gpu: 1
  limits:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: 1

nodeSelector:
  nvidia.com/gpu.present: "true"

tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Apply with:

helm upgrade --install ai-svc ./charts/ai-inference -f values.gpu.yaml

Why it matters: Without requests/limits and proper scheduling, autoscaling, binpacking, and QoS become unpredictable—kryptonite for latency-sensitive inference.

3) Design for data gravity: PVCs, object storage, and model preload

Models and datasets are big. Make data locality explicit.

values.storage.yaml:

persistence:
  enabled: true
  existingClaim: ""         # or leave blank to create
  size: 200Gi
  storageClassName: "fast-ssd"

env:
  - name: MODEL_ID
    value: meta-llama/Llama-2-7b-hf

volumeMounts:
  - name: models
    mountPath: /models

volumes:
  - name: models
    persistentVolumeClaim:
      claimName: {{ include "ai-inference.fullname" . }}-models

initContainers:
  - name: fetch-model
    image: ghcr.io/yourorg/model-sync:latest
    env:
      - name: MODEL_ID
        valueFrom:
          secretKeyRef:
            name: hf-credentials
            key: model-id
    volumeMounts:
      - name: models
        mountPath: /models

Why it matters: Preloading models on PVCs slashes cold-starts. Using initContainers decouples model acquisition from your main image and lets you rotate credentials safely.

4) Keep environments clean: layer values and externalize secrets

  • Use a base values.yaml plus small env overlays:
helm upgrade --install ai-svc ./charts/ai-inference \
  -f values.yaml \
  -f values.gpu.yaml \
  -f values.prod.yaml
  • Don’t commit secrets. Create them once and reference them:
kubectl create secret generic hf-credentials \
  --from-literal=model-id=meta-llama/Llama-2-7b-hf \
  -n ai

values.yaml (reference only):

envFrom:
  - secretRef:
      name: hf-credentials

Why it matters: Values layering prevents copy-paste environments. Secrets live in your cluster’s secret manager, not in Git.

5) Upgrade safely: diff, probe, and roll back fast

  • Lint and dry-run against the API server before you touch live objects:
helm lint ./charts/ai-inference
helm template ai-svc ./charts/ai-inference -f values.prod.yaml | kubectl apply -n ai --dry-run=server -f -
  • Use helm-diff to preview changes:
helm plugin install https://github.com/databus23/helm-diff
helm diff upgrade ai-svc ./charts/ai-inference -f values.prod.yaml
  • Ensure health checks are accurate (don’t just probe TCP 80): values.yaml
readinessProbe:
  httpGet:
    path: /healthz
    port: http
  initialDelaySeconds: 10
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /livez
    port: http
  initialDelaySeconds: 30
  periodSeconds: 10
  • Roll back instantly if SLOs degrade:
helm rollback ai-svc  # or specify a revision: helm history ai-svc

Why it matters: Diffs prevent surprise mutations; probes ensure rollouts only proceed when your model is actually serving; rollbacks buy you time.

Real‑world quickstart: Scaffold an AI inference chart

Create a new chart and wire in AI‑friendly defaults.

1) Scaffold:

helm create ai-inference

2) Set your image and GPU resources in values.yaml:

image:
  repository: nvcr.io/nvidia/tritonserver
  tag: 23.10-py3
  pullPolicy: IfNotPresent

command: ["/opt/tritonserver/bin/tritonserver"]
args:
  - "--model-repository=/models"

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
    nvidia.com/gpu: 1
  limits:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: 1

nodeSelector:
  nvidia.com/gpu.present: "true"

tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

service:
  type: ClusterIP
  port: 8000

volumeMounts:
  - name: models
    mountPath: /models
volumes:
  - name: models
    persistentVolumeClaim:
      claimName: ai-models-pvc

3) Create a models PVC and deploy:

kubectl apply -n ai -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ai-models-pvc
  namespace: ai
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 200Gi
  storageClassName: fast-ssd
EOF

helm upgrade --install ai-svc ./ai-inference -n ai --create-namespace
kubectl rollout status deploy/ai-svc -n ai

4) Verify:

kubectl get pods -n ai -o wide
kubectl logs -n ai deploy/ai-svc | tail -n +1

Optional: expose via a LoadBalancer or Ingress once you’ve validated internal traffic.

Conclusion and Next Steps

Helm turns AI deployment from artisanal YAML into a disciplined release process:

  • Package and store charts like real artifacts.

  • Encode GPU, data, and health as values—not tribal knowledge.

  • Upgrade with diffs and rollbacks, not hope.

Your next step:

  • Install kubectl and Helm (commands above).

  • Scaffold a chart with helm create and add GPU resources, storage, and probes.

  • Wire CI to run helm lint, helm template | kubectl --dry-run=server, and helm diff.

  • Promote the same chart through environments with layered values.

Have a favorite AI stack (Ray, Triton, TGI) you want Helmified? Tell me what you’re running, and I’ll share a tuned values.yaml you can drop into prod.

Related posts