Posted on
• Artificial Intelligence

Artificial Intelligence Kubernetes Case Studies

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence on Kubernetes: Practical Case Studies and a Bash-First Playbook

AI workloads have outgrown single GPUs and one-off scripts. Teams now need a repeatable way to serve, train, and observe models across environments—without hand-tuning machines or manually babysitting jobs. Kubernetes gives AI teams a common, automated control plane for CPU/GPU scheduling, autoscaling, networking, and security so the focus can return to models and data—not glue code.

In this article, we’ll walk through why Kubernetes is a strong fit for AI, and then dig into three concrete case studies you can adapt today. You’ll get copy-paste Bash and YAML, plus install steps for kubectl and Helm on apt, dnf, and zypper so you can get hands-on fast.

Why Kubernetes for AI is worth your time

  • Portability and consistency: The same containers and manifests run on your laptop, data center, or the cloud.

  • Elasticity and cost control: Scale pods up/down automatically; isolate GPU nodes for training; use spot/preemptible nodes for batch jobs.

  • Reproducibility: Containerized runtimes and declarative manifests make experiments and deployments traceable and repeatable.

  • Multi-tenancy and safety: Namespaces, RBAC, resource quotas, Pod Security, and NetworkPolicies keep teams productive and safe.

  • Ecosystem power: Mature operators and charts for distributed training (Ray), inference (Seldon Core, KServe), and observability (Prometheus/Grafana).

Prerequisites: Install kubectl and Helm

You’ll use kubectl to interact with your cluster and Helm to install operators and add-ons.

Note: Replace v1.30 with the Kubernetes minor version you target.

Debian/Ubuntu (apt):

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# kubectl
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

# Helm
curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install -y helm

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y dnf-plugins-core curl

# kubectl
cat << 'EOF' | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
EOF
sudo dnf install -y kubectl

# Helm
sudo rpm --import https://baltocdn.com/helm/signing.asc
cat << 'EOF' | sudo tee /etc/yum.repos.d/helm.repo
[helm]
name=Helm
baseurl=https://baltocdn.com/helm/stable/rpm
enabled=1
gpgcheck=1
gpgkey=https://baltocdn.com/helm/signing.asc
EOF
sudo dnf install -y helm

openSUSE/SLES (zypper):

sudo zypper install -y curl

# kubectl
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
sudo zypper ar https://pkgs.k8s.io/core:/stable:/v1.30/rpm/ kubernetes
sudo zypper refresh
sudo zypper install -y kubectl

# Helm
sudo rpm --import https://baltocdn.com/helm/signing.asc
sudo zypper ar https://baltocdn.com/helm/stable/rpm helm
sudo zypper refresh
sudo zypper install -y helm

Verify:

kubectl version --client
helm version

You’ll also need a Kubernetes cluster with the right node types (CPU and optionally GPU). Managed clusters (EKS/AKS/GKE), on-prem, or local (kind/k3d/minikube) all work.

Case Study 1: Real-time model serving with a platform operator (Seldon Core)

Problem/value: A product team wants to serve multiple models behind stable endpoints, roll out safely, and collect request metrics—without building a framework from scratch.

Approach: Use a mature inference operator (Seldon Core) that standardizes model servers, traffic routing, and telemetry. This keeps app teams focused on features while the platform team manages the control plane.

Install cert-manager (for webhooks) and Seldon Core via Helm:

# cert-manager (admission webhooks + certs)
helm repo add jetstack https://charts.jetstack.io
kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --set installCRDs=true

# Seldon Core operator
helm repo add seldon https://storage.googleapis.com/seldon-charts
kubectl create namespace seldon-system
helm install seldon-core seldon/seldon-core-operator \
  --namespace seldon-system \
  --set usageMetrics.enabled=true

Deploy a simple scikit-learn Iris classifier from Seldon’s public model bucket:

kubectl create namespace ml
cat << 'EOF' | kubectl apply -n ml -f -
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: sklearn-iris
spec:
  predictors:
  - name: default
    replicas: 1
    graph:
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/iris
EOF

Validate:

kubectl get pods -n ml
kubectl get svc -n ml -l seldon-deployment-id=sklearn-iris

Tip:

  • Use Helm values to enable canary releases and request logging.

  • Back the service with HPA to auto-scale on CPU or custom metrics.

What teams learned:

  • Standardizing on an operator accelerates onboarding and reduces one-off serving code.

  • Canary rollouts catch regressions before full traffic shifts.

Case Study 2: Distributed training and batch inference with Ray on Kubernetes

Problem/value: Data scientists need to scale Python-based training/evaluation jobs across many nodes without losing the ergonomics of native Python.

Approach: Use Ray with the KubeRay operator. Kubernetes handles node scaling and placement; Ray provides distributed primitives (Actors, Datasets, Train/Tune/Serve).

Install KubeRay:

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
kubectl create namespace ray
helm install kuberay-operator kuberay/kuberay-operator -n ray

Create a RayCluster (CPU example; swap in GPU nodes as needed):

cat << 'EOF' | kubectl apply -n ray -f -
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: ray-cluster
spec:
  rayVersion: "2.9.0"
  headGroupSpec:
    serviceType: ClusterIP
    rayStartParams:
      num-cpus: "1"
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.9.0
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
  workerGroupSpecs:
  - replicas: 2
    minReplicas: 1
    maxReplicas: 4
    groupName: small-group
    rayStartParams:
      num-cpus: "2"
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.9.0
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
EOF

Submit a RayJob that estimates Pi (replace with your training script):

cat << 'EOF' | kubectl apply -n ray -f -
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: pi-job
spec:
  entrypoint: >
    python -c "import ray,random,math; ray.init(); 
    @ray.remote
    def sample(n): 
        import random, math
        inside=0
        for _ in range(n): 
            x,y=random.random(),random.random()
            inside += (x*x + y*y) <= 1.0
        return inside
    futures=[sample.remote(250000) for _ in range(16)]
    inside=sum(ray.get(futures))
    print('Pi ~', 4.0 * inside / (16*250000))"
  rayClusterSpec:
    rayVersion: "2.9.0"
    headGroupSpec:
      rayStartParams: { num-cpus: "1" }
      template:
        spec:
          containers:
          - name: ray-head
            image: rayproject/ray:2.9.0
    submitterPodTemplate:
      spec:
        restartPolicy: Never
EOF

Watch job status and logs:

kubectl get rayjobs -n ray
kubectl logs -n ray job/pi-job -f --all-containers

GPU training? Add resources to the worker containers:

resources:
  limits:
    nvidia.com/gpu: 1

Make sure your cluster has GPU nodes and the NVIDIA device plugin (see next case).

What teams learned:

  • Ray on Kubernetes keeps Python-first workflows while unlocking elastic scaling.

  • Preemptible/spot nodes plus checkpointing cut training cost without derailing runs.

Case Study 3: GPU-aware scheduling for vision and generative models

Problem/value: A CV/GenAI team needs predictable GPU access for training and inference, while sharing a cluster with other workloads.

Approach: Use Node Feature Discovery (NFD) to label hardware, and the NVIDIA device plugin so pods can request GPUs with a simple resource limit. This gives clean scheduling and avoids “GPU drift.”

Install NFD and the NVIDIA device plugin:

# Node Feature Discovery
helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts
helm repo update
kubectl create namespace nfd
helm install nfd nfd/node-feature-discovery -n nfd

# NVIDIA device plugin
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
kubectl create namespace gpu
helm install nvidia-device-plugin nvdp/nvidia-device-plugin -n gpu

Run a GPU test pod with CUDA and nvidia-smi:

cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
  - name: nvidia-smi
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    command: ["/bin/bash","-c"]
    args: ["nvidia-smi && python -c 'import torch,sys;print(\"torch cuda:\", torch.cuda.is_available())' || true"]
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

Check output:

kubectl logs gpu-test -f

Tip:

  • Use taints/tolerations and node selectors to isolate GPU pools.

  • For A100/H100 partitioning, enable MIG and match on NFD labels.

  • Combine with HPA/VPA and a queueing layer for fairness and high utilization.

What teams learned:

  • Declarative GPU requests simplify multi-team sharing.

  • NFD labels drive smart placement for model families (e.g., tensor cores vs. memory-bound).

Best practices distilled

  • Standardize interfaces: Pick one serving operator and one training/distribution substrate per org (e.g., Seldon Core + Ray). Fewer knobs, faster teams.

  • Codify SLOs: Use probes, resource requests/limits, and autoscalers. If it’s not in YAML, it won’t scale reliably.

  • Secure by default: Namespaces, RBAC, NetworkPolicies, Pod Security, and image signing/attestation.

  • Observe everything: Prometheus + Grafana for system and model metrics, plus structured logs. Consider request-level logging and model drift metrics.

  • Optimize costs: Right-size requests, enable scale-to-zero for idle endpoints, and checkpoint to survive preemptions.

Conclusion and CTA

Kubernetes doesn’t “do AI” for you—but it removes the toil of scheduling, scaling, and operating the systems around your models. The three case studies above show a path from ad-hoc scripts to reproducible, elastic, and secure AI platforms using tools you can install today with kubectl and Helm.

Your next steps:

  • Stand up a test cluster and complete Case Study 3 to validate GPU scheduling.

  • Pick either Case Study 1 (serving) or Case Study 2 (training) as your first standardized workflow.

  • Wrap the winning pattern in a Helm chart or internal template so every new project starts production-ready.

Have a particular model or constraint in mind? Tell me your hardware, cloud/on-prem, and framework, and I’ll sketch a tailored manifest and rollout plan you can paste into your repo.

Related posts