Posted on
• Artificial Intelligence

Artificial Intelligence Kubernetes Administration

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Kubernetes Administration: A Bash-first Playbook

If your AI services slow to a crawl at peak hours, your GPUs sit idle off-peak, or upgrades feel like russian roulette, you’re leaving performance and money on the table. AI workloads are spiky, GPU time is expensive, and clusters get noisy fast. The fix is not a shiny dashboard—it’s disciplined, automatable, Bash-friendly Kubernetes administration tuned for AI.

This guide gives you a practical workflow to run and operate AI workloads on Kubernetes using familiar command-line tools. You’ll set up a reliable admin toolbox, make your cluster GPU-aware, autoscale the right things, build observability that actually helps, and ship models safely. All commands are ready to paste.

Why AI + Kubernetes administration matters now

  • Predictable performance: Inference latencies and training throughput depend on correct resource requests, node features, and scheduling.

  • Cost control: GPUs only pay for themselves when utilized. Right-sizing and robust autoscaling matter.

  • Uptime during change: Canary and blue/green rollouts reduce risk when shipping new models and CUDA/driver changes.

  • Team velocity: A Bash-first workflow is repeatable, automatable, and works over SSH anywhere.

1) Set up your Bash admin toolbox

You’ll use kubectl, Helm, and jq constantly. Install them once, script forever.

Prerequisites (if you don’t have curl/gnupg yet):

# Debian/Ubuntu (apt)
sudo apt-get update
sudo apt-get install -y curl gnupg ca-certificates

# Fedora/RHEL/CentOS (dnf)
sudo dnf install -y curl gnupg2 ca-certificates 'dnf-command(config-manager)'

# openSUSE/SLES (zypper)
sudo zypper refresh
sudo zypper install -y curl gpg2 ca-certificates

Install kubectl:

# Debian/Ubuntu (apt) - official Kubernetes repo
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

# Fedora/RHEL/CentOS (dnf) - official Kubernetes repo
sudo dnf config-manager --add-repo https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
sudo dnf install -y kubectl

# openSUSE/SLES (zypper) - official Kubernetes repo
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
sudo zypper addrepo https://pkgs.k8s.io/core:/stable:/v1.30/rpm/ kubernetes
sudo zypper refresh
sudo zypper install -y kubectl

Install Helm:

# Debian/Ubuntu (apt)
curl -fsSL https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /etc/apt/keyrings/helm.gpg
echo "deb [signed-by=/etc/apt/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install -y helm

# Fedora/RHEL/CentOS (dnf)
sudo rpm --import https://baltocdn.com/helm/signing.asc
sudo dnf config-manager --add-repo https://baltocdn.com/helm/stable/rpm/helm.repo
sudo dnf install -y helm

# openSUSE/SLES (zypper)
sudo rpm --import https://baltocdn.com/helm/signing.asc
sudo zypper addrepo https://baltocdn.com/helm/stable/rpm/ helm-stable
sudo zypper refresh
sudo zypper install -y helm

Install jq (handy for Kubectl JSON):

# Debian/Ubuntu
sudo apt-get install -y jq

# Fedora/RHEL/CentOS
sudo dnf install -y jq

# openSUSE/SLES
sudo zypper install -y jq

Optional: bash-completion (quality of life):

# Debian/Ubuntu
sudo apt-get install -y bash-completion

# Fedora/RHEL/CentOS
sudo dnf install -y bash-completion

# openSUSE/SLES
sudo zypper install -y bash-completion

Quality-of-life aliases:

# Add to ~/.bashrc
alias k='kubectl'
complete -o default -F __start_kubectl k 2>/dev/null || true
alias kgp='kubectl get pods -o wide'
alias kgn='kubectl get nodes -o wide'
alias kctx='kubectl config current-context'

2) Make the cluster GPU‑aware (NVIDIA)

AI workloads need GPU discovery and scheduling. Configure nodes, then deploy the device plugin.

On each GPU node: install NVIDIA Container Toolkit and configure the runtime.

# Debian/Ubuntu (apt)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

# Fedora/RHEL/CentOS (dnf)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo tee /etc/pki/rpm-gpg/NVIDIA-GPG-KEY > /dev/null
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf clean expire-cache
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

# openSUSE/SLES (zypper)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
sudo zypper addrepo https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo
sudo rpm --import https://nvidia.github.io/libnvidia-container/gpgkey
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

Deploy the NVIDIA device plugin:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.15.0/nvidia-device-plugin.yml

Label GPU nodes (optional but useful for scheduling):

kubectl label node <gpu-node-name> accelerator=nvidia --overwrite

Request GPUs in your Pods/Deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pytorch-infer
spec:
  replicas: 1
  selector: { matchLabels: { app: pytorch-infer } }
  template:
    metadata: { labels: { app: pytorch-infer } }
    spec:
      nodeSelector:
        accelerator: nvidia
      containers:
      - name: server
        image: nvcr.io/nvidia/pytorch:24.04-py3
        resources:
          limits:
            nvidia.com/gpu: "1"
          requests:
            cpu: "1"
            memory: "4Gi"
        command: ["bash","-lc","python serve.py"]

Pro tip: always request CPU and memory with GPUs to prevent noisy-neighbor contention.

3) Autoscale the right things (and not the wrong ones)

  • Use HPA for CPU‑bound pre/post‑processing, gateways, and model routers.

  • Avoid HPA directly on GPU inference pods unless you have robust SLOs and warmup logic.

  • Use PriorityClasses so GPU inference evicts non-critical workloads first during pressure.

Install Metrics Server (required for HPA):

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update
helm upgrade --install metrics-server metrics-server/metrics-server -n kube-system

Create an HPA for a CPU-bound preprocessor:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: preproc-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: preproc
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65

Add a PriorityClass so GPU inference wins scheduling:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-gpu
value: 100000
globalDefault: false
description: "Prefer GPU inference pods over batch."
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pytorch-infer
spec:
  template:
    spec:
      priorityClassName: critical-gpu

If you’re on a cloud with node autoscaling, pair HPA with Cluster Autoscaler to grow/shrink GPU node groups based on pending GPU pods.

4) Ship models safely with Helm + canaries

Package your inference stack with Helm values for model versions, resources, and probes. Then canary before you go all-in.

Helm chart skeleton:

.
├── charts/
├── templates/
│   ├── deployment.yaml
│   └── service.yaml
└── values.yaml

values.yaml (toggle model version and traffic weight):

image: nvcr.io/nvidia/pytorch:24.04-py3
model:
  uri: s3://ml-bucket/resnet50-v2/
resources:
  cpu: "1"
  memory: "4Gi"
gpu: 1
canary:
  enabled: true
  weight: 10

Canary rollout:

helm upgrade --install infer charts/infer -n ai --create-namespace \
  --set image=nvcr.io/nvidia/pytorch:24.08-py3 \
  --set model.uri=s3://ml-bucket/resnet50-v3/ \
  --set canary.enabled=true --set canary.weight=20

kubectl rollout status deploy/infer -n ai

Lock SLOs with PodDisruptionBudget and readiness gates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: infer-pdb
spec:
  minAvailable: 90%
  selector:
    matchLabels:
      app: infer

Roll back instantly if error rates or latency spike:

helm rollback infer 1 -n ai

5) Observe what matters (GPUs, QPS, latency)

Install a Prometheus stack and scrape GPU metrics so you can see utilization, memory, ECC errors, and throttling.

Install Prometheus and Grafana:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Install NVIDIA DCGM exporter on GPU nodes:

helm repo add nvidia https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm upgrade --install dcgm-exporter nvidia/dcgm-exporter -n monitoring

Quick checks:

# Top GPU pods by GPU memory (requires dcgm-exporter)
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 &

# In PromQL UI:
# avg GPU utilization by node (0..100)
avg(dcgm_sm_occupancy) by (node)

# GPU memory used MiB by pod
sum by (pod) (dcgm_fb_used)

Fast debugging loop:

# List not-ready pods and why
kubectl get pods -A | awk '/0\/1|0\/[0-9]+/{print $1,$2}' | while read ns pod; do
  echo "== $ns/$pod =="; kubectl -n "$ns" describe pod "$pod" | sed -n '1,120p' | sed 's/^/  /'; echo; done

# Tail logs from all replicas matching a label
kubectl logs -n ai -l app=infer -f --max-log-requests=20

Set an availability SLO alert (example rule):

groups:

- name: ai-slo
  rules:
  - alert: GPUSaturationHigh
    expr: avg_over_time(dcgm_sm_occupancy[5m]) > 0.95
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU SM occupancy >95% for 10m"
      description: "Pods may be throttling; consider scaling out or revising batch sizes."

Real-world pattern: GPU pool isolation and cost control

  • Label production GPU nodes accelerator=nvidia,env=prod; dev/test nodes accelerator=nvidia,env=dev.

  • Use namespace ResourceQuotas so teams cannot over-request GPUs.

  • Use separate node groups (or taints/tolerations) so batch training never starves latency-critical inference.

Example quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-gpu-quota
  namespace: team-a
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
    requests.cpu: "32"
    requests.memory: "128Gi"

Conclusion and next steps

AI workloads punish sloppy ops. With a solid Bash toolbox, GPU-aware scheduling, targeted autoscaling, and GPU‑first observability, you’ll get higher utilization, lower latency, and safer rollouts.

Your next steps:

  • Install kubectl, Helm, and jq on your admin box.

  • GPU-enable your nodes and deploy the NVIDIA device plugin.

  • Add Metrics Server, Prometheus, and DCGM exporter for visibility.

  • Wrap your inference stack in a Helm chart and run a canary.

Want a turnkey, GitOps-ready version of this? Build these steps into a repo with environment folders (dev/stage/prod) and wire it to Argo CD or Flux. Then enjoy push‑button, auditable AI operations.

Related posts