Posted on
• Artificial Intelligence

Artificial Intelligence Infrastructure Automation Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Infrastructure Automation Best Practices (for Bash-first DevOps)

AI stacks are evolving faster than most teams can manually keep up with. One missed driver version, an unpinned container tag, or a snowflake node can burn days of GPU time—and your budget. The fix is not just better scripts; it’s disciplined automation across the entire AI lifecycle.

This guide distills field-tested best practices you can apply today from a Linux terminal. You’ll get practical Bash-friendly workflows, IaC patterns, and install commands for apt, dnf, and zypper wherever tools are cited.

Why this matters

  • Reproducibility: AI experiments must be easy to rerun months later.

  • Velocity: Provision GPUs, storage, and networks in minutes—not weeks.

  • Cost control: Automation makes it easy to right-size and tear down.

  • Compliance: Versioned, reviewable code beats ad-hoc shell history.

5 best practices you can implement now

1) Describe everything as code (IaC + CaC)

Keep infra and configuration in version control so you can audit, roll back, and clone environments predictably.

  • Infra as Code (IaC): Terraform for cloud/GPU nodes, VPCs, security groups.

  • Config as Code (CaC): Ansible for packages, runtime config, users, and hardening.

  • Repo hygiene: Use Makefiles or small Bash wrappers to standardize operations.

Minimal repo skeleton:

ai-infra/
├─ terraform/         # networks, subnets, GPU nodes, EBS/NFS, ...
├─ ansible/
│  ├─ inventories/
│  ├─ roles/
│  └─ playbooks/
├─ containers/
│  └─ training/
├─ k8s/
│  ├─ base/
│  └─ overlays/
└─ Makefile

Install core tooling:

  • apt (Debian/Ubuntu):
sudo apt update
sudo apt install -y git make ansible jq
# Terraform (HashiCorp apt repo)
sudo apt install -y gnupg software-properties-common wget
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y terraform
  • dnf (Fedora/RHEL/CentOS Stream):
sudo dnf install -y git make ansible jq dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/$(. /etc/os-release && echo $ID)/hashicorp.repo
sudo dnf install -y terraform
  • zypper (openSUSE/SLES):
sudo zypper refresh
sudo zypper install -y git make ansible jq gpg2
sudo rpm --import https://rpm.releases.hashicorp.com/gpg
# openSUSE repo for HashiCorp
sudo zypper addrepo https://rpm.releases.hashicorp.com/opensuse/hashicorp.repo
sudo zypper refresh
sudo zypper install -y terraform

Example: minimal Terraform for a GPU VM (cloud-agnostic pseudo, adapt to your provider):

# terraform/main.tf (example with variables)
terraform {
  required_version = ">= 1.6.0"
}

provider "aws" { region = var.region }

resource "aws_instance" "gpu" {
  ami           = var.gpu_ami
  instance_type = "g5.2xlarge"
  subnet_id     = var.subnet_id
  tags = { role = "ai-gpu", env = var.env }
}

output "gpu_ip" { value = aws_instance.gpu.public_ip }

2) Standardize containers and pin versions

Containers are the contract between data scientists and ops. Pin everything (CUDA, cuDNN, PyTorch/TensorFlow, OS) to avoid “works on my machine.”

Prefer rootless Podman for security, or Docker if required.

Install Podman (recommended):

  • apt:
sudo apt update
sudo apt install -y podman buildah skopeo
  • dnf:
sudo dnf install -y podman buildah skopeo
  • zypper:
sudo zypper refresh
sudo zypper install -y podman buildah skopeo

Optional: Install Docker Engine

  • apt:
sudo apt update
sudo apt install -y docker.io
sudo usermod -aG docker $USER
  • dnf (Fedora often packages Docker as moby):
sudo dnf install -y moby-engine docker-compose-plugin
sudo usermod -aG docker $USER
  • zypper:
sudo zypper refresh
sudo zypper install -y docker
sudo usermod -aG docker $USER

Enable GPU in containers via NVIDIA Container Toolkit:

  • apt:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
 | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
 | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
  • dnf:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
 | sed 's#rpm https://#rpm [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
 | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf clean expire-cache
sudo dnf install -y nvidia-container-toolkit
  • zypper:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
 | sed 's#rpm https://#rpm [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
 | sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit

Example pinned Dockerfile for PyTorch:

# containers/training/Dockerfile
FROM nvcr.io/nvidia/pytorch:24.05-py3       # pin exact tag
ENV PIP_NO_CACHE_DIR=1 \
    PYTHONUNBUFFERED=1
RUN pip install --no-deps --upgrade pip==24.0 \
 && pip install torchmetrics==1.4.0 hydra-core==1.3.2

Build and run (Podman):

cd containers/training
podman build -t registry.example.com/ai/train:24.05 .
podman run --rm --security-opt=label=disable --hooks-dir=/usr/share/containers/oci/hooks.d \
  --device nvidia.com/gpu=all \
  -v $PWD:/workspace -w /workspace \
  registry.example.com/ai/train:24.05 python your_script.py

With Docker:

docker build -t registry.example.com/ai/train:24.05 containers/training
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all -v $PWD:/workspace -w /workspace registry.example.com/ai/train:24.05 python your_script.py

Real-world payoff: Pinning base images and deps stopped a team’s “Monday works, Friday breaks” problem after upstream PyPI updates changed default solver behavior.

3) Make GPU node setup idempotent

Avoid snowflake servers. Use Ansible to converge hosts to a known-good state—repeatably.

Simplified Ansible playbook to enable containerized GPU workloads:

# ansible/playbooks/gpu-node.yml

- hosts: gpu_nodes
  become: true
  tasks:
    - name: Ensure baseline packages
      package:
        name: [curl, git, jq]
        state: present

    - name: Install NVIDIA Container Toolkit repo and package (Debian/Ubuntu)
      when: ansible_pkg_mgr == "apt"
      shell: |
        set -euo pipefail
        distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor | tee /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg >/dev/null
        curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
         | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
         | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
        apt-get update
        apt-get install -y nvidia-container-toolkit

    - name: Install NVIDIA Container Toolkit (RPM: Fedora/RHEL/CentOS)
      when: ansible_pkg_mgr == "dnf"
      shell: |
        set -euo pipefail
        distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor | tee /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg >/dev/null
        curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
         | sed 's#rpm https://#rpm [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
         | tee /etc/yum.repos.d/nvidia-container-toolkit.repo
        dnf clean expire-cache
        dnf install -y nvidia-container-toolkit

    - name: Install NVIDIA Container Toolkit (SUSE/openSUSE)
      when: ansible_pkg_mgr == "zypper"
      shell: |
        set -euo pipefail
        distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor | tee /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg >/dev/null
        curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
         | sed 's#rpm https://#rpm [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
         | tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
        zypper -n refresh
        zypper -n install nvidia-container-toolkit

Verification helpers:

nvidia-smi || echo "Driver not found (ensure vendor-supported driver is installed)"
podman run --rm --device nvidia.com/gpu=all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
# or with Docker:
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

Tip: Keep the OS driver lifecycle documented and automated per distro using vendor-supported repos. Apply changes via Ansible and test on a canary GPU node first.

4) Orchestrate GPUs with Kubernetes (and schedule sanely)

Kubernetes gives you bin-packing, isolation, and repeatability. Add the NVIDIA device plugin so Pods can request GPUs explicitly.

Install kubectl:

  • apt:
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl
  • dnf:
sudo dnf install -y kubernetes-client || sudo dnf install -y kubectl
  • zypper:
sudo zypper refresh
sudo zypper install -y kubectl

Install Helm:

  • apt:
curl -fsSL https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | \
  sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm
  • dnf:
sudo dnf install -y helm
  • zypper:
sudo zypper refresh
sudo zypper install -y helm

Deploy NVIDIA device plugin:

helm repo add nvidia https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade --install nvidia-device-plugin nvidia/k8s-device-plugin -n kube-system

Request a GPU in a Pod:

# k8s/base/pod-gpu.yaml
apiVersion: v1
kind: Pod
metadata: { name: cuda-vectoradd }
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.2.0-base-ubuntu22.04
      command: ["bash","-lc","nvidia-smi && sleep 5"]
      resources:
        limits:
          nvidia.com/gpu: 1

Apply and check:

kubectl apply -f k8s/base/pod-gpu.yaml
kubectl logs -f pod/cuda-vectoradd

Real-world payoff: Moving ad-hoc training to K8s with GPU requests and per-namespace quotas reduced preemption fights and boosted cluster GPU utilization by 20–35%.

5) Build observability and cost guardrails

If you can’t measure it, you can’t optimize it. Track GPU health, memory, and utilization; alert on idle or thrashing jobs.

Install Prometheus stack and NVIDIA DCGM exporter:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add nvidia-dcgm https://nvidia.github.io/dcgm-exporter
helm repo update

# Prometheus + Grafana
helm upgrade --install o11y prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

# GPU metrics
helm upgrade --install dcgm-exporter nvidia-dcgm/dcgm-exporter -n monitoring

Quick check for a metric from Prometheus:

export PROM=$(kubectl get svc -n monitoring -l app=kube-prometheus-stack-prometheus -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}{.items[0].status.loadBalancer.ingress[0].ip}')
curl -s "http://$PROM/api/v1/query?query=DCGM_FI_PROF_GR_ENGINE_ACTIVE" | jq .

Policy examples:

  • Alert on nodes with GPUs < 10% utilized for > 30 minutes.

  • Kill or requeue jobs exceeding VRAM thresholds.

  • Auto-scale node groups by GPU queue depth.

Real-world payoff: One team cut cloud GPU costs ~28% by alerting on idle GPUs and auto-scaling down nights/weekends.

Common pitfalls to avoid

  • Floating tags like latest for CUDA/PyTorch images.

  • Manual driver installs without documenting the exact repo and version.

  • Mixing container runtimes on the same node without testing GPU hooks.

  • Skipping e2e tests for provisioning changes (run a tiny GPU workload after every infra change).

Conclusion and next steps (CTA)

Small, consistent automation steps compound into reliable, fast, and cost-effective AI platforms.

Your 60-minute starter plan: 1) Install Terraform, Ansible, and Podman/Docker using the commands above. 2) Create a repo skeleton and commit a pinned training container. 3) Provision one GPU VM with Terraform and converge it with your Ansible playbook. 4) Run a smoke test: nvidia-smi in a container. 5) If you use Kubernetes, install the NVIDIA device plugin and deploy the sample Pod.

Want a jump-start? Wrap these into a Makefile with targets like make up, make test-gpu, and make down so anyone on your team can reproduce the stack with two commands.

Related posts