Posted on
• Artificial Intelligence

Artificial Intelligence Linux Upgrade Planning

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Upgrade Planning: A Bash-First Playbook

If your models have to be online Monday morning, you can’t afford a Friday upgrade that quietly breaks CUDA, drivers, or your container runtime. AI stacks are powerful—and fragile. This guide shows you a practical, Bash-first way to plan Linux upgrades for AI workloads with minimal downtime and maximum predictability.

You’ll learn:

  • Why AI stacks are uniquely sensitive to Linux upgrades

  • How to inventory and freeze critical components

  • How to snapshot, stage, and validate upgrades

  • How to roll back fast when something goes sideways

Why this matters for AI on Linux

  • Kernel/driver ABI coupling is real: GPU drivers (NVIDIA/AMD), CUDA/ROCm, and low-level libraries must align with your kernel. A casual kernel upgrade can cascade into “driver not loaded,” “no CUDA device,” or segfaults.

  • Containers aren’t a silver bullet: They package userspace, not the kernel/driver layers. GPU support still depends on host drivers and kernel modules.

  • Downtime is costly: Training interruptions waste GPU-hours; serving outages break SLAs. Upgrade planning is cheaper than firefighting.

The Playbook (5 steps)

1) Baseline your AI stack (inventory first)

Capture what you have before you change it. This makes planning (and rollback) precise.

Install baseline tooling:

  • apt (Ubuntu/Debian)
sudo apt update
sudo apt install -y curl jq git pciutils python3 python3-venv python3-pip lvm2 snapper podman
# Optional build tools
sudo apt install -y build-essential
# Optional snapshots on Ubuntu/Debian
sudo apt install -y timeshift
  • dnf (Fedora/RHEL/CentOS Stream)
sudo dnf install -y curl jq git pciutils python3 python3-virtualenv python3-pip lvm2 snapper podman dnf-plugins-core
# Optional build tools
sudo dnf groupinstall -y "Development Tools"
  • zypper (openSUSE/SLE)
sudo zypper refresh
sudo zypper install -y curl jq git pciutils python3 python3-virtualenv python3-pip lvm2 snapper podman
# Optional build tools pattern
sudo zypper install -y -t pattern devel_basis

Quick inventory commands:

echo "Kernel: $(uname -r)"
lsb_release -ds 2>/dev/null || cat /etc/os-release
lspci | egrep -i 'nvidia|amd|ati'
df -h /
free -h

GPU checks (run the ones that apply):

  • NVIDIA (after driver install, see below): nvidia-smi

  • AMD ROCm: rocm-smi

Python/ML:

python3 -c "import sys; print(sys.version)"
python3 -c "import torch, json; print(json.dumps({'torch':getattr(torch,'__version__',None),'cuda':getattr(torch.version,'cuda',None),'gpu':torch.cuda.is_available()}))" || echo "PyTorch not installed"

Container runtimes:

podman --version || true
docker --version || true

Install or verify GPU tools (optional but recommended):

  • NVIDIA drivers and tools

    • apt (Ubuntu/Debian):
    • Easiest: sudo ubuntu-drivers autoinstall then reboot
    • Or pin a known version:
    sudo apt install -y nvidia-driver-535 nvidia-cuda-toolkit
sudo reboot
    • dnf (Fedora; requires RPM Fusion):
    sudo dnf install -y https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                    https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
    • zypper (openSUSE; add NVIDIA repo, then install):
    # Tumbleweed example (adjust for Leap/SLE as needed)
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed NVIDIA
sudo zypper install -y nvidia-driver
sudo reboot
    • Verify: nvidia-smi shows driver and CUDA version mappings.

  • AMD ROCm (check supported hardware/OS combos in AMD docs)

    • apt/dnf/zypper typically provide packages after enabling AMD ROCm repos:
    # Visit https://rocm.docs.amd.com/ for repo setup for your distro, then:
sudo apt install -y rocm-smi   # or
sudo dnf install -y rocm-smi   # or
sudo zypper install -y rocm-smi
    • Verify: rocm-smi

GPU containers (optional, NVIDIA):

# apt
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit

# dnf
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /etc/pki/rpm-gpg/NVIDIA-GPG-KEY
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit

# zypper
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh && sudo zypper install -y nvidia-container-toolkit

Tip: Podman is available everywhere and great for CPU tests. For GPU acceleration with containers, ensure the NVIDIA toolkit is installed and configured for your runtime.

2) Freeze and snapshot before you touch anything

Package holds/locks prevent accidental major changes mid-upgrade.

  • apt:
sudo apt-mark hold nvidia-driver* cuda* libcudnn* linux-generic linux-headers* containerd* docker* podman*
  • dnf:
sudo dnf install -y dnf-plugins-core
sudo dnf versionlock add akmod-nvidia xorg-x11-drv-nvidia* kernel* containerd* docker* podman*
  • zypper:
sudo zypper addlock 'nvidia*' 'cuda*' 'kernel*' 'docker*' 'podman*'

Create a rollback snapshot:

  • Btrfs + snapper (recommended on Fedora/openSUSE, and available on Ubuntu if root is Btrfs):
# If not configured yet (Btrfs root):
sudo snapper -c root create-config /
sudo snapper -c root create -d "pre-upgrade"
  • LVM (snapshot of root LV; adapt LV names carefully):
sudo lvcreate -L 5G -s -n root_pre_upg /dev/vg0/root
  • Timeshift (Ubuntu/Debian; RSYNC or Btrfs):
sudo timeshift --create --comments "pre-upgrade" --tags D

Also export an inventory file for audit and rollback notes:

{
  echo "{"
  echo "\"date\":\"$(date --iso-8601=seconds)\","
  echo "\"kernel\":\"$(uname -r)\","
  echo "\"os\":\"$(. /etc/os-release; echo $PRETTY_NAME)\","
  echo "\"gpu\":\"$(lspci | egrep -i 'nvidia|amd|ati' | tr -d '"')\""
  echo "}"
} > preupgrade-inventory.json

3) Stage upgrades in isolation (containers and venvs)

Before touching the host, validate new userland versions in sandboxes.

Python virtual env:

python3 -m venv ~/ai-next
source ~/ai-next/bin/activate
pip install --upgrade pip
pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu121  # match your target CUDA
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
deactivate

Container sanity test (CPU):

podman run --rm python:3.11-slim python -c "print('ok')"

GPU container sanity (NVIDIA; Docker or Podman with toolkit configured):

# Docker example
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi

If your staged tests fail, you’ve saved yourself a host-level outage.

4) Execute the upgrade with a plan

  • Prefer smaller, well-understood steps (e.g., minor kernel + security updates) over big-bang upgrades.

  • Upgrade order (typical NVIDIA AI host): 1) Kernel/security updates (no driver changes) 2) Reboot and verify current driver still binds 3) Upgrade NVIDIA driver to a version compatible with the new kernel 4) Upgrade CUDA/cuDNN libraries to match your frameworks 5) Update containers/venvs; rebuild images

  • Keep maintenance windows and a rollback timer. If postflight checks aren’t green in N minutes, roll back.

Examples:

  • apt:
sudo apt update
sudo apt upgrade -y
sudo reboot
# Then (when ready) remove holds selectively and upgrade drivers/CUDA
sudo apt-mark unhold nvidia-driver* cuda*
sudo apt install -y nvidia-driver-535 nvidia-cuda-toolkit
sudo reboot
  • dnf:
sudo dnf upgrade -y
sudo reboot
# Then driver/toolkit as per your plan
  • zypper:
sudo zypper refresh
sudo zypper update -y
sudo reboot

5) Post-upgrade validation and rollback

Automated quickcheck script:

#!/usr/bin/env bash
set -euo pipefail
log(){ echo "[$(date --iso-8601=seconds)] $*"; }
fail(){ log "FAIL: $*"; exit 1; }

log "Kernel: $(uname -r)"
log "Disk:"; df -h / | tail -1
log "Memory:"; free -h

if command -v nvidia-smi >/dev/null 2>&1; then
  log "NVIDIA:"
  nvidia-smi || fail "nvidia-smi failed"
fi

if command -v rocm-smi >/dev/null 2>&1; then
  log "ROCm:"
  rocm-smi || fail "rocm-smi failed"
fi

python3 - <<'PY' || exit 0
import json, sys
res = {}
try:
  import torch
  res["torch"]=torch.__version__
  res["cuda"]=getattr(torch.version,"cuda",None)
  res["gpu_ok"]=torch.cuda.is_available()
except Exception as e:
  res["torch_error"]=str(e)
print(json.dumps(res))
PY

if command -v podman >/dev/null 2>&1; then podman --version; fi
if command -v docker  >/dev/null 2>&1; then docker --version; fi

log "OK: post-upgrade checks done"

If validation fails, roll back quickly:

  • snapper (Btrfs):
sudo snapper -c root list
sudo snapper -c root rollback <PRE_SNAPSHOT_ID>
sudo reboot
  • LVM:
sudo lvconvert --merge /dev/vg0/root_pre_upg
sudo reboot
  • Timeshift:
sudo timeshift --restore
  • dnf history (userland rollback):
sudo dnf history
sudo dnf history undo <ID>

Real-world scenario (condensed)

Goal: Update a Ubuntu LTS host’s kernel and keep CUDA working.

1) Inventory and snapshot:

uname -r; nvidia-smi
sudo timeshift --create --comments "pre-kernel-upg" --tags D

2) Freeze drivers:

sudo apt-mark hold nvidia-driver* cuda*

3) Upgrade kernel/security only:

sudo apt update && sudo apt install -y linux-generic
sudo reboot

4) Verify driver still binds:

nvidia-smi  # should succeed

5) If moving to a newer CUDA toolchain later:

sudo apt-mark unhold nvidia-driver* cuda*
sudo apt install -y nvidia-driver-535 nvidia-cuda-toolkit
sudo reboot

6) Run the post-upgrade script; if anything breaks, restore with Timeshift.

Conclusion and next steps

Upgrades don’t have to be risky. With a simple, repeatable plan—inventory, freeze, snapshot, stage, execute, validate—you can keep your AI stack stable and secure.

Your next step:

  • Save this playbook in your team wiki.

  • Automate the inventory and postflight script on every AI node.

  • Pilot your next upgrade in a container or VM this week before touching production.

If you’d like a turnkey Bash script that wraps the full workflow for your distro and GPU stack, let me know your environment (distro/version, GPU vendor, container runtime), and I’ll tailor one for you.

Related posts