Posted on
• Artificial Intelligence

Artificial Intelligence Linux Operations Checklist

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Operations Checklist: From “It Works on My Machine” to Reproducible, GPU‑Ready

You’ve got models to train, deadlines to meet, and GPUs sitting idle because the system “almost” works: missing drivers, mismatched CUDA, package pinning chaos, jobs that crash overnight. This checklist is your shortcut from an ad‑hoc setup to a stable, reproducible, and performance‑tuned AI stack on Linux.

What you’ll get:

  • A practical sequence to validate hardware, install the right components, and avoid common pitfalls.

  • Actionable, copy‑pasteable commands (apt, dnf, zypper included).

  • Real‑world tips to harden, monitor, and scale your AI workloads.

Why a checklist?

  • AI workloads are unforgiving: small mismatches (CUDA vs. driver, kernel vs. module) cause opaque failures.

  • Reproducibility saves time and money: portable environments end “works on my machine.”

  • Performance is compound: proper drivers, power modes, I/O, and monitoring unlock your hardware’s potential.

1) Baseline System Readiness (10–20 min)

Goal: Ensure your OS, CPU, GPU, and toolchain are ready for AI workloads.

  • Check CPU flags (look for AVX/AVX2/AVX-512 if you use CPU inference or BLAS):
lscpu | egrep -i 'avx|sse|aes'
  • Identify GPUs and kernel:
lspci | grep -i -E 'nvidia|amd|display|vga'
uname -r
  • Install baseline tools and dev toolchain:

APT (Debian/Ubuntu)

sudo apt update
sudo apt install -y git curl wget jq build-essential \
  python3 python3-pip python3-venv \
  htop nvtop lm-sensors fio podman

DNF (Fedora/RHEL-family with dnf)

sudo dnf -y install git curl wget jq \
  python3 python3-pip python3-virtualenv \
  htop nvtop lm_sensors fio podman
sudo dnf -y groupinstall "Development Tools"

Zypper (openSUSE/SLE)

sudo zypper refresh
sudo zypper install -y git curl wget jq gcc gcc-c++ make \
  python3 python3-pip python3-virtualenv \
  htop nvtop sensors fio podman
  • Detect sensors (then check temps/fans):
sudo sensors-detect --auto
sensors

Tip: Keep your firmware and microcode updated; many weird stability issues disappear after BIOS/microcode updates.

2) GPU Drivers and CUDA Sanity (15–30 min)

Goal: Install and verify the correct proprietary driver (NVIDIA example) for compute workloads.

  • Ubuntu/Debian (apt)
sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot
  • Fedora (dnf) via RPM Fusion
sudo dnf install -y \
  https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
  • openSUSE Leap (zypper)
. /etc/os-release
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/$VERSION_ID NVIDIA
sudo zypper install -y nvidia-driver-G06
sudo reboot
  • openSUSE Tumbleweed (zypper)
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed NVIDIA
sudo zypper install -y nvidia-driver-G06
sudo reboot
  • Verify driver + CUDA:
nvidia-smi
# Optional: keep GPUs ready for headless compute
sudo nvidia-smi -pm 1

Real-world tip: If you rely on containers, the NVIDIA Container Toolkit lets containers see your GPUs (see step 4).

3) Reproducible Python Environments (5–15 min)

Goal: Pin dependencies and isolate projects so experiments are repeatable.

  • Create a project virtual environment:
mkdir -p ~/ai-project && cd ~/ai-project
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
  • Install frameworks matching your hardware:
    • CPU-only example:
pip install "torch>=2.2" torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
  • CUDA example (match CUDA version shown by nvidia-smi, e.g., cu121):
pip install "torch>=2.2" torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
  • Verify CUDA availability in PyTorch:
python - << 'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
print("CUDA device count:", torch.cuda.device_count())
if torch.cuda.is_available():
    print("GPU 0:", torch.cuda.get_device_name(0))
PY
  • Freeze dependencies to lock experiments:
pip freeze > requirements.txt
# Later, reproduce with:
# python -m venv .venv && source .venv/bin/activate
# pip install -r requirements.txt

Tip: For team workflows, commit requirements.txt and include a short “bootstrap.sh” that creates the venv and installs requirements.

4) Containers for Isolation and Portability (10–25 min)

Goal: Run experiments in clean, disposable, GPU‑aware containers.

  • Install Docker Engine (optional, if you prefer Docker over Podman):

APT

sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

DNF (Fedora)

sudo dnf install -y moby-engine
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Zypper (openSUSE)

sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
  • Install NVIDIA Container Toolkit (enables GPU access inside containers):

APT

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

DNF

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
  | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Zypper

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
  | sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
# For Docker runtime integration:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
  • Test GPU inside a container:
docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi
  • Prefer Podman? Keep it rootless for safety. With the NVIDIA toolkit installed, Podman uses OCI hooks to expose GPUs on supported distros:
podman run --rm --security-opt=no-new-privileges \
  nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi

Note: If GPUs are not visible in Podman, ensure the NVIDIA OCI hook is present under /usr/share/containers/oci/hooks.d/ per your distro’s NVIDIA toolkit docs.

5) Performance, Monitoring, and Ops Hygiene (10–20 min)

Goal: Keep jobs fast, observable, and stable.

  • Live GPU view and quick triage:
watch -n 1 nvidia-smi
nvtop
  • Verify clocks and persistence for headless servers:
sudo nvidia-smi -pm 1
# Optional: lock application clocks (know what you’re doing)
# sudo nvidia-smi -lgc 1500,2100
  • Check temps, fans, throttling:
sensors
dmesg --ctime | egrep -i 'nvrm|pci|dma|iommu|gpu|thermal'
  • I/O sanity (your dataset drive matters more than you think):
fio --name=randread --filename=/path/to/dataset/testfile --size=4G \
    --bs=256k --iodepth=32 --rw=randread --direct=1
  • Resource controls for runaway jobs (systemd scope example):
systemd-run --user --scope -p MemoryMax=24G -p CPUQuota=400% \
  bash -lc 'source ~/ai-project/.venv/bin/activate && python train.py'
  • Run long jobs resiliently under systemd:
sudo tee /etc/systemd/system/train.service > /dev/null <<'EOF'
[Unit]
Description=AI Training Job
After=network-online.target

[Service]
User=YOURUSER
WorkingDirectory=/home/YOURUSER/ai-project
Environment="PYTHONUNBUFFERED=1"
ExecStart=/home/YOURUSER/ai-project/.venv/bin/python train.py
Restart=on-failure
MemoryMax=48G
CPUQuota=600%
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now train.service
journalctl -u train.service -f

Real-world checklist before you hit “go”:

  • Drivers loaded, nvidia-smi clean, persistence mode on.

  • Project venv built, requirements.txt pinned.

  • Data path verified, I/O baseline measured.

  • Monitoring open (nvtop), logs tailing (journalctl -f).

  • If containerized: toolkit installed, docker run --gpus all works.

Bonus: Quick “Hello, GPU” in 60 seconds

mkdir -p ~/ai-hello && cd ~/ai-hello
python3 -m venv .venv && source .venv/bin/activate
python -m pip install --upgrade pip
pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision
python - << 'PY'
import torch
x = torch.randn(1024,1024,device='cuda')
y = torch.randn(1024,1024,device='cuda')
print("OK, CUDA:", torch.cuda.is_available(), "Result:", (x@y).sum().item())
PY

Conclusion and Next Steps

You now have a hardened baseline to run AI workloads on Linux—drivers verified, environments pinned, containers GPU‑aware, and monitoring in place. The next step is to standardize this checklist in your org:

  • Turn the commands above into a bootstrap script for new machines.

  • Bake a golden container image (with your frameworks and CUDA match) and version it.

  • Add systemd units for your critical training/eval pipelines and log aggregation.

Want a one‑file bootstrap script for your distro and GPU stack? Tell me your distro version, GPU model, and preferred framework, and I’ll generate a ready‑to‑run installer.

Related posts