Posted on
• Artificial Intelligence

Artificial Intelligence Linux Infrastructure Reviews

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI on Linux: A Practical Infrastructure Review (with Bash-first Setup)

You can buy the latest GPUs and still leave half your performance on the table if your Linux stack isn’t tuned. Conversely, a modest box with the right kernel, drivers, container strategy, and monitoring can punch well above its weight. This article reviews the AI-on-Linux landscape from a practitioner’s perspective and gives you concrete, Bash-friendly steps to get productive, reproducible, and fast.

What you’ll get:

  • A quick mental model for choosing a distro and kernel for AI

  • Actionable setup steps for essentials, containers, and performance hygiene

  • Real-world examples and commands you can paste today

  • Install commands for apt, dnf, and zypper where packages are mentioned

Why this matters

  • AI is bandwidth- and latency-sensitive. Driver or kernel mismatches can silently crater throughput.

  • Reproducibility is non-negotiable for debugging and collaboration. Containers and virtual environments save days.

  • Observability prevents guesswork. Knowing if you’re compute-, memory-, IO-, or network-bound changes everything.

  • The Linux ecosystem gives you the knobs: cgroups v2, io_uring, NUMA, container runtimes, and battle-tested schedulers.

What “good” infrastructure looks like

  • Stable base OS with a well-supported kernel for your hardware

  • Clean GPU stack (if applicable) and container runtime that exposes accelerators

  • Minimal, reproducible userspace (venv/containers), pinned to known-good versions

  • Basic performance hygiene: CPU governors, NUMA awareness, fast local scratch, and telemetry

1) Pick the right distro and kernel (stability vs. velocity)

A quick rule of thumb:

  • Ubuntu LTS (22.04/24.04): Great workstation/server default. Strong community and vendor docs.

  • Rocky Linux 9 / RHEL 9: Solid for clusters and enterprise; conservative but predictable.

  • Fedora: Fast-moving; useful for brand-new hardware enablement.

  • openSUSE Leap (stable) or Tumbleweed (rolling): Leap for servers; Tumbleweed if you need the latest kernels on the desktop.

Checklist:

  • Confirm kernel and drivers support your CPU/GPU/NICs.

  • Keep firmware up to date (BIOS/UEFI, NIC, SSD).

  • Favor the distro’s LTS/HWE kernels unless you need new features for your devices.

Quick system info:

uname -r
. /etc/os-release && printf "%s %s\n" "$NAME" "$VERSION"
lspci | egrep -i 'nvidia|amd|mlx|infiniband'

2) Install the essentials (dev tools, Python, monitoring)

Install baseline tools. Use your package manager below.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential git curl wget python3 python3-venv python3-pip htop nvtop numactl lshw pciutils sysstat fio
  • Fedora/RHEL/Rocky (dnf):
sudo dnf -y update
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git curl wget python3 python3-pip python3-virtualenv htop nvtop numactl lshw pciutils sysstat fio
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y git curl wget python3 python3-venv python3-pip htop nvtop numactl lshw pciutils sysstat fio

Create a Python virtual environment (repeat per project):

python3 -m venv ~/ai-venv
source ~/ai-venv/bin/activate
pip install --upgrade pip wheel
pip install numpy pandas jupyterlab
# CPU-friendly AI frameworks:
pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision torchaudio
pip install tensorflow

Tip: Keep per-project venvs and requirements.txt. Pin known-good versions for reproducibility.

3) Containers: the easiest path to reproducible AI

Containers isolate toolchains so you don’t fight system packages.

Install a container engine.

  • Debian/Ubuntu (apt) — Docker and/or Podman:
sudo apt update
sudo apt install -y docker.io podman
sudo systemctl enable --now docker
sudo usermod -aG docker $USER  # log out/in to apply
  • Fedora/RHEL/Rocky (dnf) — Podman-first; Docker optional:
sudo dnf install -y podman
podman --version
# Optional Docker on Fedora:
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
  • openSUSE/SLES (zypper) — Docker and/or Podman:
sudo zypper install -y docker podman
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

CPU-only example: run PyTorch in a container:

docker run --rm -it python:3.11-slim bash -lc "pip install --index-url https://download.pytorch.org/whl/cpu torch && python -c 'import torch;print(torch.__version__, torch.ones(1).device)'"

NVIDIA GPU with Docker (requires a working NVIDIA driver on the host and NVIDIA Container Toolkit):

docker run --rm --gpus all nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi

Note: Installing GPU drivers and the NVIDIA Container Toolkit depends on your distro and driver version. Use the official vendor docs to add the appropriate repository and install nvidia-container-toolkit, then sudo nvidia-ctk runtime configure and restart Docker/Containerd. For Podman, your distribution’s integration with NVIDIA may require enabling OCI hooks.

Why containers?

  • Same image on laptop, server, and CI

  • No system-wide CUDA/Python churn

  • Fast rollback when something breaks

4) Performance hygiene that pays off

  • Lock CPU governor to “performance” during training:

    • Debian/Ubuntu (apt):
    sudo apt install -y linux-tools-common linux-tools-$(uname -r)
sudo cpupower frequency-set -g performance
    • Fedora/RHEL/Rocky (dnf):
    sudo dnf install -y kernel-tools
sudo cpupower frequency-set -g performance
    • openSUSE/SLES (zypper):
    sudo zypper install -y cpupower
sudo cpupower frequency-set -g performance

  • Be NUMA-aware on multi-socket systems:

numactl --hardware
numactl --interleave=all python train.py   # simple starting point
  • Use fast local scratch and check IO:
df -h
sudo iostat -xz 1
fio --name=randread --rw=randread --bs=4k --size=2G --iodepth=32 --direct=1 --numjobs=4 --runtime=60 --group_reporting
  • Keep an eye on thermals, clocks, and load:
htop
nvtop        # NVIDIA GPUs
nvidia-smi   # if NVIDIA driver is installed
  • Pin container CPU/memory to avoid noisy neighbors:
docker run --cpus=12 --memory=24g --rm -it <image> bash

5) Real-world patterns (what works in practice)

  • Single-GPU workstation (Ubuntu LTS):

    • Install essentials (apt block above).
    • Use the distro’s tested NVIDIA driver (e.g., ubuntu-drivers autoinstall) or vendor instructions if you need specific CUDA.
    • Use Docker for projects; keep CUDA in the container. Verify with:
    docker run --rm --gpus all nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi

  • Headless inference node (Rocky Linux 9):

    • Prefer Podman for rootless containers:
    sudo dnf install -y podman
podman run --rm -it python:3.11-slim bash
    • For CPU-bound models, scale via systemd-run or a simple reverse proxy; keep one venv/container per service.

  • Small lab cluster:

    • Start with a simple scheduler (Slurm) to queue jobs. Install packages:
    • Debian/Ubuntu (apt): sudo apt install -y slurm-wlm
    • Fedora/RHEL/Rocky (dnf): sudo dnf install -y slurm slurm-slurmctld slurm-slurmd
    • openSUSE/SLES (zypper): sudo zypper install -y slurm
    • Configure a shared /home or /project and a local NVMe scratch on each node. Use containers for per-job environments.

What we consistently see:

  • Containers reduce “works on my machine” issues to near-zero.

  • Pinning governors + basic NUMA hints often yield 5–20% speedups on CPUs.

  • IO bottlenecks are common; a fast NVMe scratch directory for datasets/checkpoints helps training and eval.

Optional: quick Bash snippets you’ll reuse

  • Check GPU visibility in containers (NVIDIA):
docker run --rm --gpus all --ipc=host nvidia/cuda:12.2.0-base-ubuntu22.04 bash -lc "nvidia-smi && ldconfig -p | grep cuda"
  • Create a project venv and record versions:
python3 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip wheel
pip freeze | tee requirements.txt
  • Simple benchmark harness start:
python - <<'PY'
import time, torch
x = torch.randn([4096,4096])
t=time.time()
for _ in range(100):
    y = x @ x
print("s:", time.time()-t)
PY

Conclusion and next steps

The fastest path to reliable AI on Linux is surprisingly simple: 1) Pick a stable distro and stay on supported kernels. 2) Use containers for the AI stack; keep system packages lean. 3) Do the easy performance wins: CPU governor, NUMA hinting, fast scratch. 4) Monitor everything; let data, not hunches, drive tuning.

Your next step:

  • Turn this into action on one machine today. Install the essentials with your package manager, spin up a containerized model, and baseline performance. As you standardize across your team or cluster, you’ll feel the debugging tax and downtime shrink.

If you want a follow-up post with distro-specific NVIDIA/ROCm container runtime guides and hardened production templates, say the word and tell me your target distro and GPUs.

Related posts