The AI Hardware Checklist for Linux: A Bash-First Guide

Before you pull multi-gigabyte models or compile hours-long kernels, do this: spend 15 minutes verifying your hardware. A quick, reproducible checklist can save you days of chasing driver gremlins, PCIe bottlenecks, or thermal throttling when your training jobs mysteriously crawl.

This post gives you a practical, Bash-oriented AI hardware checklist you can run on any Linux machine. You’ll learn why each step matters, get copy-paste commands, and see what “good” looks like. Follow along, capture the results, and you’ll know exactly where you stand.

Why this matters (and what you’ll avoid)

• Avoid silent slowdowns: A GPU negotiating down to PCIe x4 instead of x16 can quietly halve throughput.

• Catch driver gaps early: CUDA/ROCm mismatches often show up only when you import a framework.

• Size the box to the job: Knowing VRAM, system RAM, and disk throughput up front helps right-size models and batch sizes.

• Reproduce reliably: A simple, shareable report lets teammates or vendors help you quickly.

Prerequisites: Install the tooling once

These utilities are in standard repos and safe to install on most systems. They help you inventory hardware, verify drivers, and do basic performance sanity checks.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y pciutils usbutils lshw dmidecode lm-sensors nvme-cli smartmontools fio sysbench podman python3-venv git
# Optional build tools if you'll compile libs:
sudo apt install -y build-essential

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf -y install pciutils usbutils lshw dmidecode lm_sensors nvme-cli smartmontools fio sysbench podman python3-venv git
# Optional build tools:
sudo dnf -y groupinstall "Development Tools"

openSUSE Leap/Tumbleweed or SLE (zypper):

sudo zypper refresh
sudo zypper install -y pciutils usbutils lshw dmidecode lm_sensors nvme-cli smartmontools fio sysbench podman python3-venv git
# Optional build tools:
sudo zypper install -y -t pattern devel_basis

Initialize sensors once:

sudo sensors-detect --auto

Tip: Create a directory to capture outputs so you can paste them into tickets or docs later.

mkdir -p ~/ai-hw-check && cd ~/ai-hw-check

---

The 5‑part AI Hardware Checklist

1) System inventory and firmware sanity

Goal: Confirm CPU topology, NUMA layout, BIOS/UEFI version, and GPU presence. This establishes a baseline.

Commands:

lscpu | tee lscpu.txt
numactl --hardware 2>/dev/null | tee numa.txt || echo "numactl not installed"
sudo dmidecode -t bios | tee bios.txt
sudo lshw -short | tee lshw-short.txt
lspci -nnk | grep -A3 -E "VGA|3D|Display" | tee gpus.txt

What to look for:

• lscpu: Check “Flags” for avx/avx2/avx512 on x86_64 (many inference libs require at least AVX2), or “Features” like neon on ARM.

• NUMA: If you have multiple CPU sockets, plan to pin processes to minimize cross-NUMA memory traffic.

• BIOS: Note the version/date. For multi-GPU rigs, ensure “Above 4G Decoding” and “Resizable BAR” (often helps dGPUs) are available and on.

Real-world example: A dual-socket box training slower than a single-socket workstation turned out to be cross-NUMA memory access. Pinning workers per NUMA node plus affinitizing memory improved throughput by ~20%.

---

2) GPU readiness: drivers, modules, and PCIe lanes

Goal: Ensure your GPU is detected, the right driver/module is loaded, and the PCIe link is running at expected width/speed.

Detect GPUs:

lspci | grep -Ei "vga|3d|display" | tee vga.txt

Check kernel modules:

lsmod | grep -E 'nvidia|amdgpu' | tee modules-gpu.txt

If vendor tools are installed, verify:

# NVIDIA (installed with proprietary driver)
nvidia-smi | tee nvidia-smi.txt

# AMD ROCm (installed from ROCm repo/tooling)
rocminfo | tee rocminfo.txt
rocm-smi | tee rocm-smi.txt

Check PCIe link width/speed (x16 Gen4, etc.):

GPU_SLOT=$(lspci | awk '/VGA compatible controller: NVIDIA|VGA compatible controller: Advanced Micro Devices/{print $1; exit}')
[ -n "$GPU_SLOT" ] && sudo lspci -vv -s "$GPU_SLOT" | grep -E 'LnkCap|LnkSta' | tee pcie-link.txt || echo "No discrete GPU found"

What to look for:

• Modules: nvidia or amdgpu should appear. If only nouveau shows for NVIDIA, you’re likely on the open driver (OK for desktop, not for CUDA).

• nvidia-smi/rocm-smi: Reported VRAM, driver/runtime versions, ECC status (on datacenter GPUs), temperature, and throttle reasons.

• PCIe: Expect x16 on desktop/workstation boards. x8 is fine on many platforms; x4 is often a sign the card is in a slow slot.

Note on installation: NVIDIA and ROCm drivers/toolkits come from vendor repositories (outside base repos). Use the official docs for your distro and GPU generation to install/upgrade drivers, CUDA, or ROCm. After install, rerun the checks above.

---

3) Memory and storage headroom

Goal: Validate you have enough system RAM for your model pipelines and that disk I/O won’t bottleneck dataset loads or checkpoints.

RAM snapshot:

free -h | tee free.txt
grep -E 'MemTotal|SwapTotal' /proc/meminfo | tee meminfo.txt

NVMe/SATA health:

# NVMe example (adjust /dev/nvme0 if needed)
sudo nvme smart-log /dev/nvme0 | tee nvme-smart.txt 2>/dev/null || echo "nvme-cli failed"
# SATA example (adjust /dev/sda)
sudo smartctl -a /dev/sda | tee sda-smart.txt

Quick disk throughput with fio (read and write; change the directory to your dataset/checkpoint path):

fio --name=ai-disk-rw --size=2G --filename=./fio-test.bin --bs=1M --rw=readwrite --direct=1 --iodepth=32 --numjobs=1 --time_based=0 | tee fio-rw.txt
rm -f ./fio-test.bin

What to look for:

• Swap: If swap is small/disabled and RAM is tight, large data loaders may OOM. Plan accordingly.

• NVMe temps and media errors: High temps or rising media errors foreshadow throttling/failures.

• fio: For modern NVMe, expect hundreds of MB/s to multiple GB/s for sequential 1M blocks. If you see HDD-like numbers on an NVMe, check power states, PCIe slots, or that you’re testing the right path.

Real-world example: A dataset stored on a networked HDD mount added 15 minutes per epoch. Moving it to local NVMe cut epoch time by 40%.

---

4) CPU capability and quick stress sanity

Goal: Confirm instruction sets for optimized libs and that the CPU can sustain load without immediate thermal throttling.

Instruction sets:

lscpu | grep -Ei 'avx|sse|aes|sha|asimd|neon' | tee cpu-flags.txt

Short CPU stress (watch temps in another terminal with watch -n1 sensors):

sysbench cpu --cpu-max-prime=20000 run | tee sysbench-cpu.txt

What to look for:

• Flags: AVX2 is common for modern x86_64 inference; some wheels fall back (slower) if missing. On ARM, asimd/neon is standard; look for SVE on newer chips.

• Thermals: If temps spike and clocks dip fast, clean dust, improve airflow, or adjust power limits.

---

5) Containers and reproducibility smoke test

Goal: Ensure a container runtime works so you can use vendor images or standardized dev environments.

Podman hello:

podman run --rm docker.io/hello-world

Check that basic networking and exec work:

podman run --rm -ti docker.io/library/alpine:latest sh -lc "uname -a && cat /etc/os-release"

If you plan GPU containers, install the relevant vendor container runtime/toolkit (NVIDIA Container Toolkit or ROCm device plugins) per vendor docs, then re-run a framework container and confirm it sees the GPU. Once configured, a typical check looks like:

# Example only; requires vendor runtime configuration
podman run --rm --device nvidia.com/gpu=all docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

---

Putting it together: one-shot capture

Run a quick bundle and save results:

#!/usr/bin/env bash
set -euo pipefail
OUT=~/ai-hw-check && mkdir -p "$OUT"
{
  echo "=== Timestamp ==="; date
  echo "=== Kernel ==="; uname -a
  echo "=== CPU ==="; lscpu
  echo "=== NUMA ==="; numactl --hardware 2>/dev/null || echo "numactl N/A"
  echo "=== BIOS ==="; sudo dmidecode -t bios
  echo "=== PCIe GPUs ==="; lspci -nnk | grep -A3 -E "VGA|3D|Display"
  echo "=== GPU Modules ==="; lsmod | grep -E 'nvidia|amdgpu' || true
  echo "=== NVIDIA SMI ==="; command -v nvidia-smi && nvidia-smi || echo "nvidia-smi N/A"
  echo "=== ROCm SMI ==="; command -v rocm-smi && rocm-smi || echo "rocm-smi N/A"
  echo "=== PCIe Link (first dGPU) ==="
  SLOT=$(lspci | awk '/VGA compatible controller: NVIDIA|VGA compatible controller: Advanced Micro Devices/{print $1; exit}')
  [ -n "${SLOT:-}" ] && sudo lspci -vv -s "$SLOT" | grep -E 'LnkCap|LnkSta' || echo "No dGPU"
  echo "=== Memory ==="; free -h
  echo "=== Sensors ==="; sensors || true
} | tee "$OUT/summary.txt"
echo "Saved to $OUT"

Make it executable and run:

chmod +x ./ai-hw-snapshot.sh
./ai-hw-snapshot.sh

---

Common findings (and quick fixes)

• GPU at x4 instead of x16: Move the card to a CPU-attached x16 slot; check BIOS for bifurcation/slot settings.

• Nouveau loaded on NVIDIA: Install the proprietary driver via vendor docs; blacklist nouveau if needed.

• Disk I/O bottleneck: Keep datasets/checkpoints on NVMe; set num_workers and prefetch to hide I/O.

• Missing AVX2 on older CPUs: Use CPU-only wheels compiled for generic x86_64 or run inference on GPU.

• Overheating: Clean dust filters, reapply thermal paste on aging systems, or set less aggressive power limits.

---

Conclusion and next steps

You now have a fast, Bash-first way to validate AI hardware on Linux. Use it whenever you commission a new workstation, migrate to a fresh kernel, or change drivers. Save the outputs in a repo/wiki so your team can compare machines and reproduce results.

Next steps:

• Run the checklist and keep the ~/ai-hw-check folder for your records.

• Install/verify your preferred framework (PyTorch, TensorFlow, JAX) and run a tiny model to confirm end-to-end.

• If you use containers, finish GPU runtime setup via vendor docs and re-check with a framework container.

Questions or want a zero-to-ready script tailored to your distro and GPU? Reach out with your summary.txt, and I’ll help you automate the rest.