Posted on
• Artificial Intelligence

Artificial Intelligence Linux Health Checks Explained

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Health Checks Explained

AI workloads are ruthless stress tests for Linux systems. One missing kernel flag, a throttling GPU, or a tired NVMe can quietly turn a training run that should finish overnight into a multi-day slog. The value of a repeatable health check: you find issues before they find you—saving time, money, and sanity.

This guide explains what to check, why it matters for AI workloads, and gives you concrete, Bash-friendly steps you can automate on any Linux box.

Why AI-focused health checks matter

  • AI stacks are “tall”: kernel, drivers, libraries, containers, and frameworks must align. Small version drifts cause big slowdowns.

  • GPUs are sensitive to thermals, power limits, and PCIe link states. Performance can silently degrade.

  • Dataset I/O dominates training time. One misbehaving drive or filesystem option can bottleneck the whole pipeline.

  • Memory pressure (RAM, VRAM, and NUMA placement) often causes OOM kills, paging storms, or fragmented memory that hurts throughput.

  • Monitoring catches issues early—before you burn compute hours on a misconfigured node.

Install the core toolkit (apt, dnf, zypper)

These tools are lightweight, distro-native, and cover CPU, memory, storage, sensors, and GPU visibility.

  • apt (Debian/Ubuntu):
sudo apt update
sudo apt install -y pciutils nvtop lm-sensors smartmontools nvme-cli sysstat fio stress-ng htop glances numactl
  • dnf (Fedora/RHEL/CentOS Stream; enable EPEL where needed):
sudo dnf install -y pciutils nvtop lm_sensors smartmontools nvme-cli sysstat fio stress-ng htop glances numactl
  • zypper (openSUSE/SLES):
sudo zypper refresh
sudo zypper install -y pciutils nvtop sensors smartmontools nvme-cli sysstat fio stress-ng htop glances numactl

Note:

  • lm-sensors package name is lm-sensors on apt, lm_sensors on dnf, and sensors on zypper.

  • nvidia-smi comes with the NVIDIA driver; use your vendor’s driver install method.

5-step AI health check you can run today

Each step includes Bash commands you can paste into a terminal. Run as a user, add sudo where needed.

1) Profile the host: kernel, CPU, memory, and GPUs

What you learn: baseline versions and hardware so you can spot drift between machines or over time.

echo "== OS and kernel =="
uname -a
lsb_release -a 2>/dev/null || cat /etc/os-release

echo "== CPU and NUMA =="
lscpu
numactl --hardware

echo "== Memory =="
free -h
grep -i huge /proc/meminfo | sed 's/^/  /'

echo "== PCIe GPUs =="
lspci | egrep -i 'vga|3d|nvidia|amd'
command -v nvidia-smi >/dev/null && nvidia-smi --query-gpu=name,driver_version,cuda_version,pci.bus_id --format=csv

Why it matters:

  • Kernel/driver/library mismatches are a top cause of runtime instability and poor performance.

  • NUMA layouts affect large-batch training; pinning processes to sockets can improve throughput.

Tip: For interactive GPU view:

nvtop

2) GPU sanity: thermals, clocks, ECC, and PCIe link health (NVIDIA)

What you learn: is the GPU delivering expected performance headroom, or silently throttling?

# Quick health summary
nvidia-smi

# Detailed info (temps, ECC, clocks, BAR sizes, power, PCIe)
nvidia-smi -q | sed -n '1,200p'

# Live utilization and power draw (press Ctrl+C to stop)
nvidia-smi dmon -s pucmt

Action items:

  • Temperatures consistently >80–85°C during load indicate cooling issues (dust, airflow, fan curves).

  • Power cap too low? Look for “Power Draw” near the “Power Limit”.

  • ECC errors >0 on data-center GPUs can cause instability; investigate if rising.

  • PCIe should match expected width/speed (e.g., x16 Gen4/Gen5). See “Current Link Speed/Width” in nvidia-smi -q.

If you don’t have NVIDIA GPUs:

  • Use sensors for general temps:
sudo sensors-detect --auto
watch -n 1 sensors
  • For AMD GPUs, you can also inspect PCIe with lspci:
GPU_BUS=$(lspci | awk '/VGA|3D|Display/ {print $1; exit}')
sudo lspci -vv -s "$GPU_BUS" | egrep -i 'LnkSta|LnkCap|Speed|Width'

3) Memory and CPU stability under AI-like pressure

Why it matters: Model training is heavy on memory bandwidth and vector math. You want to catch OOM risks and thermal throttling early.

  • Check swap and overcommit:
echo "== Swap =="
swapon --show
grep -E '^(Vm|Commit|Mem|Swap)' /proc/meminfo | sed 's/^/  /'

echo "== Overcommit =="
cat /proc/sys/vm/overcommit_memory

Set sensible defaults for training (example: avoid aggressive overcommit on shared nodes):

sudo sysctl -w vm.overcommit_memory=0
  • Stress test CPU/Memory for a short burst (safe default: 60s):
sudo stress-ng --matrix 8 --matrix-size 1024 --ram 75% --timeout 60s --metrics-brief

If errors, throttling, or kernel warnings appear in dmesg, fix cooling, BIOS power limits, or memory config before running real workloads.

4) Storage health and dataset I/O

Why it matters: Slow or failing drives can cut your samples/sec in half.

  • S.M.A.R.T. health:
sudo smartctl --scan
# Replace /dev/nvme0 or /dev/sda as appropriate:
sudo smartctl -H -A /dev/nvme0
sudo smartctl -H -A /dev/sda
  • NVMe-specific logs:
sudo nvme list
sudo nvme smart-log /dev/nvme0
  • Quick I/O micro-benchmark (non-destructive, temp file):
cd /tmp
fio --name=ai-io --size=2G --filename=/tmp/ai-io.tmp --bs=1M --rw=readwrite --iodepth=32 --direct=1 --numjobs=1 --time_based=1 --runtime=60 --group_reporting
rm -f /tmp/ai-io.tmp

Interpretation:

  • Sequential read/write in the hundreds of MB/s to a few GB/s is normal for NVMe; HDDs are far lower.

  • High latency or IOPS collapse indicates contention, failing media, or a filesystem mount option mismatch.

5) Thermals, throttling, and live monitoring

Why it matters: Sustained AI workloads are marathons. You need continuous visibility.

  • Sensor baseline:
sudo sensors-detect --auto
watch -n 1 sensors
  • Live system dashboard (CPU, mem, disk, net, temps):
glances
  • Brief utilization snapshots you can script:
iostat -x 1 5
vmstat 1 5

If you routinely see high CPU steal (virtualized), IO wait, or GPU thermals peaking, address the root: cooling, placement, BIOS power limits, or noisy neighbors on shared infra.

Three quick real-world fixes

1) PCIe link stuck slow

  • Symptom: GPUs at low utilization, samples/sec far below spec.

  • Check:

nvidia-smi -q | egrep -i 'Current Link|Max Link'
GPU_BUS=$(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader | head -n1 | sed 's/^0000://')
sudo lspci -vv -s "$GPU_BUS" | egrep -i 'LnkSta|LnkCap'
  • Fixes: Reseat GPU, use correct slot, update BIOS, disable aggressive ASPM, verify risers/cables.

2) Silent OOM on big batches

  • Symptom: Training crashes or slows; dmesg shows OOM killer.

  • Check and adjust swap (if appropriate for your workload) and batch sizes:

free -h
swapon --show

3) Dataset I/O bottleneck

  • Symptom: GPU underutilized while dataloader waits on disk.

  • Check drive health and throughput (smartctl, fio). Move hot datasets to NVMe or a local SSD scratch space, increase dataloader workers, and ensure your filesystem and mount options are tuned (e.g., noatime on data volumes).

One-command Bash health check you can customize

Save as ai-healthcheck.sh and run. It prints WARN/OK flags you can push to logs or monitoring.

#!/usr/bin/env bash
set -euo pipefail

echo "== AI Health Check ($(date)) =="

warn=0
ok()   { printf "OK   - %s\n" "$1"; }
fail() { printf "WARN - %s\n" "$1"; warn=1; }

# OS/Kernel
kernel=$(uname -r)
ok "Kernel: $kernel"

# CPU/Mem summary
mem=$(free -h | awk '/Mem:/ {print $2 " total, " $3 " used"}')
ok "Memory: $mem"

# Swap
if swapon --show | grep -q .; then
  ok "Swap: enabled"
else
  fail "Swap: disabled (OK for some HPC, but can trigger OOM on mixed workloads)"
fi

# NVIDIA GPU (optional)
if command -v nvidia-smi >/dev/null; then
  gpus=$(nvidia-smi -L | wc -l)
  ok "NVIDIA GPUs detected: $gpus"
  util=$(nvidia-smi --query-gpu=temperature.gpu,clocks.sm,power.draw --format=csv,noheader 2>/dev/null | head -n1)
  ok "GPU snapshot (temp, sm_clock, power): $util"
  # Check PCIe link width/speed quick-parse
  if nvidia-smi -q | egrep -qi 'Current Link Speed.*Gen[3-6]'; then
    ok "PCIe link speed: looks modern"
  else
    fail "PCIe link: verify speed/width (may be Gen1/low width)"
  fi
else
  ok "No NVIDIA driver found (skipping GPU checks)"
fi

# Sensors
if command -v sensors >/dev/null; then
  if sensors 2>/dev/null | egrep -q '([0-9]{2,3}\.[0-9]°C)'; then
    ok "Sensors: temperatures readable"
  else
    fail "Sensors: run 'sudo sensors-detect --auto'"
  fi
fi

# Storage quick checks
if command -v smartctl >/dev/null; then
  if sudo smartctl -H /dev/nvme0 >/dev/null 2>&1; then
    health=$(sudo smartctl -H /dev/nvme0 | awk -F: '/overall-health/ {gsub(/ /,""); print $2}')
    [[ "$health" == "PASSED" ]] && ok "NVMe health: PASSED" || fail "NVMe health: $health"
  fi
fi

# Recent kernel errors
if dmesg | tail -n 200 | egrep -qi 'error|failed|iommu|pcie|nvme|gpu|oom'; then
  fail "Recent dmesg shows potential errors; review 'dmesg -T'"
else
  ok "dmesg: no recent critical errors detected"
fi

if [[ $warn -eq 0 ]]; then
  echo "== Result: PASS =="
else
  echo "== Result: WARN (see items above) =="
fi

Run:

chmod +x ai-healthcheck.sh
./ai-healthcheck.sh

Conclusion and next steps (CTA)

Health checks don’t have to be complicated. With a handful of native tools and a scripted checklist, you can:

  • Validate drivers, thermals, and PCIe links before wasting a single GPU-hour.

  • Catch failing disks and memory pressure before they break runs.

  • Build reproducible baselines across nodes and time.

Next steps:

  • Automate the script via cron/systemd timers on every AI node.

  • Capture outputs to a central log or monitoring system (e.g., pipe to a log collector).

  • Extend the check to validate your actual framework stack (e.g., tiny Torch or TensorFlow inference smoke test) and dataset paths.

If you want a ready-to-run, distro-agnostic script bundle with HTML summaries, let me know your environment (NVIDIA/AMD, distro, single node vs. cluster), and I’ll tailor it to your setup.

Related posts