Artificial Intelligence Cluster Health Checks with Bash: Catch Issues Before They Cost You

If you run AI workloads on a GPU cluster, every minute of degraded performance is money burned—slow training, missed deadlines, and frustrated teams. The hard truth: many issues hide in plain sight. A flaky NIC, a throttled GPU, a misaligned driver across nodes—any of these can silently knock 10–30% off throughput.

This guide shows you how to build practical, Bash-first health checks for AI clusters. You’ll learn why health checks matter, which tools to install, and 3–5 actionable checks you can automate today—across Kubernetes, Slurm, or bare metal.

Why health checks are non-negotiable

• AI clusters are heterogeneous by nature. Mismatched GPU driver/toolkit versions across nodes can break containers or silently disable features like peer-to-peer (P2P).

• Training is I/O and network hungry. Small spikes in disk latency or RDMA errors can cascade into step time variance.

• GPUs throttle to protect themselves. Thermal and power cap events are common and can cut effective performance drastically if unnoticed.

• Schedulers mask symptoms. Pending pods/jobs might look like capacity issues when it’s actually a sick node or a broken device plugin.

A consistent, automated health routine stops firefighting and preserves your model throughput.

What you’ll use (and how to install it)

The following tools are widely available on Linux and easy to script. Install the ones you need on all nodes.

• pdsh or pssh (parallel remote execution)

  • apt: sudo apt-get update && sudo apt-get install -y pdsh (or pssh)
  • dnf: sudo dnf install -y pdsh (or pssh)
  • zypper: sudo zypper install -y pdsh (or pssh)

• sysstat (iostat, mpstat, sar)

  • apt: sudo apt-get install -y sysstat
  • dnf: sudo dnf install -y sysstat
  • zypper: sudo zypper install -y sysstat

• smartmontools (disk SMART)

  • apt: sudo apt-get install -y smartmontools
  • dnf: sudo dnf install -y smartmontools
  • zypper: sudo zypper install -y smartmontools

• nvme-cli (NVMe health)

  • apt: sudo apt-get install -y nvme-cli
  • dnf: sudo dnf install -y nvme-cli
  • zypper: sudo zypper install -y nvme-cli

• ethtool (NIC stats)

  • apt: sudo apt-get install -y ethtool
  • dnf: sudo dnf install -y ethtool
  • zypper: sudo zypper install -y ethtool

• nvtop (optional, interactive GPU viewer)

  • apt: sudo apt-get install -y nvtop
  • dnf: sudo dnf install -y nvtop
  • zypper: sudo zypper install -y nvtop

• pciutils (optional, for PCIe link speed/width checks)

  • apt: sudo apt-get install -y pciutils
  • dnf: sudo dnf install -y pciutils
  • zypper: sudo zypper install -y pciutils

• kubectl (if you use Kubernetes)

  • apt: sudo apt-get install -y kubectl
  • dnf: sudo dnf install -y kubectl || sudo dnf install -y kubernetes-client
  • zypper: sudo zypper install -y kubectl || sudo zypper install -y kubernetes-client

• Slurm client tools (if you use Slurm)

  • apt: sudo apt-get install -y slurm-client
  • dnf: sudo dnf install -y slurm
  • zypper: sudo zypper install -y slurm

Note: nvidia-smi ships with the NVIDIA driver. If nvidia-smi is missing, install the proper NVIDIA driver for your distribution and GPU.

---

Core checks you should automate

Below are five checks that consistently surface real issues in GPU clusters. Each can be run once or scheduled. Examples assume a node naming pattern like node01..node32; adjust for your environment.

1) Consistency check: drivers, CUDA, GPUs across all nodes

Mismatched drivers or mixed GPU SKUs often cause instability or hidden performance cliffs.

• On a single node:

echo "== GPU and driver versions =="
nvidia-smi --query-gpu=name,driver_version,compute_cap,temperature.gpu --format=csv

• Across many nodes with pdsh:

NODES="node[01-32]"
pdsh -w $NODES 'hostname; nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv,noheader' | sort | uniq -c

• Check CUDA libraries inside your AI containers to ensure toolkit/driver compatibility:

docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi

• Verify PCIe link width/speed (common cause of 10–20% slowdowns when degraded):

nvidia-smi -q | grep -A3 "PCI"
# Or for detail:
sudo lspci -vv | grep -A10 -i nvidia | egrep 'LnkCap|LnkSta'

What to look for:

• Exactly one driver version across identical nodes.

• Compute capability matches expectations for your GPUs.

• PCIe link speed/width matches slot capability (e.g., Gen4 x16, not x8).

2) GPU health: thermals, throttling, ECC, utilization

Thermal and power caps regularly limit performance during long runs.

• Quick utilization and memory snapshot:

nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,memory.used,memory.total --format=csv

• Detailed health (temperatures, power, clocks, ECC):

nvidia-smi -q -d TEMPERATURE,POWER,PERFORMANCE,ECC | sed -n '1,150p'

• Watch throttling over time (p=power, m=memory, u=util):

sudo nvidia-smi dmon -s pmu -d 1 -c 60

• Across nodes (flag any GPU above 85C or with non-zero ECC):

pdsh -w $NODES 'nvidia-smi -q -d TEMPERATURE,ECC' | egrep "Hostname:|GPU Current Temp|ECC Errors"

What to look for:

• Temps comfortably below throttling thresholds (typically < 85C).

• ECC error counts at 0 for training jobs that demand determinism.

• Utilization that matches your workload profile (e.g., not pegged at 0% unintentionally).

3) Scheduler health: Kubernetes or Slurm

It’s not just nodes—your scheduler and GPU plugins must be healthy.

• Kubernetes basics:

kubectl get nodes -o wide
kubectl describe node <node-name> | egrep "Allocatable|Capacity|Conditions"
kubectl get pods -A --field-selector=status.phase!=Running -o wide
# If metrics-server is present:
kubectl top nodes
kubectl top pods -A

• Check the NVIDIA device plugin is advertising GPUs:

kubectl get daemonset -A | grep -i nvidia
kubectl -n kube-system logs daemonset/nvidia-device-plugin-daemonset --tail=100

• Slurm basics:

sinfo -R            # Reasons nodes are down/drained
squeue -t pending -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
scontrol show node <node-name> | egrep "Gres|State|Reason"

What to look for:

• No NotReady nodes; consistent GPU capacity/allocatable.

• No long-running pending pods/jobs due to GPU device plugin issues or node labels/taints.

• Queue backlogs that align with expected capacity.

4) Storage I/O and disk health

I/O stalls can elongate data loader steps and underutilize GPUs.

• I/O saturation and latency:

iostat -xz 2 3   # look at %util and await for busy disks

• Smart health (SATA/SAS):

sudo smartctl -H /dev/sda
sudo smartctl -a /dev/sda | egrep "Reallocated|CRC|Pending|Uncorrect"

• NVMe health:

sudo nvme smart-log /dev/nvme0 | egrep "critical_warning|temperature|media_errors|num_err_log_entries"

What to look for:

• Disks near 100% utilization during training jobs.

• Rising reallocated/pending sectors or NVMe media errors—replace before failure.

5) Network fabric sanity: drops, errors, speed

Networking issues show up as step-time jitter and sync hangs.

• Interface-level errors/drops:

ip -s link

• Driver counters (example for eno1; use your interface):

sudo ethtool -S eno1 | egrep "rx_*err|tx_*err|drop|fail|timeout"

• Quick socket and TCP summary:

ss -s

What to look for:

• Non-zero error/drop counters increasing over time.

• Mismatched NIC speeds/duplex.

• Congestion—validate your data path and storage layout.

---

A ready-to-run node health script (Bash)

Use this as a daily check. It exits non-zero on “bad” findings so you can alert via cron or your scheduler.

#!/usr/bin/env bash
# ai-node-health.sh
set -euo pipefail

THRESH_TEMP=85     # C
THRESH_POWERCAP=1  # 1 event triggers warning
BAD=0

echo "host=$(hostname)"
date

if ! command -v nvidia-smi >/dev/null 2>&1; then
  echo "ERROR: nvidia-smi not found (install NVIDIA driver)"; exit 2
fi

echo "== GPU summary =="
nvidia-smi --query-gpu=index,name,driver_version,temperature.gpu,power.draw,clocks.sm,utilization.gpu,memory.used,memory.total --format=csv

# Temperature check
HOT=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader | awk -v t=$THRESH_TEMP '$1>t{print}')
if [[ -n "${HOT}" ]]; then
  echo "WARN: GPU temperature above ${THRESH_TEMP}C"
  BAD=1
fi

# ECC check (if supported)
if nvidia-smi -q -d ECC | grep -q "ECC Mode"; then
  ECCERR=$(nvidia-smi -q -d ECC | grep -A5 "ECC Errors" | grep -E "Single|Double" | awk -F: '{gsub(/ /,"",$2); if($2>0) print}')
  if [[ -n "${ECCERR}" ]]; then
    echo "WARN: Non-zero ECC error counters"
    BAD=1
  fi
fi

echo "== CPU/Mem snapshot =="
free -h || true
mpstat 1 3 || true
vmstat 1 3 || true

echo "== Disk I/O (quick) =="
iostat -xz 1 2 || true

echo "== Disk health =="
for dev in /dev/nvme*n1 /dev/sd[a-z]; do
  [[ -e "$dev" ]] || continue
  if [[ "$dev" == /dev/nvme* ]]; then
    sudo nvme smart-log "$dev" | egrep "critical_warning|temperature|media_errors|num_err_log_entries" || true
  else
    sudo smartctl -H "$dev" || true
  fi
done

echo "== Network counters =="
ip -s link || true

exit $BAD

Make it executable:

chmod +x ai-node-health.sh

Run locally:

./ai-node-health.sh

Run across the cluster with pdsh:

NODES="node[01-32]"
pdsh -w $NODES 'bash -s' < ./ai-node-health.sh | tee cluster-health-$(date +%F).log

Schedule daily via cron on a jump host and alert on non-zero exits.

---

Real-world example: the “everything slowed down 15%” mystery

A team noticed epoch times creeping up. GPUs looked fine at a glance. The health script flagged two nodes with lower PCIe link width. A reseat restored x16 lanes and performance recovered. Lesson: verify PCIe link health alongside temps and utilization.

Check:

nvidia-smi -q | grep -A5 "PCI"
sudo lspci -vv | grep -A10 -i nvidia | egrep 'LnkCap|LnkSta'

---

Bonus: operator-friendly one-liners

• Compare driver versions across all nodes:

pdsh -w node[01-32] 'nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1' | awk '{print $2}' | sort | uniq -c

• List pods not using GPUs but scheduled on GPU nodes (Kubernetes):

kubectl get pods -A -o jsonpath='{range .items[?(@.spec.nodeName!="")]}{.spec.nodeName}{"\t"}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].resources.limits.nvidia\.com/gpu}{"\n"}{end}' | awk -F'\t' '$4==""'

• See nodes with increasing RX/TX errors:

for i in {1..5}; do date; ip -s link | egrep "^[0-9]|errors|dropped"; sleep 5; done

---

Conclusion and next steps

Health checks are the cheapest performance boost you’ll ever buy. Start with Bash, automate the five checks above, and run them daily. Next steps:

• Drop ai-node-health.sh into version control, tune thresholds, and schedule it with cron or your scheduler.

• Use pdsh/pssh to cover the entire fleet and keep logs in a central place.

• Add dashboards later with your preferred stack (e.g., Prometheus + Grafana, plus NVIDIA DCGM exporter) once the basics are automated.

If you want a copy-paste–ready starter, begin by installing the tools, saving the script, and running your first full-cluster sweep today.