Posted on
• Artificial Intelligence

Artificial Intelligence Server Health Checks

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Server Health Checks: A Bash-First Playbook

Your model is fast—until it isn’t. At 2 a.m., a single hot GPU, a clogged disk queue, or a thrashing memory subsystem can crater throughput, spike latency, and burn your inference SLOs. The fix? Lightweight, automated health checks tuned for AI workloads—implemented with nothing more than Bash and a few standard Linux tools.

This guide explains why AI-specific health checks matter, shows you what to monitor, and gives you a drop-in Bash script plus systemd/cron wiring so you can start getting signal today.

Why AI servers need specialized health checks

  • AI workloads stress different components than typical web apps. GPUs (temperature, ECC, VRAM), I/O pipelines (dataset streaming), and memory pressure matter as much as CPU.

  • Small degradations compound. A GPU running at 88°C might begin throttling; a few seconds of I/O pressure every minute can line up with batch jobs; a single flaky disk can stall a multi-GPU training node.

  • Early warnings prevent firefights. Simple checks on pressure, temps, and endpoints catch issues hours before pager-worthy incidents.

What “healthy” looks like (practical targets)

Use these as starting points and tune for your hardware/SLOs:

  • CPU load: 1‑min load < 0.9 × core count (sustained >1× often signals contention)

  • Memory: MemAvailable > 10% and SwapUsed < 5%

  • Linux pressure (PSI): cpu.some avg10 < 0.5; io.some avg10 < 1.0; memory.some avg10 < 1.0

  • GPU (NVIDIA): temperature < 85°C; VRAM headroom > 10%; ECC enabled (if supported)

  • Disks: SMART overall health PASSED

  • Endpoints: local inference health < 200 ms and HTTP 200

1) Install the tools

We’ll use curl, jq (optional for JSON endpoints), smartmontools, lm-sensors, nvme-cli, and ping.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq smartmontools lm-sensors nvme-cli iputils-ping
  • Fedora/RHEL/CentOS/Alma/Rocky (dnf):
sudo dnf install -y curl jq smartmontools lm_sensors nvme-cli iputils
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y curl jq smartmontools sensors nvme-cli iputils

Then initialize sensors (optional but recommended for CPU temps):

sudo sensors-detect
# Answer "yes" to safe prompts, then:
sensors

Notes:

  • NVIDIA GPUs: install the proprietary driver; it provides nvidia-smi.

  • For SMART/NVMe details you’ll need root to query devices.

2) Drop-in Bash AI health check script

Save as /usr/local/bin/ai-healthcheck.sh and make it executable.

#!/usr/bin/env bash
# ai-healthcheck.sh — Minimal AI server health checks
# Exit codes: 0=OK, 1=WARN, 2=CRIT
set -euo pipefail

# Tunables (override via env)
CPU_LOAD_RATIO_WARN="${CPU_LOAD_RATIO_WARN:-0.9}"          # load1 > cores * ratio -> WARN
MEM_AVAILABLE_MIN_PCT="${MEM_AVAILABLE_MIN_PCT:-10}"       # %
SWAP_MAX_PCT="${SWAP_MAX_PCT:-5}"                          # %
CPU_PSI_SOME_MAX="${CPU_PSI_SOME_MAX:-0.5}"                # avg10
IO_PSI_SOME_MAX="${IO_PSI_SOME_MAX:-1.0}"
MEM_PSI_SOME_MAX="${MEM_PSI_SOME_MAX:-1.0}"
GPU_TEMP_MAX="${GPU_TEMP_MAX:-85}"                         # °C
GPU_MEM_HEADROOM_MIN_PCT="${GPU_MEM_HEADROOM_MIN_PCT:-10}" # %
CPU_TEMP_MAX="${CPU_TEMP_MAX:-90}"                          # °C (optional, via lm-sensors)
NETWORK_MAX_MS="${NETWORK_MAX_MS:-200}"                    # ms per URL
HEALTHCHECK_URLS="${HEALTHCHECK_URLS:-http://127.0.0.1:8000/health}"

overall=0  # 0 OK, 1 WARN, 2 CRIT
issues=()

note_warn() { issues+=("WARN: $1"); (( overall < 1 )) && overall=1; }
note_crit() { issues+=("CRIT: $1"); overall=2; }

pct() { awk -v n="$1" -v d="$2" 'BEGIN{ if(d==0){print 0}else{printf("%.1f", (n/d)*100)} }'; }
gtf() { awk -v a="$1" -v b="$2" 'BEGIN{ if(a>b) exit 0; else exit 1 }'; }

section() { echo "== $1 =="; }

# 1) CPU load vs cores
section "CPU"
cores=$(nproc)
load1=$(awk '{print $1}' /proc/loadavg)
echo "Cores: $cores  Load1: $load1"
awk -v l="$load1" -v c="$cores" -v r="$CPU_LOAD_RATIO_WARN" 'BEGIN{ if (l > c*r) exit 0; else exit 1 }' \
  && note_warn "High CPU load: $load1 > $CPU_LOAD_RATIO_WARN×$cores"

# 2) Memory and swap
section "Memory"
read -r MemTotal_kB MemAvailable_kB SwapTotal_kB SwapFree_kB < <(
  awk '/MemTotal:/{mt=$2} /MemAvailable:/{ma=$2} /SwapTotal:/{st=$2} /SwapFree:/{sf=$2}
       END{printf "%s %s %s %s", mt, ma, st, sf}' /proc/meminfo
)
mem_avail_pct=$(pct "$MemAvailable_kB" "$MemTotal_kB")
swap_used_pct=0
if (( SwapTotal_kB > 0 )); then
  swap_used_kB=$((SwapTotal_kB - SwapFree_kB))
  swap_used_pct=$(pct "$swap_used_kB" "$SwapTotal_kB")
fi
echo "MemAvailable: ${mem_avail_pct}%  SwapUsed: ${swap_used_pct}%"
awk -v p="$mem_avail_pct" -v min="$MEM_AVAILABLE_MIN_PCT" 'BEGIN{ if (p < min) exit 0; else exit 1 }' \
  && note_warn "Low memory available: ${mem_avail_pct}% < ${MEM_AVAILABLE_MIN_PCT}%"
awk -v p="$swap_used_pct" -v max="$SWAP_MAX_PCT" 'BEGIN{ if (p > max) exit 0; else exit 1 }' \
  && note_warn "High swap usage: ${swap_used_pct}% > ${SWAP_MAX_PCT}%"

# 3) Linux pressure (PSI) signals
section "Pressure (PSI avg10)"
psi_val() { awk -v k="$1" '$1==k {for(i=1;i<=NF;i++) if ($i ~ /avg10=/){split($i,a,"="); print a[2]}}' "$2"; }
cpu_psi=$(psi_val some /proc/pressure/cpu)
io_psi=$(psi_val some /proc/pressure/io)
mem_psi=$(psi_val some /proc/pressure/memory)
echo "cpu.some: ${cpu_psi}  io.some: ${io_psi}  memory.some: ${mem_psi}"
awk -v v="$cpu_psi" -v m="$CPU_PSI_SOME_MAX" 'BEGIN{ if (v>m) exit 0; else exit 1 }' && note_warn "CPU pressure high: $cpu_psi > $CPU_PSI_SOME_MAX"
awk -v v="$io_psi"  -v m="$IO_PSI_SOME_MAX"  'BEGIN{ if (v>m) exit 0; else exit 1 }' && note_warn "I/O pressure high: $io_psi > $IO_PSI_SOME_MAX"
awk -v v="$mem_psi" -v m="$MEM_PSI_SOME_MAX" 'BEGIN{ if (v>m) exit 0; else exit 1 }' && note_warn "Memory pressure high: $mem_psi > $MEM_PSI_SOME_MAX"

# 4) GPU (NVIDIA) checks
section "GPU"
if command -v nvidia-smi >/dev/null 2>&1; then
  mapfile -t gpus < <(nvidia-smi --query-gpu=uuid,temperature.gpu,memory.total,memory.used,ecc.mode.current --format=csv,noheader,nounits 2>/dev/null || true)
  if ((${#gpus[@]}==0)); then
    echo "No NVIDIA GPUs detected by nvidia-smi."
  else
    for line in "${gpus[@]}"; do
      IFS=',' read -r uuid temp_c mem_total_mb mem_used_mb ecc_mode <<<"$(echo "$line" | sed 's/, /,/g')"
      mem_headroom_pct=$(awk -v u="$mem_used_mb" -v t="$mem_total_mb" 'BEGIN{ if (t==0) print 0; else printf("%.1f", 100 - (u/t)*100) }')
      echo "GPU $uuid  Temp: ${temp_c}°C  VRAM: ${mem_used_mb}/${mem_total_mb} MB (headroom ${mem_headroom_pct}%)  ECC: ${ecc_mode}"
      awk -v tc="$temp_c" -v max="$GPU_TEMP_MAX" 'BEGIN{ if (tc>max) exit 0; else exit 1 }' && note_warn "GPU $uuid hot: ${temp_c}°C > ${GPU_TEMP_MAX}°C"
      awk -v h="$mem_headroom_pct" -v min="$GPU_MEM_HEADROOM_MIN_PCT" 'BEGIN{ if (h<min) exit 0; else exit 1 }' && note_warn "GPU $uuid low VRAM headroom: ${mem_headroom_pct}% < ${GPU_MEM_HEADROOM_MIN_PCT}%"
      if [[ "$ecc_mode" =~ [Dd]isabled ]]; then
        note_warn "GPU $uuid ECC disabled"
      fi
    done
  fi
else
  echo "nvidia-smi not found (skip NVIDIA GPU checks)."
fi

# Optional CPU temperature via lm-sensors
if command -v sensors >/dev/null 2>&1; then
  section "CPU temperature (lm-sensors)"
  max_cpu_temp=$(sensors 2>/dev/null | grep -Eo '[+][0-9]+(\.[0-9]+)?°C' | tr -d '+°C' | awk 'NR==1{max=$1} {if($1>max)max=$1} END{if(max=="")max=0; print max}')
  echo "Max CPU temp: ${max_cpu_temp}°C"
  awk -v tc="$max_cpu_temp" -v max="$CPU_TEMP_MAX" 'BEGIN{ if (tc>max) exit 0; else exit 1 }' && note_warn "CPU hot: ${max_cpu_temp}°C > ${CPU_TEMP_MAX}°C"
fi

# 5) Disk health (SMART)
section "Disks (SMART)"
if [[ $EUID -ne 0 ]]; then
  echo "Run as root to read SMART; skipping."
else
  found=0
  for dev in /dev/sd? /dev/nvme?n?; do
    [[ -b "$dev" ]] || continue
    found=1
    if smartctl -H "$dev" >/tmp/sm.$$ 2>&1; then
      if grep -q "PASSED" /tmp/sm.$$; then
        echo "$dev: PASSED"
      else
        echo "$dev: CHECK NEEDED"
        note_warn "SMART health not PASSED on $dev"
      fi
    else
      echo "$dev: SMART query failed"
      note_warn "SMART query failed on $dev"
    fi
    rm -f /tmp/sm.$$
  done
  ((found==0)) && echo "No block devices matched (/dev/sd? /dev/nvme?n?)"
fi

# 6) Network / endpoint health
section "Endpoints"
for url in $HEALTHCHECK_URLS; do
  read -r code t_total < <(curl -sS -o /dev/null -w "%{http_code} %{time_total}" --max-time 5 "$url" || echo "000 9.999")
  ms=$(awk -v t="$t_total" 'BEGIN{printf("%.0f", t*1000)}')
  echo "$url -> $code in ${ms}ms"
  [[ "$code" != "200" ]] && note_crit "Endpoint $url returned $code"
  awk -v x="$ms" -v m="$NETWORK_MAX_MS" 'BEGIN{ if (x>m) exit 0; else exit 1 }' && note_warn "Endpoint $url slow: ${ms}ms > ${NETWORK_MAX_MS}ms"
done

section "Summary"
if (( overall==0 )); then
  echo "OK: all checks passed."
elif (( overall==1 )); then
  printf "%s\n" "${issues[@]}"
  echo "Overall: WARN"
else
  printf "%s\n" "${issues[@]}"
  echo "Overall: CRIT"
fi

exit "$overall"

Make it executable:

sudo install -m 0755 ai-healthcheck.sh /usr/local/bin/ai-healthcheck.sh

Run it:

/usr/local/bin/ai-healthcheck.sh

Environment overrides (examples):

MEM_AVAILABLE_MIN_PCT=15 GPU_TEMP_MAX=82 HEALTHCHECK_URLS="http://127.0.0.1:8000/health https://your-s3-endpoint" /usr/local/bin/ai-healthcheck.sh

3) Wire it up with systemd (or cron)

Systemd oneshot service and timer:

# /etc/systemd/system/ai-healthcheck.service
[Unit]
Description=AI Server Health Check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ai-healthcheck.sh

# /etc/systemd/system/ai-healthcheck.timer
[Unit]
Description=Run AI health check every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Unit=ai-healthcheck.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-healthcheck.timer
sudo systemctl list-timers ai-healthcheck.timer
journalctl -u ai-healthcheck.service -n 50 --no-pager

Cron alternative:

# Run every 5 minutes; log warnings/errors to syslog
echo '*/5 * * * * root /usr/local/bin/ai-healthcheck.sh || logger -t ai-healthcheck "unhealthy (exit $?)"' | sudo tee /etc/cron.d/ai-healthcheck

4) Real-world usage patterns

  • Pre-flight before training: run the script; ensure GPUs have VRAM headroom and temps < 80°C; verify disk SMART passes on data volumes.

  • Inference SLO watch: add your model server health URL to HEALTHCHECK_URLS; keep NETWORK_MAX_MS aligned with your p95.

  • Capacity signals: sustained cpu.some or io.some pressure > 1.0 is an early indicator to scale out, pin I/O, or revisit batch sizing.

  • Kubernetes liveness/readiness: package the script in your image and exec it from a probe to block rollout on known-bad nodes.

Example endpoint validation with jq (optional pretty print):

curl -s http://127.0.0.1:8000/metrics | head
curl -s http://127.0.0.1:8000/health | jq .

If you need jq:

  • apt:
sudo apt update && sudo apt install -y jq
  • dnf:
sudo dnf install -y jq
  • zypper:
sudo zypper install -y jq

5) Troubleshooting notes

  • No GPU section? Install NVIDIA drivers so nvidia-smi is present.

  • No temps from lm-sensors? Re-run sudo sensors-detect, then sudo systemctl restart kmod or reboot.

  • SMART needs root; run the service as root or add appropriate udev rules.

  • PSI files missing? You need a kernel with PSI enabled (most modern distros have it).

Conclusion and next steps

AI infrastructure rewards early, lightweight signals. Start with this Bash-first health check, tune the thresholds to your SLOs, and schedule it with systemd or cron. From there, consider exporting these metrics to Prometheus, paging on CRIT, and baking pre-flight checks into every training and deployment workflow.

Call to action:

  • Install the toolchain using your package manager.

  • Drop in the script and run it once interactively.

  • Wire it to systemd and add your inference endpoints.

  • Iterate thresholds based on a week of observations.

A few minutes now can save you that 2 a.m. firefight later.

Related posts