Posted on
• Artificial Intelligence

Artificial Intelligence Server Maintenance with Bash

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Server Maintenance with Bash: A Practical Playbook

AI servers work hard: they crunch large models, stream predictions, and shuffle terabytes of data. When they fail at 3 a.m., you need visibility and fast recovery. You don’t need another heavyweight agent to get there—you can do a lot with Bash, systemd, and a few standard Linux tools.

This guide shows you how to keep AI servers healthy using Bash:

  • Why Bash is a great fit

  • What to install

  • 3–5 actionable scripts and timers you can drop in today

  • How to schedule and alert on issues

If you run GPU inference or training nodes, these patterns will save you time and outages.

Why Bash for AI server maintenance?

  • It’s everywhere: no new runtime or agent. Bash + coreutils = works on almost every Linux box.

  • It’s composable: glue together nvidia-smi, smartctl, nvme, df, systemctl, and curl quickly.

  • It’s debuggable: read the script, test one line at a time, no mystery.

  • It’s automatable: run from systemd timers, cron, or CI without extra services.

Prerequisites: tools you’ll use

We’ll rely on common CLI tools. Install them with your package manager.

Packages:

  • lm-sensors (CPU temps and fan speeds)

  • smartmontools (disk SMART health)

  • nvme-cli (NVMe health and wear)

  • curl (alerts/webhooks)

  • rsync (backups and syncs)

  • tmux (optional, for interactive maintenance)

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y lm-sensors smartmontools nvme-cli curl rsync tmux
sudo sensors-detect --auto

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y lm_sensors smartmontools nvme-cli curl rsync tmux
sudo sensors-detect --auto

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y lm_sensors smartmontools nvme-cli curl rsync tmux
sudo sensors-detect --auto

Notes:

  • NVIDIA GPU info comes from nvidia-smi, which ships with the NVIDIA driver. If you use AMD, similar checks can be done with rocm-smi—swap the GPU queries accordingly.

  • These scripts assume systemd. If you don’t use systemd, adapt the timers to cron.

1) A quick, actionable health check for AI workloads

This script checks CPU/memory pressure, root disk usage, GPU temperature/utilization (if NVIDIA GPUs are present), and storage health. It logs to syslog and optionally posts to a Slack/Teams/Discord webhook if thresholds are exceeded.

Install it:

sudo tee /usr/local/bin/ai-health.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Thresholds (tune to your environment)
DISK_PCT_WARN=90         # % used on /
MEM_PCT_WARN=95          # % of RAM used
GPU_TEMP_WARN=85         # Celsius
NVME_WARN_ON_CRIT=1      # Non-zero 'critical_warning' is bad

HOST="$(hostname -f 2>/dev/null || hostname)"
TS="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
ALERTS=()
LOG_PREFIX="[ai-health] $TS $HOST"

has() { command -v "$1" >/dev/null 2>&1; }

# Memory pressure
read -r MEM_TOTAL_KB MEM_AVAIL_KB < <(awk '/MemTotal:/ {t=$2} /MemAvailable:/ {a=$2} END {print t, a}' /proc/meminfo)
if [[ -n "${MEM_TOTAL_KB:-}" && -n "${MEM_AVAIL_KB:-}" && "$MEM_TOTAL_KB" -gt 0 ]]; then
  MEM_USED_PCT=$(( (100 * (MEM_TOTAL_KB - MEM_AVAIL_KB)) / MEM_TOTAL_KB ))
  [[ "$MEM_USED_PCT" -ge "$MEM_PCT_WARN" ]] && ALERTS+=("RAM pressure high: ${MEM_USED_PCT}% used")
fi

# Root disk usage
ROOT_PCT=$(df -P / | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
[[ -n "$ROOT_PCT" && "$ROOT_PCT" -ge "$DISK_PCT_WARN" ]] && ALERTS+=("Root filesystem nearly full: ${ROOT_PCT}% used")

# GPU checks (NVIDIA)
if has nvidia-smi; then
  mapfile -t GPU_LINES < <(nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits 2>/dev/null || true)
  for line in "${GPU_LINES[@]}"; do
    IFS=',' read -r IDX NAME TEMP UTIL MEM_USED MEM_TOTAL <<<"$(echo "$line" | sed 's/, /,/g')"
    if [[ -n "${TEMP:-}" && "$TEMP" -ge "$GPU_TEMP_WARN" ]]; then
      ALERTS+=("GPU$IDX ($NAME) hot: ${TEMP}C")
    fi
  done
fi

# NVMe health (if any)
if has nvme; then
  while read -r DEV TYPE; do
    [[ "$TYPE" != "disk" ]] && continue
    [[ "$DEV" != nvme* ]] && continue
    CW=$(nvme smart-log "/dev/$DEV" 2>/dev/null | awk '/critical_warning/ {print $3}' || echo "")
    if [[ -n "$CW" && "$CW" != "0x00" && "$NVME_WARN_ON_CRIT" -eq 1 ]]; then
      ALERTS+=("NVMe /dev/$DEV critical_warning: $CW")
    fi
  done < <(lsblk -ndo NAME,TYPE)
fi

# SATA/other SMART quick health
if has smartctl; then
  while read -r DEV TYPE; do
    [[ "$TYPE" != "disk" ]] && continue
    [[ "$DEV" == nvme* ]] && continue
    H=$(smartctl -H "/dev/$DEV" 2>/dev/null | awk -F: '/SMART overall-health/ {gsub(/^[ \t]+/,"",$2); print $2}')
    [[ "$H" == "PASSED" || -z "$H" ]] || ALERTS+=("SMART failing: /dev/$DEV ($H)")
  done < <(lsblk -ndo NAME,TYPE)
fi

# Log outcome
if (( ${#ALERTS[@]} )); then
  MSG="$LOG_PREFIX ALERT: $(printf '%s; ' "${ALERTS[@]}")"
  echo "$MSG"
  logger -t ai-health "$MSG"

  # Optional webhook alert (Slack, Teams, Discord, etc.)
  if [[ -n "${AI_HEALTH_WEBHOOK:-}" ]] && has curl; then
    PAYLOAD=$(printf '{"text":"%s"}' "$(echo "$MSG" | sed 's/"/\"/g')")
    curl -fsS -m 5 -H 'Content-Type: application/json' -d "$PAYLOAD" "$AI_HEALTH_WEBHOOK" >/dev/null || true
  fi
  exit 1
else
  MSG="$LOG_PREFIX OK"
  echo "$MSG"
  logger -t ai-health "$MSG"
fi
EOF
sudo chmod +x /usr/local/bin/ai-health.sh

Usage:

  • Optional: export AI_HEALTH_WEBHOOK (Slack/Teams/Discord incoming webhook URL) in the systemd service environment (shown next) or in /etc/environment.

  • Run ad hoc: sudo /usr/local/bin/ai-health.sh

2) Schedule the health check with systemd timers

Create a oneshot service and a 5-minute timer.

sudo tee /etc/systemd/system/ai-health.service > /dev/null <<'EOF'
[Unit]
Description=AI Server Health Check

[Service]
Type=oneshot
Environment=AI_HEALTH_WEBHOOK=
ExecStart=/usr/local/bin/ai-health.sh
EOF

sudo tee /etc/systemd/system/ai-health.timer > /dev/null <<'EOF'
[Unit]
Description=Run AI Server Health Check every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=30s
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ai-health.timer
sudo systemctl status ai-health.timer --no-pager

Tip:

  • Set Environment=AI_HEALTH_WEBHOOK=https://... in ai-health.service to enable alerts.

  • Check logs: journalctl -u ai-health.service -n 50 --no-pager

3) GPU-aware watchdog for your inference service

Some inference servers silently stall: the process is “running” but GPUs are idle for hours. This watchdog restarts a systemd unit if GPUs stay idle for too long, or if the unit stops.

Install it:

sudo tee /usr/local/bin/gpu-watchdog.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Name of your service (edit this!)
SERVICE="ai-inference.service"

# Idle thresholds
IDLE_UTIL_PCT=5         # Consider GPU idle below this utilization
RESTART_AFTER_N_RUNS=6  # e.g., with a 10m timer, 6 runs = ~1 hour of idleness
STATE_DIR="/run/gpu-watchdog"
STATE_FILE="$STATE_DIR/idle.count"

mkdir -p "$STATE_DIR"
count=0
[[ -f "$STATE_FILE" ]] && read -r count < "$STATE_FILE" || true

is_active() { systemctl is-active --quiet "$SERVICE"; }

avg_gpu_util() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null \
      | awk '{sum+=$1; n+=1} END {if(n>0) print sum/n; else print -1}'
  else
    echo -1
  fi
}

# Ensure service is running
if ! is_active; then
  logger -t gpu-watchdog "Service $SERVICE is not active; starting"
  systemctl start "$SERVICE" || true
  echo 0 > "$STATE_FILE"
  exit 0
fi

UTIL=$(avg_gpu_util)
if [[ "$UTIL" -ge 0 && "${UTIL%.*}" -lt "$IDLE_UTIL_PCT" ]]; then
  count=$((count + 1))
else
  count=0
fi
echo "$count" > "$STATE_FILE"

if [[ "$count" -ge "$RESTART_AFTER_N_RUNS" ]]; then
  logger -t gpu-watchdog "GPU idle for prolonged period (avg util ~${UTIL}%). Restarting $SERVICE"
  systemctl restart "$SERVICE" || true
  echo 0 > "$STATE_FILE"
fi
EOF
sudo chmod +x /usr/local/bin/gpu-watchdog.sh

Timer and service:

sudo tee /etc/systemd/system/gpu-watchdog.service > /dev/null <<'EOF'
[Unit]
Description=GPU Watchdog for AI Inference Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/gpu-watchdog.sh
EOF

sudo tee /etc/systemd/system/gpu-watchdog.timer > /dev/null <<'EOF'
[Unit]
Description=Run GPU Watchdog every 10 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
AccuracySec=1min
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now gpu-watchdog.timer

Customize:

  • Set SERVICE= in the script to your unit (e.g., vllm.service, textgen.service, torchserve.service).

  • For AMD, replace the utilization query with rocm-smi --showuse.

4) Safe, incremental backups of models and checkpoints with rsync

Models and checkpoints change frequently. Use rsync with hard-linked snapshots for space-efficient, point-in-time copies. This works best when backing up to a local path or an NFS mount. For remote SSH targets, you can still mirror with rsync (snapshots require extra remote-side logic).

Create excludes and script:

sudo tee /etc/ai-backup-excludes.txt > /dev/null <<'EOF'
# Examples to skip temp files
*.tmp
cache/
wandb/
node_modules/
__pycache__/
EOF

sudo tee /usr/local/bin/ai-backup.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# What to back up
SRC_DIRS=(
  /srv/models
  /srv/checkpoints
)
# Destination must be a local path (e.g., mounted NFS) for snapshots
DEST="/var/backups/ai"
STAMP="$(date +%F)"     # e.g., 2026-07-07

mkdir -p "$DEST"
SNAP_DIR="$DEST/$STAMP"
CUR_DIR="$DEST/current"

# Create a snapshot by cloning the last state via hard links, then update
if [[ -d "$CUR_DIR" ]]; then
  cp -al "$CUR_DIR" "$SNAP_DIR"
else
  mkdir -p "$SNAP_DIR"
fi

# Sync latest into 'current'
mkdir -p "$CUR_DIR"
rsync -aHAX --delete --delete-excluded --exclude-from=/etc/ai-backup-excludes.txt \
  "${SRC_DIRS[@]}" "$CUR_DIR/"

# Retention: keep 14 days of snapshots
find "$DEST" -maxdepth 1 -type d -name '20[0-9][0-9]-*' -mtime +14 -print -exec rm -rf {} +

logger -t ai-backup "Backup complete to $DEST ($STAMP)"
EOF
sudo chmod +x /usr/local/bin/ai-backup.sh

Timer and service (daily at 03:15):

sudo tee /etc/systemd/system/ai-backup.service > /dev/null <<'EOF'
[Unit]
Description=AI models/checkpoints backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ai-backup.sh
EOF

sudo tee /etc/systemd/system/ai-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Daily AI backup at 03:15

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ai-backup.timer

Notes:

  • If you must push to a remote SSH server, a simple mirror works: rsync -a --delete /srv/models/ user@backup:/backups/$HOST/models/ Snapshots on the remote side require server-side commands or a backup system.

5) One-liners you’ll actually use

  • See which processes hold the most GPU memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
  • Kill a runaway GPU process safely (confirm PID first):
sudo kill -15 <PID>; sleep 5; sudo kill -9 <PID> || true
  • Verify CPU temps and fans:
sensors
  • Quick view of NVMe wear and warnings:
nvme smart-log /dev/nvme0
  • Top disk consumers under a directory:
sudo du -xh /var | sort -h | tail -40

Conclusion and next steps

With a handful of Bash scripts and systemd timers, you can:

  • Detect pressure early (RAM, disk, GPU temps)

  • Alert and self-heal stalled inference services

  • Protect models and checkpoints with fast, incremental backups

Your next steps: 1) Install the prerequisite tools (apt/dnf/zypper commands above). 2) Drop in ai-health.sh and enable the timer. 3) Add a webhook to get alerts where you work. 4) Configure the GPU watchdog for your inference unit. 5) Turn on daily backups and verify restore paths.

Start with health checks, then add the watchdog and backups. Small Bash habits lead to big reliability gains.

Related posts