Posted on
• Artificial Intelligence

Intelligent System Health Checks Using Bash

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Intelligent System Health Checks Using Bash

Most “health check” scripts are either too naive (spamming you any time CPU spikes for a second) or too heavy (requiring a full-blown monitoring stack you don’t want on every node). What if you could get context-aware, low-noise, actionable checks with nothing but Bash and a few standard utilities?

This article shows how to build intelligent, self-tuning system health checks in Bash that:

  • Adapt thresholds based on recent history (baselining)

  • Correlate multiple metrics before alarming (reduce noise)

  • Emit machine- and human-readable output

  • Run anywhere you can run a shell (VMs, containers, edge devices)

You’ll get a ready-to-use script, installation steps for common distros, and examples that reflect real-world pains like disk growth, I/O wait storms, or transient network flaps.

Why Bash is a valid choice

  • Ubiquity and zero vendor lock-in: Bash and coreutils exist on almost every Linux box, from servers to containers.

  • Low overhead: No agents, databases, or daemons needed.

  • Composability: Easy to schedule with cron or systemd timers; pipe outputs to logs, mail, or webhooks.

  • Portability: Uses /proc, /sys, and standard tools that are stable across distros.

This isn’t a replacement for enterprise observability—but it’s perfect for:

  • Small estates or “pets”

  • Restricted environments (air-gapped, minimal containers)

  • First-line signal before you deploy a larger stack

Prerequisites (with apt, dnf, and zypper)

The core script uses only common tools (awk, sed, grep, df). Some optional checks use packages below. Install what you need.

  • sysstat (iostat/mpstat) [optional: we’ll also provide a fallback]

  • smartmontools (SMART disk health) [optional]

  • lm-sensors (temperatures) [optional]

  • bc (floating point arithmetic)

  • curl (for webhooks; optional)

  • jq (pretty-print JSON; optional)

  • mailx/mailutils (email alerting; optional)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat smartmontools lm-sensors bc curl jq mailutils

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y sysstat smartmontools lm_sensors bc curl jq mailx

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y sysstat smartmontools lm_sensors bc curl jq mailx

Notes:

  • Temperatures: after installing lm-sensors, run sudo sensors-detect --auto (may prompt on some systems).

  • SMART requires root for smartctl on most systems: run script as root if you enable SMART checks.

  • If you don’t want mail or jq, skip those packages.

The script: intelligent, low-noise system health

Save as /usr/local/sbin/ishc.sh and chmod +x /usr/local/sbin/ishc.sh.

#!/usr/bin/env bash
set -euo pipefail

# Intelligent System Health Check (ishc.sh)
# - Adaptive baselines (exponential moving average)
# - Correlated alerts to reduce noise
# - Human summary + JSON output
# - Minimal dependencies (sysstat/lm-sensors/smartctl optional)

HOST="${HOSTNAME:-$(hostname)}"
STATE_FILE="/var/tmp/ishc.state"
ALPHA="0.3"          # EMA smoothing (0..1); higher = more reactive
INTERVAL_IOWAIT="1"  # seconds for iowait sampling
PING_TARGET="${PING_TARGET:-1.1.1.1}"
PING_COUNT=3
DISK_PATHS="${DISK_PATHS:-/ /var /home}" # Adjust to your mounts

# Thresholds (base). "Intelligent" logic compares to baseline too.
THRESH_LOAD_PER_CORE=2.0
THRESH_MEM_AVAILABLE_PCT=10
THRESH_IOWAIT_PCT=8
THRESH_LATENCY_MS=120
THRESH_PACKET_LOSS_PCT=5
THRESH_DISK_USED_PCT=90
THRESH_TEMP_C=85

# Utilities
has() { command -v "$1" >/dev/null 2>&1; }

calc() {
  # bc-based float math; usage: calc "100 * 0.2"
  echo "scale=6; $*" | bc -l
}

percent() {
  # percent x/y
  local x="$1" y="$2"
  if [ "$y" -eq 0 ]; then echo "0"; else calc "($x * 100) / $y"; fi
}

read_kv_state() {
  if [ -f "$STATE_FILE" ]; then
    # shellcheck disable=SC1090
    source "$STATE_FILE"
  fi
}

write_kv_state() {
  cat > "$STATE_FILE" <<EOF
BASE_LOAD_PER_CORE=${BASE_LOAD_PER_CORE:-$LOAD_PER_CORE}
BASE_IOWAIT=${BASE_IOWAIT:-$IOWAIT}
BASE_LATENCY=${BASE_LATENCY:-$LATENCY}
BASE_MEM_AVAIL_PCT=${BASE_MEM_AVAIL_PCT:-$MEM_AVAIL_PCT}
BASE_DISK_USED_PCT=${BASE_DISK_USED_PCT:-$DISK_USED_PCT_MAX}
EOF
}

ema_update() {
  local base="$1" current="$2"
  if [ -z "${base:-}" ]; then
    echo "$current"
  else
    calc "$ALPHA * $current + (1 - $ALPHA) * $base"
  fi
}

get_cores() {
  if has nproc; then nproc; else grep -c ^processor /proc/cpuinfo; fi
}

get_load_per_core() {
  local load_1m cores
  load_1m=$(awk '{print $1}' /proc/loadavg)
  cores=$(get_cores)
  calc "$load_1m / $cores"
}

get_mem_available_pct() {
  # Use MemAvailable if present; fallback to free
  if grep -q MemAvailable /proc/meminfo; then
    local total avail
    total=$(awk '/MemTotal/{print $2}' /proc/meminfo)
    avail=$(awk '/MemAvailable/{print $2}' /proc/meminfo)
    # Values in kB; percent of total that's available
    percent "$avail" "$total"
  else
    # Fallback via free (older distros)
    local total avail
    read -r _ total used free shared buff cache avail <<<"$(free -k | awk '/Mem:/{print $1,$2,$3,$4,$5,$6,$7,$7}')"
    percent "$avail" "$total"
  fi
}

get_iowait_pct() {
  # Prefer mpstat for accuracy; else sample /proc/stat over 1s
  if has mpstat; then
    mpstat 1 1 | awk '/Average|all/ {iowait=$6} END{if(iowait=="") iowait=0; print iowait}'
  else
    # fields: user nice sys idle iowait irq softirq steal guest guest_nice
    local a b
    a=$(awk '/^cpu /{print $2,$3,$4,$5,$6,$7,$8,$9,$10,$11}' /proc/stat)
    sleep "$INTERVAL_IOWAIT"
    b=$(awk '/^cpu /{print $2,$3,$4,$5,$6,$7,$8,$9,$10,$11}' /proc/stat)
    awk -v A="$a" -v B="$b" '
      BEGIN{
        split(A,aa," "); split(B,bb," ");
        sumA=0; sumB=0; for(i=1;i<=10;i++){sumA+=aa[i]; sumB+=bb[i]}
        dUser=bb[1]-aa[1]; dNice=bb[2]-aa[2]; dSys=bb[3]-aa[3]; dIdle=bb[4]-aa[4];
        dIow=bb[5]-aa[5]; dIrq=bb[6]-aa[6]; dSoft=bb[7]-aa[7]; dSteal=bb[8]-aa[8];
        total=(sumB - sumA); if(total<=0){total=1}
        printf("%.2f", (dIow*100.0)/total)
      }'
  fi
}

get_net_latency_loss() {
  # Outputs: latency_ms packet_loss_pct
  if has ping; then
    local out rtt loss
    out=$(ping -c '"$PING_COUNT"' -w 4 "$PING_TARGET" 2>/dev/null || true)
    loss=$(echo "$out" | awk -F',' '/packet loss/{gsub(/ /,""); print $3}' | tr -d '%' || echo "100")
    rtt=$(echo "$out" | awk -F'/' '/rtt|round-trip/{print $5}')
    if [ -z "$rtt" ]; then rtt=10000; fi
    echo "$rtt" "$loss"
  else
    echo "0 100"
  fi
}

get_disk_used_pct_max_and_growth() {
  # Scan defined DISK_PATHS; compute max usage and growth vs last run
  local maxp=0 thisp path
  local total_used=0 total_size=0
  for path in $DISK_PATHS; do
    if df -P "$path" >/dev/null 2>&1; then
      thisp=$(df -P "$path" | awk 'NR==2{gsub(/%/,"",$5); print $5}')
      if [ "$thisp" -gt "$maxp" ]; then maxp="$thisp"; fi
      # Summed for growth heuristics
      read -r _ size used _ <<<"$(df -P -k "$path" | awk 'NR==2{print $1,$2,$3,$4}')"
      total_used=$((total_used + used))
      total_size=$((total_size + size))
    fi
  done
  local used_pct_all=0
  if [ "$total_size" -gt 0 ]; then
    used_pct_all=$(calc "($total_used * 100) / $total_size")
  fi
  local growth="0"
  if [ -n "${BASE_DISK_USED_PCT:-}" ]; then
    growth=$(calc "$used_pct_all - $BASE_DISK_USED_PCT")
  fi
  echo "$maxp" "$growth"
}

get_max_temp_c() {
  if has sensors; then
    sensors 2>/dev/null | awk '/\+?[0-9]+(\.[0-9]+)?°C/{
      match($0,/\+?([0-9]+(\.[0-9]+)?)°C/,m); if(m[1]>max) max=m[1]
    } END{if(max=="") max=0; print max}'
  else
    echo "0"
  fi
}

get_smart_health() {
  # Return 0 if all PASS or smartctl not present, 1 if any FAIL
  if ! has smartctl; then return 0; fi
  local d rv=0
  for d in /dev/sd? /dev/nvme?n?; do
    [ -e "$d" ] || continue
    if smartctl -H "$d" 2>/dev/null | grep -q "PASSED"; then :; else rv=1; fi
  done
  return "$rv"
}

# Collect current metrics
LOAD_PER_CORE=$(get_load_per_core)
MEM_AVAIL_PCT=$(get_mem_available_pct)
IOWAIT=$(get_iowait_pct)
read LATENCY PACKET_LOSS <<<"$(get_net_latency_loss)"
read DISK_USED_PCT_MAX DISK_GROWTH_PCT <<<"$(get_disk_used_pct_max_and_growth)"
MAX_TEMP_C=$(get_max_temp_c)
SMART_FAIL=0; if ! get_smart_health; then SMART_FAIL=1; fi

# Load previous baselines
read_kv_state

# Update baselines
BASE_LOAD_PER_CORE=$(ema_update "${BASE_LOAD_PER_CORE:-}" "$LOAD_PER_CORE")
BASE_IOWAIT=$(ema_update "${BASE_IOWAIT:-}" "$IOWAIT")
BASE_LATENCY=$(ema_update "${BASE_LATENCY:-}" "$LATENCY")
BASE_MEM_AVAIL_PCT=$(ema_update "${BASE_MEM_AVAIL_PCT:-}" "$MEM_AVAIL_PCT")
BASE_DISK_USED_PCT=$(ema_update "${BASE_DISK_USED_PCT:-}" "$DISK_USED_PCT_MAX")

# Intelligent alerting
SEVERITY=0
ALERTS=()

add_alert() {
  local sev="$1" msg="$2"
  ALERTS+=("$sev|$msg")
  if [ "$sev" -gt "$SEVERITY" ]; then SEVERITY="$sev"; fi
}

# 1) High load only if it exceeds both static and baseline, and corroborated by I/O wait
if (( $(echo "$LOAD_PER_CORE > $THRESH_LOAD_PER_CORE" | bc -l) )) && \
   (( $(echo "$LOAD_PER_CORE > ($BASE_LOAD_PER_CORE * 1.6)" | bc -l) )); then
  if (( $(echo "$IOWAIT > 5" | bc -l) )); then
    add_alert 2 "High load per core ($LOAD_PER_CORE) with iowait ${IOWAIT}%"
  else
    add_alert 1 "Elevated load per core ($LOAD_PER_CORE) vs baseline ($BASE_LOAD_PER_CORE)"
  fi
fi

# 2) Low memory only if MemAvailable is low and swapping observed recently (heuristic via iowait or growth)
if (( $(echo "$MEM_AVAIL_PCT < $THRESH_MEM_AVAILABLE_PCT" | bc -l) )); then
  add_alert 2 "Low MemAvailable (${MEM_AVAIL_PCT}%)"
fi

# 3) Disk usage: alert only if high and still growing
if (( $(echo "$DISK_USED_PCT_MAX > $THRESH_DISK_USED_PCT" | bc -l) )); then
  if (( $(echo "$DISK_GROWTH_PCT > 0.5" | bc -l) )); then
    add_alert 2 "Disk critically full (max ${DISK_USED_PCT_MAX}%), +${DISK_GROWTH_PCT}% since last run"
  else
    add_alert 1 "Disk high (max ${DISK_USED_PCT_MAX}%)"
  fi
fi

# 4) Network: consider both loss and latency; compare to baseline to avoid flapping
if (( $(echo "$PACKET_LOSS > $THRESH_PACKET_LOSS_PCT" | bc -l) )); then
  add_alert 2 "Network packet loss ${PACKET_LOSS}% to ${PING_TARGET}"
elif (( $(echo "$LATENCY > $THRESH_LATENCY_MS" | bc -l) )) && \
     (( $(echo "$LATENCY > ($BASE_LATENCY * 2.2)" | bc -l) )); then
  add_alert 1 "High latency ${LATENCY}ms (baseline ${BASE_LATENCY}ms)"
fi

# 5) Temperature
if (( $(echo "$MAX_TEMP_C > $THRESH_TEMP_C" | bc -l) )); then
  add_alert 2 "High temperature ${MAX_TEMP_C}°C"
fi

# 6) SMART
if [ "$SMART_FAIL" -ne 0 ]; then
  add_alert 2 "SMART health check failing on one or more disks"
fi

# Persist new baselines
write_kv_state

# Output
TS=$(date -Iseconds)
HUMAN_SUMMARY="[$TS] $HOST: "
if [ "${#ALERTS[@]}" -eq 0 ]; then
  HUMAN_SUMMARY+="OK"
else
  for a in "${ALERTS[@]}"; do HUMAN_SUMMARY+=$(echo " | ${a#*|}"); done
fi
echo "$HUMAN_SUMMARY"

# JSON payload (simple; jq optional for pretty)
JSON=$(cat <<EOF
{"ts":"$TS","host":"$HOST","severity":$SEVERITY,
 "metrics":{
   "load_per_core":$LOAD_PER_CORE,
   "mem_available_pct":$MEM_AVAIL_PCT,
   "iowait_pct":$IOWAIT,
   "latency_ms":$LATENCY,
   "packet_loss_pct":$PACKET_LOSS,
   "disk_used_pct_max":$DISK_USED_PCT_MAX,
   "disk_growth_pct":$DISK_GROWTH_PCT,
   "max_temp_c":$MAX_TEMP_C,
   "smart_fail":$SMART_FAIL
 },
 "baseline":{
   "load_per_core":$BASE_LOAD_PER_CORE,
   "iowait_pct":$BASE_IOWAIT,
   "latency_ms":$BASE_LATENCY,
   "mem_available_pct":$BASE_MEM_AVAIL_PCT,
   "disk_used_pct":$BASE_DISK_USED_PCT
 },
 "alerts":[
   $(printf '"%s",' "${ALERTS[@]#*|}" 2>/dev/null | sed 's/,$//')
 ]
}
EOF
)
echo "$JSON"

# Exit code: 0 OK, 1 WARN, 2 CRIT
exit "$SEVERITY"

How it’s “intelligent”

  • Baselining with EMA: The script learns normal behavior (e.g., nightly ETL spikes) and only warns when you exceed both a static threshold and your own baseline.

  • Correlation: High load alone isn’t critical; high load with high I/O wait usually is.

  • Growth-aware disk alerts: It flags disks that are both full and still growing.

  • Dual output: Quick human summary plus JSON for pipelines.

3–5 actionable steps to deploy

1) Install the optional tools you plan to use

  • Debian/Ubuntu:

    • sudo apt update && sudo apt install -y sysstat smartmontools lm-sensors bc curl jq mailutils

  • Fedora/RHEL/CentOS:

    • sudo dnf install -y sysstat smartmontools lm_sensors bc curl jq mailx

  • openSUSE:

    • sudo zypper refresh && sudo zypper install -y sysstat smartmontools lm_sensors bc curl jq mailx

  • Optional post-install:

    • sudo sensors-detect --auto (temperature support)
    • Run the script as root if you enable SMART checks.

2) Drop the script and run it by hand

sudo install -m 0755 ishc.sh /usr/local/sbin/ishc.sh
sudo /usr/local/sbin/ishc.sh
  • First run initializes the baseline. Don’t panic if it’s quiet—that’s the point.

3) Schedule it

  • With cron (every 5 minutes):
# as root
echo '*/5 * * * * /usr/local/sbin/ishc.sh >> /var/log/ishc.log 2>&1' | sudo tee /etc/cron.d/ishc
sudo systemctl restart cron || sudo systemctl restart crond || true
  • With systemd timers (preferred):
sudo tee /etc/systemd/system/ishc.service >/dev/null <<'UNIT'
[Unit]
Description=Intelligent System Health Check

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/ishc.sh
UNIT

sudo tee /etc/systemd/system/ishc.timer >/dev/null <<'TIMER'
[Unit]
Description=Run Intelligent System Health Check every 5 minutes

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=ishc.service

[Install]
WantedBy=timers.target
TIMER

sudo systemctl daemon-reload
sudo systemctl enable --now ishc.timer

4) Integrate with your alerting

  • Email (optional):

    • Debian/Ubuntu:
    • sudo apt install -y mailutils
    • Fedora/RHEL/CentOS:
    • sudo dnf install -y mailx
    • openSUSE:
    • sudo zypper install -y mailx

  • Pipe summary to mail when severity > 0:

/usr/local/sbin/ishc.sh | tee -a /var/log/ishc.log | awk 'NR==1 && $0 !~ /OK$/' | mail -s "ALERT: $(hostname)" you@example.com
  • Webhook (Slack/Teams/custom):
/usr/local/sbin/ishc.sh | awk 'NR==2' | curl -sS -X POST -H 'Content-Type: application/json' -d @- https://example.com/webhook

5) Tune for your environment

  • Adjust DISK_PATHS to your important mounts:
export DISK_PATHS="/ /var /data /backups"

  • Tighten or relax thresholds by setting environment variables before running, or edit the script defaults:

    • THRESH_LOAD_PER_CORE, THRESH_MEM_AVAILABLE_PCT, THRESH_IOWAIT_PCT, THRESH_LATENCY_MS, THRESH_PACKET_LOSS_PCT, THRESH_DISK_USED_PCT, THRESH_TEMP_C

  • Increase ALPHA for faster adaptation to new baselines; decrease for stability.

Real-world examples and how this helps

  • Nightly ETL on a database host: Classic checks page you nightly. EMA baseline means your check knows nights are busier; it only warns if tonight is abnormally busy compared to recent nights.

  • Disk creeping due to runaway logs: The script won’t just tell you the disk is at 91%; it’ll also say “+1.8% since last run,” signaling active growth that may warrant immediate action.

  • Transient network flaps: An isolated 100ms RTT spike won’t fire if your baseline is ~90ms, but 300ms sustained with packet loss will.

  • Heat-induced throttling: Temperature checks capture thermal issues on dense hosts or poorly ventilated racks.

Notes, caveats, and extensions

  • SMART and sensors may be unavailable in containers or VMs. The script degrades gracefully if commands are missing.

  • Running as root is recommended when using SMART or reading certain logs.

  • Extend quickly:

    • Add journal correlation: journalctl -p 3 --since "5 min ago"
    • Check inode pressure: df -Pi
    • Track process count or zombie processes
    • Send JSON to a central endpoint for fleet dashboards

  • For extremely minimal environments, trim optional checks to reduce dependencies.

Conclusion and next step (CTA)

You don’t need a heavyweight agent to get meaningful, low-noise health insights. With a Bash script, a few standard tools, and some light intelligence (baselining + correlation), you can catch the right problems at the right time—without drowning in false positives.

Next steps:

  • Install the prerequisites for your distro.

  • Drop the script in /usr/local/sbin, run it once to seed the baseline, then enable the systemd timer.

  • Tailor thresholds, paths, and outputs to your environment.

  • Wire alerts to your email or webhook—and enjoy signal over noise.

If you’d like a hardened version with packaging and CI, or a variant tuned for containers/Kubernetes, say the word and I’ll share a drop-in bundle.

Related posts