Posted on
• Artificial Intelligence

Artificial Intelligence Linux Health Checks

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Health Checks: Smarter Bash For Your Servers

Your server looked fine two minutes ago. Now it’s paging you. Traditional health checks tell you what is broken after it breaks. What if your checks could also notice weirdness early, weigh what matters most, and propose next steps? That’s the promise of bringing an AI mindset to Linux health checks—without abandoning your beloved Bash.

In this post you’ll:

  • Set up a portable health-check toolchain on any major Linux distro

  • Build a single Bash script that emits clean JSON and a metrics CSV

  • Schedule it with systemd timers

  • Add a lightweight anomaly detector to catch issues before they bite

  • See real-world examples and triage hints

The goal: actionable, privacy-safe, distro-agnostic checks that run locally and evolve with your systems.

Why “AI” belongs in your health checks

  • Noise reduction: Static thresholds create alert fatigue. Simple anomaly detection helps flag unusual behavior relative to your system’s own baseline.

  • Prioritization: Not every spike is equal. A model can weigh multi-metric context (CPU + memory + disk) to gauge urgency.

  • Explainability: Pair AI scoring with human-friendly heuristics to suggest likely fixes.

  • Practicality: You don’t need GPUs or cloud APIs—just Bash plus a tiny Python helper.

1) Install the toolchain

We’ll use these packages:

  • sysstat (mpstat, iostat) for CPU and I/O

  • smartmontools for disk health

  • lm-sensors for temperatures and voltages

  • jq for JSON

  • python3 and pip for a small anomaly detector

  • bc for some shell math

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat smartmontools lm-sensors jq python3 python3-pip bc

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y sysstat smartmontools lm_sensors jq python3 python3-pip bc

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y sysstat smartmontools lm_sensors jq python3 python3-pip bc

Optional but recommended (detect sensors):

sudo sensors-detect
# Follow prompts; generally safe to accept defaults

2) A single Bash snapshot that writes JSON and CSV

Save this as /usr/local/bin/health-snapshot.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

OUTDIR="/var/log/ai-health"
mkdir -p "$OUTDIR"
TS="$(date -Is)"
HOST="$(hostname -f 2>/dev/null || hostname)"
JSON_FILE="$OUTDIR/health-$TS.json"
CSV_FILE="$OUTDIR/metrics.csv"

# Load averages (1, 5, 15)
read LOAD1 LOAD5 LOAD15 _ < /proc/loadavg

# Memory (kB)
MEM_TOTAL=$(awk '/MemTotal:/ {print $2}' /proc/meminfo)
MEM_AVAIL=$(awk '/MemAvailable:/ {print $2}' /proc/meminfo)
SWAP_TOTAL=$(awk '/SwapTotal:/ {print $2}' /proc/meminfo)
SWAP_FREE=$(awk '/SwapFree:/ {print $2}' /proc/meminfo)
MEM_USED_KB=$((MEM_TOTAL - MEM_AVAIL))
MEM_USED_PCT=$(awk -v u="$MEM_USED_KB" -v t="$MEM_TOTAL" 'BEGIN{printf "%.2f", (u/t)*100}')
SWAP_USED_PCT=$(awk -v t="$SWAP_TOTAL" -v f="$SWAP_FREE" 'BEGIN{ if (t==0) {print 0} else {printf "%.2f", ((t-f)/t)*100}}')

# CPU usage via mpstat
CPU_IDLE=$(mpstat 1 1 | awk '/Average:/ && $3 ~ /all|CPU/ {print $(NF)}')
CPU_IDLE=${CPU_IDLE:-0}
CPU_USED_PCT=$(awk -v idle="$CPU_IDLE" 'BEGIN{printf "%.2f", 100-idle}')

# Disks usage (per mount)
DISKS_JSON=$(df -P -B1 | awk 'NR>1 {printf "{\"fs\":\"%s\",\"mount\":\"%s\",\"size\":%s,\"used\":%s,\"avail\":%s,\"usep\":\"%s\"}\n",$1,$6,$2,$3,$4,$5}' \
  | jq -s 'map(.usep=(.usep|sub("%$";"")|tonumber))')

# Temperatures (best effort)
SENSORS_JSON="{}"
if command -v sensors >/dev/null; then
  if sensors -j >/dev/null 2>&1; then
    SENSORS_JSON=$(sensors -j || echo "{}")
  else
    SENSORS_JSON=$(sensors 2>/dev/null | jq -R -s 'split("\n")' 2>/dev/null || echo "{}")
  fi
fi

# Top CPU processes
TOPPROCS=$(ps -eo pid,comm,%cpu,%mem --no-headers --sort=-%cpu | head -n 5 \
  | awk '{printf "{\"pid\":%s,\"comm\":\"%s\",\"cpu\":%s,\"mem\":%s}\n",$1,$2,$3,$4}' | jq -s '.')

# Network summary (sockets)
NET_SUMMARY=$(ss -s 2>/dev/null || echo "ss not available")

# Simple heuristics
DISK_ALERT=$(echo "$DISKS_JSON" | jq '[.[]|select(.usep>=85)]|length>0')
MEM_ALERT=$(awk -v p="$MEM_USED_PCT" 'BEGIN{print (p>=90)?"true":"false"}')
CPU_ALERT=$(awk -v p="$CPU_USED_PCT" 'BEGIN{print (p>=90)?"true":"false"}')

# Build JSON
jq -n \
  --arg ts "$TS" \
  --arg host "$HOST" \
  --arg load1 "$LOAD1" --arg load5 "$LOAD5" --arg load15 "$LOAD15" \
  --arg cpu_used "$CPU_USED_PCT" \
  --arg mem_used "$MEM_USED_PCT" --arg swap_used "$SWAP_USED_PCT" \
  --arg net_summary "$NET_SUMMARY" \
  --argjson disks "$DISKS_JSON" \
  --argjson sensors "$SENSORS_JSON" \
  --argjson top "$TOPPROCS" \
  --argjson disk_alert "$DISK_ALERT" \
  --argjson mem_alert "$MEM_ALERT" \
  --argjson cpu_alert "$CPU_ALERT" \
  '{
    ts:$ts, host:$host,
    load:{l1:($load1|tonumber), l5:($load5|tonumber), l15:($load15|tonumber)},
    cpu:{used:($cpu_used|tonumber)},
    memory:{used_pct:($mem_used|tonumber), swap_used_pct:($swap_used|tonumber)},
    disks:$disks,
    sensors:$sensors,
    top_processes:$top,
    net_summary:$net_summary,
    alerts:{cpu:$cpu_alert, memory:$mem_alert, disk:$disk_alert}
  }' | tee "$JSON_FILE" >/dev/null

# Append CSV row (for anomaly detection)
MAX_DISK=$(echo "$DISKS_JSON" | jq '[.[].usep]|max // 0')
if [[ ! -f "$CSV_FILE" ]]; then
  echo "ts,load1,cpu_used,mem_used_pct,swap_used_pct,max_disk_usep" > "$CSV_FILE"
fi
echo "$TS,$LOAD1,$CPU_USED_PCT,$MEM_USED_PCT,$SWAP_USED_PCT,$MAX_DISK" >> "$CSV_FILE"

echo "Wrote $JSON_FILE and updated $CSV_FILE"

Make it executable:

sudo install -m 0755 /usr/local/bin/health-snapshot.sh /usr/local/bin/health-snapshot.sh

Optional: If you also want disk SMART health, add this snippet inside the script (after DISKS_JSON), but note parsing can vary by device:

# SMART quick overall health check (optional)
if command -v smartctl >/dev/null; then
  for d in $(lsblk -ndo NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}'); do
    smartctl -H "$d" 2>/dev/null | awk -v dev="$d" '/SMART overall-health/ {print dev": "$0}'
  done
fi

3) Schedule it with systemd timers (every 5 minutes)

Create a one-shot service: /etc/systemd/system/ai-health-snapshot.service

[Unit]
Description=AI Health Snapshot

[Service]
Type=oneshot
ExecStart=/usr/local/bin/health-snapshot.sh

Create a timer: /etc/systemd/system/ai-health-snapshot.timer

[Unit]
Description=Run AI Health Snapshot every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Enable and test:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-health-snapshot.timer
sudo systemctl start ai-health-snapshot.service
journalctl -u ai-health-snapshot.service -n 50 --no-pager
ls -1 /var/log/ai-health/

4) Add a lightweight AI anomaly detector

We’ll train an Isolation Forest on your own history to flag unusual states and suggest triage steps.

Install Python deps (via pip):

python3 -m pip install --user numpy scikit-learn

Create /usr/local/bin/anomaly-detect.py:

#!/usr/bin/env python3
import csv, sys, os
import numpy as np
from sklearn.ensemble import IsolationForest

csv_file = sys.argv[1] if len(sys.argv) > 1 else "/var/log/ai-health/metrics.csv"

with open(csv_file) as f:
    rows = list(csv.DictReader(f))

if len(rows) < 16:
    print("Not enough history (need >=16 rows) for anomaly detection.")
    sys.exit(0)

X = np.array([[float(r['load1']),
               float(r['cpu_used']),
               float(r['mem_used_pct']),
               float(r['swap_used_pct']),
               float(r['max_disk_usep'])] for r in rows])

# Train on all but the latest row, score the latest
train, latest = X[:-1], X[-1].reshape(1, -1)

clf = IsolationForest(n_estimators=200, contamination='auto', random_state=42)
clf.fit(train)
score = clf.decision_function(latest)[0]   # higher = more normal
pred = clf.predict(latest)[0]              # -1 anomalous, 1 normal

# Explain with simple z-scores
means = train.mean(axis=0)
stds = train.std(axis=0)
zs = (latest[0] - means) / np.where(stds == 0, 1, stds)
labels = ['load1', 'cpu_used', 'mem_used_pct', 'swap_used_pct', 'max_disk_usep']
top_idx = int(np.argmax(np.abs(zs)))

print(f"Anomaly={'YES' if pred == -1 else 'no'} score={score:.3f}")
print("Top deviating metric:", labels[top_idx], f"z={zs[top_idx]:.2f}")

if pred == -1:
    hints = []
    if labels[top_idx] == 'max_disk_usep' and latest[0][4] > 85:
        hints.append("Disk nearing capacity. Clean large logs (journalctl --disk-usage, logrotate), prune containers, or expand the filesystem.")
    if labels[top_idx] == 'mem_used_pct' and latest[0][2] > 90:
        hints.append("High memory pressure. Check top processes (ps/top), tune services, or add RAM.")
    if labels[top_idx] == 'cpu_used' and latest[0][1] > 90:
        hints.append("Sustained CPU saturation. Find hot processes, consider cgroup limits, or scale out.")
    if labels[top_idx] == 'swap_used_pct' and latest[0][3] > 20:
        hints.append("Swap usage rising. Investigate memory leaks, tune vm.swappiness, reduce overcommit, or add RAM.")
    if labels[top_idx] == 'load1' and latest[0][0] > (os.cpu_count() or 1):
        hints.append("Load exceeds CPU cores. Check I/O wait (iostat), run queue length, and lock contention.")
    for h in hints:
        print("Hint:", h)

Make it executable:

sudo install -m 0755 /usr/local/bin/anomaly-detect.py /usr/local/bin/anomaly-detect.py

Add a timer to run it hourly and log results: /etc/systemd/system/ai-health-anomaly.service

[Unit]
Description=AI Health Anomaly Detection

[Service]
Type=oneshot
ExecStart=/usr/local/bin/anomaly-detect.py

/etc/systemd/system/ai-health-anomaly.timer

[Unit]
Description=Run AI Health Anomaly Detection hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-health-anomaly.timer
journalctl -u ai-health-anomaly.service -n 20 --no-pager

5) Real-world playbook (what to look for)

  • Disk fills up “out of nowhere”

    • Symptom: anomaly flags max_disk_usep; JSON shows a mount > 90%.
    • Triage:
    • Check large files: sudo du -xhd1 / | sort -h | tail
    • Journal size: journalctl --disk-usage
    • Containers/images: docker system df or podman system df
    • Consider logrotate and alert thresholds at 80/85/90%

  • Memory pressure and swap creep

    • Symptom: mem_used_pct high, swap_used_pct trending up
    • Triage:
    • Top processes: ps -eo pid,comm,rss,%mem --sort=-rss | head
    • Page activity: vmstat 1 5
    • Tune: lower vm.swappiness, fix leaks, right-size caches

  • CPU saturation or run queue spikes

    • Symptom: cpu_used > 90, load1 > cores
    • Triage:
    • Hot processes: top -H or pidstat -u 1
    • I/O wait: iostat -xz 1 3 (from sysstat)
    • Consider cgroups, concurrency limits, or scaling out

  • Thermal throttling (hidden performance killer)

    • Symptom: sensors show high temps
    • Triage:
    • Clean airflow, verify fans
    • Laptop/edge devices: governers, tlp (where applicable)
    • Sustained heat often correlates with CPU slowdowns

Conclusion and Call To Action

You don’t need a data science team to make your Linux health checks smarter. Start with the Bash snapshot, schedule it, and layer in the anomaly detector. As your metrics CSV grows, your system’s own baseline becomes its best defense against surprise outages.

Next steps:

  • Deploy the snapshot script and timers to a staging server today

  • Watch /var/log/ai-health for a day or two

  • Enable the anomaly timer and review the logs

  • Iterate: add service-specific checks (databases, queues), and expand heuristics

If you want a follow-up, I can share examples for service-specific rules (PostgreSQL, NGINX, Kubernetes nodes) and guidance for alerting via email, Slack, or Prometheus-compatible exporters.

Stay proactive. Let Bash do the collecting—and let a tiny bit of AI help you see the story in the data.

Related posts