Posted on
• Artificial Intelligence

Artificial Intelligence Bash Monitoring Scripts

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Bash Monitoring Scripts: Smarter Ops With Just Shell

What if 30 lines of Bash could warn you about a runaway process, a spiking error rate, or a disk about to overheat—before users notice? Heavy observability stacks are great, but they’re not always practical on edge nodes, air‑gapped systems, or tiny VMs. This article shows how to add “just enough AI” to your Bash monitoring so it adapts, learns baselines, and flags anomalies in real time.

You’ll get practical scripts that:

  • Stream metrics from the kernel and common tools

  • Apply lightweight “intelligence” (EWMA, z-scores) to detect anomalies

  • Alert on CPU/memory spikes, log surges, and SMART temperature drift

  • Run under cron or systemd—no heavyweight agents required

Why Bash AI Monitoring Is Valid

  • Ubiquitous: Bash and coreutils are on every Linux box.

  • Transparent: Easy to audit, tweak, and extend with one-liners.

  • Composable: Pipes, files, and standard tools make powerful pipelines.

  • Low overhead: Perfect for constrained hosts and “observe-without-impact.”

  • Good-enough intelligence: Simple adaptive stats (exponentially weighted averages, running z-scores) detect real anomalies without full ML stacks.

Prerequisites (install once)

We’ll reference a few standard packages. Install what you need:

  • sysstat (iostat, mpstat, sar)

  • gawk (awk with extras)

  • jq (JSON parsing; used for SMART JSON)

  • lm-sensors (hardware sensors; optional)

  • smartmontools (disk SMART health)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat gawk jq lm-sensors smartmontools
# Optional: enable sysstat collection on Debian-based systems
sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat 2>/dev/null || true
sudo systemctl enable --now sysstat
# Optional: sensor detection (non-interactive)
sudo sensors-detect --auto || true

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y sysstat gawk jq lm_sensors smartmontools
sudo systemctl enable --now sysstat
sudo sensors-detect --auto || true

openSUSE (zypper):

sudo zypper install -y sysstat gawk jq lm_sensors smartmontools
sudo systemctl enable --now sysstat
sudo sensors-detect --auto || true

Note: sensors-detect can load/advise kernel modules; review output if running on production.

1) Collect: Lightweight, Reliable Metrics

Script: collect CPU and memory usage every second into CSV (epoch,cpu_pct,mem_pct). No external deps beyond /proc.

#!/usr/bin/env bash
# collect_cpu_mem.sh
# Outputs: epoch,cpu_pct,mem_pct

set -euo pipefail

read -r PREV_TOTAL PREV_IDLE < <(awk '/^cpu /{t=0; for(i=2;i<=NF;i++) t+=$i; print t, $5}' /proc/stat)

while :; do
  sleep 1

  # CPU
  read -r CUR_TOTAL CUR_IDLE < <(awk '/^cpu /{t=0; for(i=2;i<=NF;i++) t+=$i; print t, $5}' /proc/stat)
  DIFF_TOTAL=$((CUR_TOTAL - PREV_TOTAL))
  DIFF_IDLE=$((CUR_IDLE - PREV_IDLE))
  CPU_PCT=$(( 100 * (DIFF_TOTAL - DIFF_IDLE) / (DIFF_TOTAL == 0 ? 1 : DIFF_TOTAL) ))
  PREV_TOTAL=$CUR_TOTAL
  PREV_IDLE=$CUR_IDLE

  # Memory
  MEM_TOTAL_KB=$(awk '/MemTotal:/{print $2}' /proc/meminfo)
  MEM_AVAIL_KB=$(awk '/MemAvailable:/{print $2}' /proc/meminfo)
  MEM_USED_PCT=$(( 100 * (MEM_TOTAL_KB - MEM_AVAIL_KB) / (MEM_TOTAL_KB == 0 ? 1 : MEM_TOTAL_KB) ))

  printf "%s,%s,%s\n" "$(date +%s)" "$CPU_PCT" "$MEM_USED_PCT"
done

Alternative (if sysstat installed): mpstat 1 for per-CPU or iostat -dx 1 for disk IO; sar -n DEV 1 for net. They’re battle-tested for long runs.

2) Add Intelligence: EWMA/Z-Score Anomaly Detection

We’ll apply an exponentially weighted moving average (EWMA) and variance to adapt to the host’s baseline. If a new value deviates from the learned mean by more than a z-threshold, we flag it.

#!/usr/bin/env bash
# ewma_anomaly.sh
# Usage: stream numbers (one per line) into this. It prints status lines.
# Params (optional): alpha(0-1) z_threshold
# Example: tail -f metrics | cut -d, -f2 | ./ewma_anomaly.sh 0.2 3

set -euo pipefail
ALPHA=${1:-0.2}
Z=${2:-3}

awk -v a="$ALPHA" -v zt="$Z" '
function abs(x){return x<0?-x:x}
BEGIN { mu=""; var=0; eps=1e-9; }
{
  x=$1+0
  if (mu=="") { mu=x; var=0; print "INIT", x; next }
  diff=x-mu
  mu = a*x + (1-a)*mu
  var = a*(diff*diff) + (1-a)*var
  sigma = sqrt(var+eps)
  z = (x-mu)/(sigma>0?sigma:1)
  status = (abs(z) >= zt) ? "ANOMALY" : "OK"
  printf "%s x=%.2f mu=%.2f sigma=%.2f z=%.2f\n", status, x, mu, sigma, z
}'

Example: Detect CPU spikes adaptively

./collect_cpu_mem.sh | awk -F, '{print $2}' | ./ewma_anomaly.sh 0.2 3

  • alpha: learning rate. Higher (0.2–0.4) adapts faster, lower (0.05–0.15) is more stable.

  • z-threshold: 3 is common; lower it (2–2.5) for more sensitivity.

You can also watch memory:

./collect_cpu_mem.sh | awk -F, '{print $3}' | ./ewma_anomaly.sh 0.15 2.5

3) Logs: Rate Spikes With AI-Lite

Detect surges in ERROR/CRITICAL events using EWMA of per-minute rates. This helps catch cascading failures or DDoS symptoms before saturation.

#!/usr/bin/env bash
# log_rate_ai.sh
# Usage: tail -F /var/log/syslog | ./log_rate_ai.sh "ERROR|CRITICAL" 60
# Requires: gawk

set -euo pipefail
PATTERN=${1:-ERROR}
WINDOW=${2:-60}
ALPHA=${3:-0.3}
Z=${4:-3}

gawk -v pat="$PATTERN" -v w="$WINDOW" -v a="$ALPHA" -v zt="$Z" '
BEGIN { count=0; mu=""; var=0; prev_bucket=-1; eps=1e-9 }
{
  t=systime()
  b=int(t/w)
  if (prev_bucket==-1) prev_bucket=b
  if (b!=prev_bucket) {
    rate = count/w
    if (mu=="") { mu=rate; var=0; print strftime("%F %T"), "INIT rate=" rate; }
    else {
      diff=rate-mu
      mu = a*rate + (1-a)*mu
      var = a*(diff*diff) + (1-a)*var
      sigma = sqrt(var+eps)
      z = (rate-mu)/(sigma>0?sigma:1)
      status = (z>=zt) ? "ANOMALY" : "OK"
      printf "%s %s rate=%.2f mu=%.2f sigma=%.2f z=%.2f\n", strftime("%F %T"), status, rate, mu, sigma, z
    }
    count=0
    prev_bucket=b
  }
  if ($0 ~ pat) count++
}'

Example:

tail -F /var/log/syslog | ./log_rate_ai.sh 'ERROR|CRITICAL' 60 0.3 3

Tune the window (e.g., 30s for chatty services, 120s for quieter ones).

4) Storage: SMART Temperature Drift (Predictive Failure Clues)

A rising drive temperature can precede failures. We’ll parse SMART JSON via jq and watch for anomalies.

#!/usr/bin/env bash
# smart_temp_ai.sh
# Usage: ./smart_temp_ai.sh /dev/sda | ./ewma_anomaly.sh 0.1 3
# Requires: smartmontools, jq

set -euo pipefail
DEV=${1:-/dev/sda}

while :; do
  T=$(smartctl -A -j "$DEV" 2>/dev/null | jq -r '.temperature.current // empty')
  if [ -z "$T" ]; then
    # Fallback attempt (non-JSON)
    T=$(smartctl -A "$DEV" 2>/dev/null | awk "/Temperature/ {for(i=1;i<=NF;i++) if (\$i ~ /^[0-9]+$/) last=\$i} END{print last+0}")
  fi
  [ -n "$T" ] && echo "$T"
  sleep 60
done

Run it:

./smart_temp_ai.sh /dev/sda | ./ewma_anomaly.sh 0.1 3

Optional: For CPU temps using lm-sensors

sensors | awk '/^Package id 0:/{gsub("+","",$4); gsub("°C","",$4); print $4}' | ./ewma_anomaly.sh 0.1 3

5) Wire It Up: Alerts, Cron, systemd

  • Simple alert on anomaly:
./collect_cpu_mem.sh | awk -F, '{print $2}' | ./ewma_anomaly.sh 0.2 3 | \
  awk '/ANOMALY/ { system("logger -t ai-bash \"CPU anomaly: "$0"\""); print > "/tmp/ai-cpu.alert"; fflush(); }'
  • systemd service + timer (runs log spike watcher every boot, continuously):

Service (save as /etc/systemd/system/ai-log.service):

[Unit]
Description=AI Bash Log Spike Monitor

[Service]
ExecStart=/bin/sh -c 'tail -F /var/log/syslog | /usr/local/bin/log_rate_ai.sh "ERROR|CRITICAL" 60 0.3 3'
Restart=always
RestartSec=5

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-log.service
  • Cron example (SMART temp every minute to syslog):
* * * * * /usr/local/bin/smart_temp_ai.sh /dev/sda | head -n1 | awk '{print "SMART_TEMP="$1}' | logger -t ai-bash

Real-World Patterns These Catch

  • Runaway process: CPU z-score spikes to 5+ and stays high—page early, kill or throttle before saturation.

  • Memory leak: Slow EWMA climb in mem_pct beyond normal envelope over hours—roll restart off-peak.

  • DDoS or cascading failure: ERROR rate anomaly across services—trigger autoscaling or circuit breakers.

  • Impending disk failure: Disk temperature trending up outside historical range—migrate data now, RMA drive.

Tips for Productionizing

  • Calibrate: Start with INIT-only logging for a day to learn baselines, then enable alerts.

  • Persistence: Save mu/var to files to preserve state across restarts (write last values on exit, read on start).

  • Whisper, don’t shout: Rate-limit alerts; log OK lines at debug level, anomalies at warning.

  • Context: Include PID, top offenders (ps -eo pid,pcpu,comm --sort=-pcpu | head), and recent changes in alert messages.

  • Scope: Run per-host and per-critical process streams for clarity.

Conclusion and Next Steps

You don’t need a full ML stack to get practical, adaptive monitoring. With a few Bash scripts and basic statistics, you can learn baselines, detect outliers, and respond faster—especially on minimal or isolated hosts.

Your next steps: 1) Install prerequisites with apt/dnf/zypper. 2) Drop the scripts into /usr/local/bin and make them executable. 3) Start with CPU/memory anomaly detection; tune alpha and z-thresholds. 4) Add log spike and SMART monitoring for broader coverage. 5) Wrap them in systemd services and alerts that fit your environment.

Have a great minimalist AI Bash trick or improvement? Share your snippet and results—it might save someone else’s 3 a.m. outage.

Related posts