Posted on
• Artificial Intelligence

Artificial Intelligence Raspberry Pi Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Raspberry Pi Monitoring with Bash: Catch Problems Before They Break Things

Your Raspberry Pi is the unsung hero of your homelab—running Home Assistant, Pi-hole, a camera, or a custom project 24/7. But tiny boards can silently overheat, throttle, or run out of disk space. By the time you notice, logs are gone, services crashed, or the SD card is corrupted.

This post shows how to add a lightweight “AI guardrail” to your Pi using Bash for telemetry and a tiny anomaly detector that learns what “normal” looks like—so it can alert you when something’s off.

What you’ll build:

  • A Bash telemetry collector that logs temperature, load, memory, disk, and throttling flags to CSV

  • A small scikit-learn Isolation Forest that flags anomalies relative to your Pi’s baseline behavior

  • A cron job that runs every minute and alerts you via a webhook (Slack/Discord/Mattermost) or syslog

Why this matters:

  • Baselines beat thresholds: “75°C” might be fine under a compile, but not at night when idle. AI can learn your Pi’s normal patterns.

  • On-device, no cloud: Everything runs locally on a Pi 3/4/5 without heavyweight stacks.

  • Bash-first: Glue everything with shell scripts you can audit and extend.

1) Install prerequisites

We’ll use standard Linux tools plus Python’s scikit-learn. Pick your package manager.

A) apt (Debian, Ubuntu, Raspberry Pi OS):

sudo apt update
sudo apt install -y \
  lm-sensors sysstat jq curl \
  python3 python3-pip python3-sklearn
# Optional (vcgencmd; Raspberry Pi OS usually has this already):
sudo apt install -y libraspberrypi-bin
# Optional load generator for testing:
sudo apt install -y stress-ng

B) dnf (Fedora):

sudo dnf install -y \
  lm_sensors sysstat jq curl \
  python3 python3-pip python3-scikit-learn
# Optional (vcgencmd, package availability varies by spin/repo):
# sudo dnf install -y raspberrypi-userland
# Optional load generator for testing:
sudo dnf install -y stress-ng

C) zypper (openSUSE):

sudo zypper refresh
sudo zypper install -y \
  lm_sensors sysstat jq curl \
  python3 python3-pip python3-scikit-learn
# Optional (vcgencmd, package name may be raspberrypi-userland or similar):
# sudo zypper install -y raspberrypi-userland
# Optional load generator for testing:
sudo zypper install -y stress-ng

Notes:

  • If your repo lacks scikit-learn, you can install it via pip:

    python3 -m pip install --user --upgrade pip
python3 -m pip install --user scikit-learn

  • Initialize lm-sensors once:

    sudo sensors-detect --auto || true

We’ll read temperature from /sys (works across distros on Raspberry Pi), so vcgencmd is optional; when available, we’ll also read throttling flags.

2) Collect telemetry with a simple Bash script

Create /usr/local/bin/pi_telemetry.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/var/log/pi_metrics.csv"
TMPFILE="$(mktemp)"
TIMESTAMP="$(date -Is)"

# CPU temp (millidegrees C -> C). Works on Raspberry Pi.
if [[ -r /sys/class/thermal/thermal_zone0/temp ]]; then
  CPU_TEMP_C="$(awk '{printf "%.2f", $1/1000}' /sys/class/thermal/thermal_zone0/temp)"
else
  # Fallback: try lm-sensors (value may differ by board)
  CPU_TEMP_C="$(sensors 2>/dev/null | awk '/^temp1:/{gsub("\\+|°C",""); print $2; exit} END{if(NR==0) print "NA"}')"
fi

# 1-minute load average
LOAD1="$(awk '{print $1}' /proc/loadavg)"

# Memory used (MiB)
MEM_USED_MB="$(free -m | awk '/^Mem:/{print $3}')"

# Root filesystem usage (%)
DISK_USE_PCT="$(df -P / | awk 'NR==2{gsub("%","",$5); print $5}')"

# GPU/SoC throttling flags (optional; "NA" if not available)
if command -v vcgencmd >/dev/null 2>&1; then
  THROTTLED="$(vcgencmd get_throttled 2>/dev/null | awk -F= '{print $2}')"
else
  THROTTLED="NA"
fi

# Prepare log directory
sudo mkdir -p "$(dirname "$LOG_FILE")"
if [[ ! -s "$LOG_FILE" ]]; then
  echo "timestamp,cpu_temp_c,load1,mem_used_mb,disk_root_use_pct,throttled" | sudo tee "$LOG_FILE" >/dev/null
fi

echo "$TIMESTAMP,$CPU_TEMP_C,$LOAD1,$MEM_USED_MB,$DISK_USE_PCT,$THROTTLED" > "$TMPFILE"
sudo tee -a "$LOG_FILE" < "$TMPFILE" >/dev/null
rm -f "$TMPFILE"

Make it executable:

sudo install -m 0755 pi_telemetry.sh /usr/local/bin/

Let it run once to create the CSV:

/usr/local/bin/pi_telemetry.sh && tail -n3 /var/log/pi_metrics.csv

3) Add a tiny on-device anomaly detector (Isolation Forest)

This script learns a baseline from recent rows and classifies the newest row as normal or anomalous. It’s small, fast, and works well for mixed metrics.

Create /usr/local/bin/anomaly_iforest.py:

#!/usr/bin/env python3
import csv, sys, os, json
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np

path = sys.argv[1] if len(sys.argv) > 1 else "/var/log/pi_metrics.csv"
window = int(os.environ.get("PI_IFOREST_WINDOW", "500"))     # training window
contam = float(os.environ.get("PI_IFOREST_CONTAM", "0.02"))  # expected anomaly ratio

rows = []
with open(path, newline="") as f:
    reader = csv.DictReader(f)
    for r in reader:
        try:
            # Use numeric features only
            feats = [
                float(r["cpu_temp_c"]) if r["cpu_temp_c"] != "NA" else np.nan,
                float(r["load1"]),
                float(r["mem_used_mb"]),
                float(r["disk_root_use_pct"]),
            ]
            # Replace NaN with column means later
            rows.append((r["timestamp"], feats))
        except Exception:
            continue

if len(rows) < 30:
    print(json.dumps({"status": "insufficient_data", "count": len(rows)}))
    sys.exit(0)

# Take last N rows
rows = rows[-window:] if len(rows) > window else rows

timestamps = [r[0] for r in rows]
X = np.array([r[1] for r in rows], dtype=float)

# Simple NaN imputation with column means
col_means = np.nanmean(X, axis=0)
inds = np.where(np.isnan(X))
X[inds] = np.take(col_means, inds[1])

# Train on all but the last row; score the last row
if X.shape[0] < 10:
    print(json.dumps({"status": "insufficient_data", "count": X.shape[0]}))
    sys.exit(0)

X_train = X[:-1]
x_test = X[-1].reshape(1, -1)
ts_test = timestamps[-1]

model = make_pipeline(
    StandardScaler(),
    IsolationForest(
        n_estimators=128,
        contamination=contam,
        random_state=42,
        n_jobs=1,
        warm_start=False
    )
)
model.fit(X_train)
pred = model.predict(x_test)[0]  # 1 normal, -1 anomaly
score = model.score_samples(x_test)[0]  # lower = more anomalous

out = {
    "status": "ok",
    "timestamp": ts_test,
    "anomaly": bool(pred == -1),
    "iforest_score": float(score),
    "features": {
        "cpu_temp_c": float(X[-1,0]),
        "load1": float(X[-1,1]),
        "mem_used_mb": float(X[-1,2]),
        "disk_root_use_pct": float(X[-1,3]),
    }
}
print(json.dumps(out))

Make it executable:

sudo install -m 0755 anomaly_iforest.py /usr/local/bin/

Try it:

/usr/local/bin/anomaly_iforest.py /var/log/pi_metrics.csv | jq .

Tuning tips:

  • PI_IFOREST_WINDOW controls how much recent history defines “normal” (e.g., 200–2000 rows).

  • PI_IFOREST_CONTAM sets sensitivity (0.01–0.05 are common; higher = more sensitive).

4) Automate and alert on anomalies

Create /usr/local/bin/monitor.sh to collect, score, and alert:

#!/usr/bin/env bash
set -euo pipefail

LOG_JSON="/tmp/pi_anomaly.json"
WEBHOOK_URL="${WEBHOOK_URL:-}"   # set this to your Slack/Discord/Mattermost webhook
WINDOW="${PI_IFOREST_WINDOW:-500}"
CONTAM="${PI_IFOREST_CONTAM:-0.02}"

# 1) Collect a new row
/usr/local/bin/pi_telemetry.sh

# 2) Score the latest row
PI_IFOREST_WINDOW="$WINDOW" PI_IFOREST_CONTAM="$CONTAM" \
  /usr/local/bin/anomaly_iforest.py /var/log/pi_metrics.csv > "$LOG_JSON" || {
    logger -t pi-monitor "anomaly script failed"
    exit 1
  }

ANOMALY=$(jq -r '.anomaly // empty' "$LOG_JSON" || true)
STATUS=$(jq -r '.status // empty' "$LOG_JSON" || true)

if [[ "$STATUS" != "ok" ]]; then
  logger -t pi-monitor "Status=$STATUS (waiting for more data)"
  exit 0
fi

if [[ "$ANOMALY" == "true" ]]; then
  TEXT="$(jq -r '
    "Raspberry Pi anomaly @ " + .timestamp
    + "\n  temp: " + (.features.cpu_temp_c|tostring) + " C"
    + "\n  load1: " + (.features.load1|tostring)
    + "\n  mem_used_mb: " + (.features.mem_used_mb|tostring)
    + "\n  disk%: " + (.features.disk_root_use_pct|tostring)
    + "\n  iforest_score: " + (.iforest_score|tostring)
  ' "$LOG_JSON")"

  logger -t pi-monitor "$TEXT"

  if [[ -n "${WEBHOOK_URL}" ]]; then
    PAYLOAD=$(jq -n --arg t "$TEXT" '{text:$t}')
    curl -sS -X POST -H "Content-type: application/json" \
      --data "$PAYLOAD" "$WEBHOOK_URL" >/dev/null || true
  fi
fi

Install and schedule it every minute with cron:

sudo install -m 0755 monitor.sh /usr/local/bin/
sudo crontab -e
# Add this line (runs every minute):
*/1 * * * * WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ" /usr/local/bin/monitor.sh

No webhook? The script still logs anomalies to syslog:

sudo journalctl -t pi-monitor -f

5) Real-world checks and examples

  • Overheating and throttling

    • Reboot, let the model learn a baseline for 30+ minutes.
    • Stress the CPU:
    stress-ng --cpu 4 --timeout 120
    • Expect a spike in temperature and load; the model should alert on the deviation.
    • If vcgencmd is available, you’ll see throttling flags change from 0x0 when throttling occurs.

  • Runaway process or memory leak

    • Start a memory hog or runaway Python script to grow memory usage; the detector should trigger as mem_used_mb drifts.

  • Disk filling fast

    • Copy a large file to / to force disk usage up quickly; anomalies fire because disk_root_use_pct deviates from baseline.

  • Nightly cron storms

    • If 3 AM backups briefly heat and load the Pi every day, the model learns that as “normal” for that time window; truly novel spikes still alert.

Why a tiny AI model is valid here

  • It’s contextual: Instead of naïve thresholds, it learns your Pi’s “normal” across temperature, load, memory, and disk. Nightly spikes don’t spam you—genuine deviations do.

  • It’s resource-light: Isolation Forest with a few hundred samples runs in milliseconds and uses little RAM/CPU.

  • It’s transparent: You can inspect inputs, outputs, and tune sensitivity without black boxes.

Next steps (CTA)

  • Add more signals: network errors, service restarts, journal anomalies, SD card wear indicators (e.g., mmc errors from dmesg).

  • Persist models between runs for faster scoring, or maintain day/night baselines.

  • Ship metrics to InfluxDB/Prometheus and use this anomaly flag as an alert channel.

  • Harden it: convert cron to a systemd timer, add log rotation for /var/log/pi_metrics.csv, and unit tests for your scripts.

If this helped, wire it into your Pi today, trigger a safe test anomaly with stress-ng, and tune PI_IFOREST_CONTAM until alerts feel “just right.” Your future self (and your SD card) will thank you.

Related posts