AI‑Assisted RAID Monitoring on Linux (with Bash)

Your RAID array is “healthy”… until it isn’t. SMART says “PASSED”… until a drive starts accumulating pending sectors and your rebuild window becomes a nail‑biter. Traditional threshold alarms can miss weak signals. Let’s use a tiny dose of AI to learn what “normal” looks like on your disks and flag drift before your array degrades.

This post shows how to glue together Bash, smartctl, mdadm, and a lightweight anomaly detection model to surface early‑warning signals. You’ll get install commands, collection scripts, a minimal Python model, and automation via cron.

Why AI for RAID monitoring?

• Disks age differently. Thresholds that work for one batch fail for another. Anomaly detection adapts to your hardware’s baseline.

• SMART “PASSED” can be misleading. Subtle metric drift often precedes failure (e.g., growing pending sectors, rising media errors, increasing temperature).

• RAID masks pain until it breaks. Catching weak signals lets you replace a drive during business hours instead of during a rebuild.

• It’s cheap and local. No cloud dependency; just Bash, smartctl/mdadm, and a small Python model.

What we’ll build: 1) A Bash collector that snapshots SMART and RAID state. 2) A Python IsolationForest model that learns your baseline and flags anomalies. 3) Cron jobs to collect data and alert daily.

---

1) Install prerequisites

Run these as root (or with sudo). Choose the commands for your distro.

• Debian/Ubuntu (apt):

apt update
apt install -y smartmontools mdadm nvme-cli jq python3 python3-venv python3-pip mailutils cron
systemctl enable --now cron

• RHEL/CentOS/Fedora (dnf):

dnf install -y smartmontools mdadm nvme-cli jq python3 python3-pip python3-virtualenv mailx cronie
systemctl enable --now crond

• openSUSE/SLES (zypper):

zypper refresh
zypper install -y smartmontools mdadm nvme-cli jq python3 python3-pip python3-virtualenv s-nail cron
systemctl enable --now cron

Set up a Python virtual environment for the small AI model:

mkdir -p /opt/raid-ai
python3 -m venv /opt/raid-ai/venv
/opt/raid-ai/venv/bin/pip install --upgrade pip
/opt/raid-ai/venv/bin/pip install numpy pandas scikit-learn

---

2) Collect telemetry: SMART + RAID

This script:

• Enumerates disks via lsblk.

• Captures detailed SMART JSON per device (SATA/SAS/NVMe via smartctl).

• Captures RAID status (/proc/mdstat + mdadm --detail).

• Appends a JSON line to /var/log/raid-ai/metrics.jsonl

Save as /usr/local/sbin/raid_ai_collect.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

LOG_DIR=/var/log/raid-ai
LOG_FILE="$LOG_DIR/metrics.jsonl"
mkdir -p "$LOG_DIR"
chmod 750 "$LOG_DIR"

ts=$(date -Iseconds)
host=$(hostname -f 2>/dev/null || hostname)

# Capture RAID status snapshots
mdstat=$(cat /proc/mdstat || true)
md_details_json="[]"
for a in /dev/md*; do
  [ -e "$a" ] || continue
  det=$(mdadm --detail "$a" 2>/dev/null || true)
  md_details_json=$(jq --arg arr "$a" --arg det "$det" '. + [{array: $arr, detail: $det}]' <<<"$md_details_json")
done

# Enumerate block devices
mapfile -t devs < <(lsblk -dn -o NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}')

for dev in "${devs[@]}"; do
  smart_json=$(smartctl -x -j "$dev" 2>/dev/null || true)
  [ -n "$smart_json" ] || continue
  jq -n \
    --arg ts "$ts" \
    --arg host "$host" \
    --arg dev "$dev" \
    --arg mdstat "$mdstat" \
    --argjson mdadm "$md_details_json" \
    --argjson smart "$smart_json" \
    '{ts:$ts, host:$host, dev:$dev, mdstat:$mdstat, mdadm:$mdadm, smart:$smart}' \
    >> "$LOG_FILE"
done

Permissions:

chmod +x /usr/local/sbin/raid_ai_collect.sh

Notes:

• smartctl -x -j works across ATA/NVMe, returning structured JSON we can parse later.

• This script is read‑only, but SMART may spin up idle disks. Schedule sensibly.

---

3) A minimal anomaly detector

We’ll use IsolationForest (unsupervised) to learn your disks’ baseline and score the latest snapshot per device. It combines features like temperature, power‑on hours, and key SMART attributes (reallocated, pending, uncorrectable sectors; NVMe media errors; percentage used), plus a simple RAID degraded indicator.

Save as /usr/local/sbin/raid_ai_score.py:

#!/usr/bin/env python3
import json, sys, os, math, datetime
from collections import defaultdict

import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest

LOGFILE = "/var/log/raid-ai/metrics.jsonl"

def parse_time(s):
    try:
        return datetime.datetime.fromisoformat(s.replace("Z", "+00:00"))
    except Exception:
        return None

def ata_attr(smart, attr_id, default=0.0):
    try:
        for a in smart.get("ata_smart_attributes", {}).get("table", []):
            if a.get("id") == attr_id:
                return float(a.get("raw", {}).get("value", default))
    except Exception:
        pass
    return float(default)

def pick_temp_c(smart):
    # smartctl tries to normalize to Celsius under 'temperature.current'
    t = None
    try:
        t = smart.get("temperature", {}).get("current", None)
        if t is not None:
            return float(t)
    except Exception:
        pass
    # NVMe block (fallbacks)
    try:
        t = smart.get("nvme_smart_health_information_log", {}).get("temperature", None)
        if t is not None:
            return float(t)
    except Exception:
        pass
    return np.nan

def md_degraded(rec):
    degraded = False
    for s in [rec.get("mdstat","")] + [d.get("detail","") for d in rec.get("mdadm", [])]:
        s_low = (s or "").lower()
        if "degraded" in s_low or "[u_" in s_low or "[_u" in s_low:
            degraded = True
    return 1.0 if degraded else 0.0

def extract_features(rec):
    smart = rec.get("smart", {}) or {}
    feats = {}
    feats["temp_c"] = pick_temp_c(smart)
    feats["po_hours"] = float(smart.get("power_on_time", {}).get("hours", 0.0) or 0.0)
    feats["smart_pass"] = 1.0 if smart.get("smart_status", {}).get("passed", True) else 0.0

    # ATA attributes
    feats["realloc"] = ata_attr(smart, 5, 0.0)
    feats["pending"] = ata_attr(smart, 197, 0.0)
    feats["uncorr"]  = ata_attr(smart, 198, 0.0)

    # NVMe attributes
    nvme = smart.get("nvme_smart_health_information_log", {}) or {}
    feats["nvme_media_err"] = float(nvme.get("media_errors", 0.0) or 0.0)
    feats["nvme_pct_used"]  = float(nvme.get("percentage_used", 0.0) or 0.0)

    feats["raid_degraded"] = md_degraded(rec)
    return feats

def load_records(path):
    rows = []
    if not os.path.exists(path):
        return rows
    with open(path, "r", encoding="utf-8", errors="ignore") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
                rec["ts_dt"] = parse_time(rec.get("ts",""))
                rows.append(rec)
            except Exception:
                continue
    return rows

def main():
    recs = load_records(LOGFILE)
    if not recs:
        print("No data yet. Run the collector and wait for a few snapshots.")
        return

    # Build feature frame
    data = []
    for r in recs:
        feats = extract_features(r)
        feats["dev"] = r.get("dev", "unknown")
        feats["ts"]  = r.get("ts", "")
        feats["ts_dt"] = r.get("ts_dt", None)
        data.append(feats)
    df = pd.DataFrame(data)
    df = df.replace([np.inf, -np.inf], np.nan)

    # Keep rows with at least one signal
    feature_cols = ["temp_c","po_hours","smart_pass","realloc","pending","uncorr","nvme_media_err","nvme_pct_used","raid_degraded"]
    if df[feature_cols].dropna(how="all").empty:
        print("Not enough usable SMART/RAID features found. Check smartctl output and permissions.")
        return

    # Impute NaNs to 0 for robust tree-based model; log1p for heavy-tailed counts
    for c in ["realloc","pending","uncorr","nvme_media_err","po_hours"]:
        df[c] = np.log1p(df[c].fillna(0.0))
    for c in ["temp_c","smart_pass","nvme_pct_used","raid_degraded"]:
        df[c] = df[c].fillna(0.0)

    # Train on historical data (unsupervised)
    hist = df.dropna(subset=feature_cols)
    if len(hist) < 20:
        print("Collected {} samples; IsolationForest needs more history. Let it run for ~1 day.".format(len(hist)))
        return

    X = hist[feature_cols].values
    iforest = IsolationForest(
        n_estimators=200,
        contamination=0.05,      # Top 5% anomalies flagged
        random_state=42
    )
    iforest.fit(X)

    # Latest per device
    latest_idx = hist.groupby("dev")["ts_dt"].idxmax()
    latest = hist.loc[latest_idx].copy()
    X_latest = latest[feature_cols].values
    pred = iforest.predict(X_latest)          # 1 normal, -1 anomaly
    score = -iforest.score_samples(X_latest)  # Higher = more anomalous

    latest["anomaly_score"] = score
    latest["label"] = np.where(pred==-1, "ALERT", "OK")

    # Pretty print
    lines = []
    for _, row in latest.sort_values(by="anomaly_score", ascending=False).iterrows():
        msg = (
            f"{row['label']}: {row['dev']} score={row['anomaly_score']:.3f} "
            f"(temp_c={row['temp_c']:.1f}, po_h=exp({row['po_hours']:.2f})-1, "
            f"realloc=exp({row['realloc']:.2f})-1, pending=exp({row['pending']:.2f})-1, "
            f"uncorr=exp({row['uncorr']:.2f})-1, nvme_media=exp({row['nvme_media_err']:.2f})-1, "
            f"nvme_pct_used={row['nvme_pct_used']:.1f}, raid_degraded={int(row['raid_degraded'])})"
        )
        lines.append(msg)

    print("\n".join(lines))

if __name__ == "__main__":
    main()

Make it executable:

chmod +x /usr/local/sbin/raid_ai_score.py

What the model does:

• Learns your disks’ “normal” feature distribution over time.

• Scores the newest snapshot per device.

• Flags the top ~5% most unusual devices as ALERT.

Tip: Let it collect at least a day of data before you rely on alerts.

---

4) Automate and alert

Create a tiny wrapper that runs the model and emails root if anything is ALERT.

Save as /usr/local/sbin/raid_ai_score.sh:

#!/usr/bin/env bash
set -euo pipefail

VENV=/opt/raid-ai/venv
PY=$VENV/bin/python
SCRIPT=/usr/local/sbin/raid_ai_score.py
LOG=/var/log/raid-ai/alerts.log

out="$($PY "$SCRIPT" 2>&1 || true)"
ts=$(date -Iseconds)
if [ -n "$out" ]; then
  echo "[$ts]" >> "$LOG"
  echo "$out" | tee -a "$LOG"
  if echo "$out" | grep -q "ALERT"; then
    subject="RAID-AI alerts on $(hostname)"
    if command -v mailx >/dev/null 2>&1; then
      echo "$out" | mailx -s "$subject" root
    elif command -v mail >/dev/null 2>&1; then
      echo "$out" | mail -s "$subject" root
    else
      logger -p daemon.warn "$subject: $out"
    fi
  fi
fi

Permissions:

chmod +x /usr/local/sbin/raid_ai_score.sh

Cron scheduling:

tee /etc/cron.d/raid-ai >/dev/null <<'EOF'
# Collect SMART/RAID metrics every 15 minutes
*/15 * * * * root /usr/local/sbin/raid_ai_collect.sh
# Score anomalies daily at 03:17
17 3 * * * root /usr/local/sbin/raid_ai_score.sh
EOF

Ensure cron is active (see install section). Logs:

• Raw metrics: /var/log/raid-ai/metrics.jsonl

• Alerts: /var/log/raid-ai/alerts.log

• Email to root if configured.

---

5) Validate and tune

• Dry run the collector:

/usr/local/sbin/raid_ai_collect.sh && tail -1 /var/log/raid-ai/metrics.jsonl | jq .

• After 24h, test the scorer:

/usr/local/sbin/raid_ai_score.sh

• Tune sensitivity:

  • Increase/decrease contamination in raid_ai_score.py (e.g., 0.02 for stricter alerts).
  • Add/remove features as your fleet dictates (e.g., add UDMA CRC errors, wear-leveling counts).

• Integrate with your stack:

  • Replace mail with your notifier (Slack, webhook, systemd-notify, Prometheus pushgateway).
  • Ship metrics.jsonl to your TSDB and chart top features per device.

---

Real‑world examples

• Rising pending sectors: A drive shows a slow, steady rise in attribute 197 (Current_Pending_Sector) with otherwise “PASSED” SMART. The model flags it as anomalous 2 days before the array reports degraded, letting you replace the disk during business hours.

• NVMe wear surge: percentage_used climbs unusually fast for one NVMe compared to its peers. The anomaly score spikes, prompting workload rebalancing and avoiding an abrupt capacity loss.

• Silent heat issue: A server with a partly obstructed intake runs 7–10°C hotter than its siblings. The model flags temperature drift before thermal throttling impacts performance.

---

Conclusion and next steps

AI doesn’t replace good ops hygiene or backups—but it’s a sharp tool for early detection. With a few Bash lines and a small IsolationForest, you can teach your monitoring to “listen” to your disks and RAID arrays.

Your next steps: 1) Install the prerequisites and deploy the collector today. 2) Let it run for a week to learn your baseline. 3) Turn on daily scoring and route ALERTs into your normal paging path. 4) Iterate: add features that matter to your environment and tune sensitivity.

If you found this useful, try extending it with Prometheus/Grafana panels or a systemd timer, and share your improvements back to the team. Your future 2 a.m. self will thank you.