Posted on
• Artificial Intelligence

Artificial Intelligence SAN Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence SAN Monitoring on Linux: Detect Problems Before They Become Outages

If your business depends on shared storage, a small SAN hiccup can snowball into a full-blown outage. Traditional threshold-based alerts often miss subtle signals: rising latency at 2 a.m., a queue depth creeping up during backups, or a flapping path that “self-heals” until it doesn’t. This post shows how to bring lightweight AI to your Linux/Bash toolkit to spot storage anomalies early—using tools you already trust.

You’ll learn why AI is a good fit for SAN telemetry and how to build a practical, vendor-agnostic monitoring pipeline on any Linux host that uses your SAN, complete with install commands for apt, dnf, and zypper.

Why AI for SAN monitoring?

  • SANs are complex and noisy: latency, throughput, IOPS, queue depth, path health, and workload mix all move simultaneously.

  • Fixed thresholds are brittle: what’s “normal” at noon on Monday is not normal at midnight Sunday.

  • AI can baseline your environment and flag deviations automatically, reducing false positives and surfacing real issues sooner.

  • It’s vendor-agnostic: you can start with host-side signals (iostat, multipath status) without proprietary array APIs.

What we’ll build

A minimal, production-friendly flow:

  • A Bash collector that logs per-device extended IO metrics and multipath health every minute.

  • A small Python script using Isolation Forest (unsupervised anomaly detection) to flag unusual patterns.

  • Systemd units/timers to run continuously and alert via logs.

  • Distro-agnostic installation steps.

Note: Commands that query devices or multipath typically need root.

Prerequisites: Install the tools

We’ll use:

  • sysstat (iostat) for extended device stats

  • multipath-tools (or device-mapper-multipath) for path health

  • jq for robust JSON parsing from iostat

  • Python 3 + pip (and venv) for the anomaly detector

On Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat multipath-tools jq python3 python3-venv python3-pip

On RHEL/Fedora/CentOS/Alma/Rocky (dnf):

sudo dnf install -y sysstat device-mapper-multipath jq python3 python3-pip

On SUSE/openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y sysstat multipath-tools jq python3 python3-pip

Enable multipath if applicable:

sudo systemctl enable --now multipathd

Tip (Debian/Ubuntu): iostat works immediately after install. If you also plan to use sar, enable “ENABLED=true” in /etc/default/sysstat and restart the service.

Step 1: Collect SAN metrics with Bash + iostat JSON

We’ll run iostat in JSON mode for parsing stability and include multipath path-failure counts in each record.

Create /usr/local/sbin/collect_san_metrics.sh:

#!/usr/bin/env bash
set -euo pipefail

OUT_DIR="/var/log/san-metrics"
FILE="$OUT_DIR/metrics.csv"

mkdir -p "$OUT_DIR"
chmod 750 "$OUT_DIR"

# Initialize CSV header if missing
if [ ! -s "$FILE" ]; then
  echo "ts,device,r_s,w_s,rkB_s,wkB_s,await,aqu_sz,util,mp_failed" > "$FILE"
fi

# Continuous collector: one 60s interval per loop
while true; do
  TS="$(date -Is)"
  # Count any lines hinting a failed/faulty/offline path (best-effort)
  MP_FAILED="$(multipath -ll 2>/dev/null | awk 'BEGIN{IGNORECASE=1} /failed|faulty|offline/ {c++} END{print c+0}')"

  # Capture one 60s extended interval as JSON, then parse
  if iostat -x -y -o JSON 60 1 > /tmp/iostat.json 2>/dev/null; then
    jq -r --arg ts "$TS" --arg mp "$MP_FAILED" '
      .sysstat.hosts[0].statistics[-1].disk[]
      | select(.disk_device|test("^(sd|nvme|dm-)"))
      | [
          $ts,
          .disk_device,
          (."r/s"//0),
          (."w/s"//0),
          (."rkB/s"//0),
          (."wkB/s"//0),
          (.await//0),
          (."aqu-sz"//0),
          (."%util"//0),
          $mp
        ]
      | @csv
    ' /tmp/iostat.json >> "$FILE"
  else
    # Fallback note if JSON not supported (sysstat < 12)
    echo "iostat JSON not available; please upgrade sysstat >= 12" | systemd-cat -t san-metrics || true
    sleep 60
  fi
done

Make it executable:

sudo install -m 0750 /usr/local/sbin/collect_san_metrics.sh /usr/local/sbin/collect_san_metrics.sh

Create a systemd service to run it continuously:

sudo tee /etc/systemd/system/san-metrics.service >/dev/null <<'UNIT'
[Unit]
Description=SAN metrics collector (iostat + multipath)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/collect_san_metrics.sh
User=root
Restart=always
RestartSec=5s
Nice=10
IOSchedulingClass=idle

[Install]
WantedBy=multi-user.target
UNIT

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now san-metrics.service
sudo journalctl -u san-metrics.service -f

You should see /var/log/san-metrics/metrics.csv growing with lines like:

2026-07-11T10:12:00+00:00,dm-2,120.3,45.1,8192.0,6144.0,5.2,0.8,73.5,0

Step 2: Add a lightweight AI anomaly detector (Isolation Forest)

Create a Python virtual environment and install dependencies.

On Debian/Ubuntu (apt used above for python3-venv):

sudo python3 -m venv /opt/san-ai/venv
sudo /opt/san-ai/venv/bin/pip install --upgrade pip
sudo /opt/san-ai/venv/bin/pip install pandas numpy scikit-learn

On RHEL/Fedora/CentOS/Alma/Rocky (dnf):

sudo python3 -m venv /opt/san-ai/venv
sudo /opt/san-ai/venv/bin/pip install --upgrade pip
sudo /opt/san-ai/venv/bin/pip install pandas numpy scikit-learn

On SUSE/openSUSE (zypper):

sudo python3 -m venv /opt/san-ai/venv
sudo /opt/san-ai/venv/bin/pip install --upgrade pip
sudo /opt/san-ai/venv/bin/pip install pandas numpy scikit-learn

Create /usr/local/sbin/san_detect.py:

#!/usr/bin/env /opt/san-ai/venv/bin/python
import os
import sys
import time
import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest

METRICS = "/var/log/san-metrics/metrics.csv"
MIN_SAMPLES = 200         # minimum rows per device to build a baseline
CONTAMINATION = 0.02      # expected fraction of anomalies
LOOKBACK_MINUTES = 1440   # train on ~1 day by default
TEST_WINDOW = 5           # number of most recent points to score

def log(msg):
    print(f"[san-ai] {time.strftime('%Y-%m-%d %H:%M:%S')} {msg}")

def load_data(path):
    if not os.path.exists(path):
        log(f"missing metrics file: {path}")
        sys.exit(0)
    # Read efficiently: we can tail a lot of rows if file is large
    df = pd.read_csv(path,
                     names=["ts","device","r_s","w_s","rkB_s","wkB_s","await","aqu_sz","util","mp_failed"],
                     header=0,
                     parse_dates=["ts"])
    # Keep only recent data
    cutoff = pd.Timestamp.utcnow() - pd.Timedelta(minutes=LOOKBACK_MINUTES)
    df = df[df["ts"] >= cutoff]
    # Numeric cleanup
    for c in ["r_s","w_s","rkB_s","wkB_s","await","aqu_sz","util","mp_failed"]:
        df[c] = pd.to_numeric(df[c], errors="coerce")
    df = df.replace([np.inf, -np.inf], np.nan).dropna()
    return df

def detect_multipath(df):
    recent = df.sort_values("ts").tail(1)
    if not recent.empty and int(recent["mp_failed"].iloc[0]) > 0:
        log(f"ALERT: multipath shows {int(recent['mp_failed'].iloc[0])} failed/faulty/offline paths")

def detect_per_device(df):
    alerts = 0
    features = ["r_s","w_s","rkB_s","wkB_s","await","aqu_sz","util"]
    for dev, g in df[df["device"]!="__multipath__"].groupby("device"):
        g = g.sort_values("ts")
        if len(g) < MIN_SAMPLES:
            continue
        X = g[features].values
        # Fit on historical, score recent window
        train_X = X[:-TEST_WINDOW] if len(X) > TEST_WINDOW else X
        test_X  = X[-TEST_WINDOW:]
        if len(train_X) < MIN_SAMPLES:
            continue
        model = IsolationForest(
            n_estimators=200,
            contamination=CONTAMINATION,
            random_state=42,
            n_jobs=1,
            warm_start=False
        )
        model.fit(train_X)
        preds = model.predict(test_X)      # -1 == anomaly
        scores = model.decision_function(test_X)
        for i, p in enumerate(preds):
            if p == -1:
                ts = g["ts"].iloc[len(g)-TEST_WINDOW+i]
                row = g.iloc[len(g)-TEST_WINDOW+i]
                alerts += 1
                log(f"ANOMALY device={dev} ts={ts} util={row['util']:.1f}% await={row['await']:.2f} r_s={row['r_s']:.1f} w_s={row['w_s']:.1f} score={scores[i]:.4f}")
    if alerts == 0:
        log("OK: no anomalies in last window")

def main():
    df = load_data(METRICS)
    if df.empty:
        log("no recent data to analyze")
        return
    detect_multipath(df)
    detect_per_device(df)

if __name__ == "__main__":
    main()

Make it executable:

sudo install -m 0750 /usr/local/sbin/san_detect.py /usr/local/sbin/san_detect.py

Create a systemd unit and timer to run it every 15 minutes:

sudo tee /etc/systemd/system/san-ai.service >/dev/null <<'UNIT'
[Unit]
Description=SAN AI anomaly detector

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/san_detect.py
User=root
Nice=5
UNIT

sudo tee /etc/systemd/system/san-ai.timer >/dev/null <<'TIMER'
[Unit]
Description=Run SAN AI anomaly detector every 15 minutes

[Timer]
OnBootSec=2m
OnUnitActiveSec=15m
AccuracySec=1m
Persistent=true

[Install]
WantedBy=timers.target
TIMER

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now san-ai.timer
sudo systemctl list-timers --all | grep san-ai
sudo journalctl -u san-ai.service -f

You’ll see journal lines with anomalies like:

[san-ai] 2026-07-11 10:30:02 ANOMALY device=dm-2 ts=2026-07-11 10:28:00 util=99.7% await=42.13 r_s=15.0 w_s=320.4 score=-0.1312

Step 3: Tune and harden

  • Choose the right features: We used r/s, w/s, rkB/s, wkB/s, await, aqu-sz, util. Add or remove based on your environment.

  • Warm-up/baseline: Set MIN_SAMPLES high enough (e.g., a day’s worth) before trusting alerts. Reduce to accelerate initial learning in labs.

  • Contamination: Lower CONTAMINATION (e.g., 0.01) to reduce alert volume; raise to catch more edge cases.

  • Exclude maintenance windows: You can skip writing samples during known backup windows (e.g., if date +%H is 01–03) to avoid polluting the baseline.

  • Alerting: Pipe san-ai output to your aggregator (journalbeat/rsyslog) or add a simple notifier (e.g., send to a webhook via curl within san_detect.py when an anomaly is found).

Real-world patterns this catches

  • Silent saturation: %util ~100% with rising await across multiple dm-* devices during business hours—often the first sign of underprovisioned IOPS or a hot volume.

  • Path flapping: mp_failed jumps above 0 with intermittent await spikes; usually a cable, HBA, or switch port issue.

  • Noisy neighbor: A backup or misconfigured job drives sustained small random writes; model flags deviations even if absolute throughput doesn’t breach thresholds.

  • Queue buildup: aqu-sz increasing while IOPS stays flat, suggesting backend contention or firmware/controller throttling.

Optional: Quick visualization

Even without a full TSDB, you can peek at trends:

awk -F, '$2=="dm-2"{print $1","$9}' /var/log/san-metrics/metrics.csv | tail -n 200

Feed that into gnuplot or your favorite quick charting tool. For a full dashboard later, consider pushing metrics to Prometheus via node_exporter’s textfile collector and chart in Grafana.

Conclusion and Call to Action

You don’t need a heavy AIOps platform to get real value from AI on your SAN. With a few standard Linux tools and a small Python script, you can baseline normal behavior, flag anomalies early, and investigate issues before users feel them.

Your next steps: 1. Install the collector and AI detector on one SAN-connected host. 2. Let it learn for a day, then begin triaging anomalies in the journal. 3. Tune contamination and features, then roll out to more hosts. 4. Integrate alerts into your existing pipeline (syslog/Slack/email) and add dashboards.

If you found this useful, try extending it with per-volume tags, window-aware models (weekday/weekend), or array vendor APIs for deeper context—while keeping Bash at the heart of your Linux-first workflow.

Related posts