Artificial Intelligence for Site Reliability Engineers (SREs): A Bash‑First Playbook

Incidents always seem to land at 3 a.m., alerts multiply like rabbits, and capacity questions show up right after a product launch. What if you could add a thin layer of AI—invoked from good old Bash—to catch anomalies earlier, deduplicate noisy alerts, and forecast capacity before things melt?

This article shows you practical, low-friction ways to add AI to your Linux SRE toolkit without overhauling your stack. We’ll wire up a few small scripts, lean on Python libraries where it makes sense, and orchestrate everything with systemd timers. You’ll end up with:

• A 1‑minute metrics collector (CPU load, memory).

• An anomaly watchdog to flag weird system behavior.

• An alert de‑duplicator that groups similar errors.

• A daily capacity forecaster so you can stay in front of growth.

All of this remains Bash-first, transparent, and easy to tear back out if it doesn’t help.

Why AI for SRE Is Worth Your Time

• It’s not “boil the ocean.” Lightweight models like Isolation Forest (for anomalies) and TF‑IDF + clustering (for alert grouping) are proven, low-risk, and easy to run locally.

• You already have the data. Linux exposes rich signals via /proc, journald, and sysstat. Feed those into simple models and you get meaningful wins without a data engineering project.

• It scales your attention. Let machines sift noise and highlight the outliers. You stay focused on the few events that matter.

What You’ll Build

• Metrics collector: writes timestamp, 1‑minute load, and memory usage to CSV every minute.

• Anomaly watchdog: learns “normal” from recent data and flags outliers.

• Alert de‑duplicator: groups similar error logs so on-call sees one item instead of twenty.

• Capacity forecaster: estimates tomorrow’s load and warns if you’ll cross safe utilization.

Everything is controlled via systemd timers and lives under /opt/sre-ai (code) and /var/lib/sre-ai (data).

---

Prerequisites and Install

We’ll use Python for the small AI parts, and Bash/systemd for glue.

Install OS packages:

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip jq sysstat

• Fedora/RHEL (dnf):

sudo dnf install -y python3 python3-pip python3-virtualenv jq sysstat

• openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-pip python3-virtualenv jq sysstat

Create directories and a Python virtual environment for the AI bits:

sudo mkdir -p /opt/sre-ai /var/lib/sre-ai /var/log/sre-ai
sudo chown -R "$USER":"$USER" /opt/sre-ai /var/lib/sre-ai /var/log/sre-ai

python3 -m venv /opt/sre-ai/venv
source /opt/sre-ai/venv/bin/activate
pip install --upgrade pip
pip install numpy pandas scikit-learn statsmodels
deactivate

Note:

• If python3 -m venv isn’t available, install the “venv/virtualenv” packages shown above per distro.

• sysstat gives you sar/iostat; we’ll use /proc directly for the minimal collector, but sysstat is handy for deeper troubleshooting.

---

1) Collect Metrics Every Minute (Bash + systemd)

Write a minimal collector that appends one CSV line per minute:

/opt/sre-ai/collect_metrics.sh

#!/usr/bin/env bash
set -euo pipefail

DIR=/var/lib/sre-ai
FILE="$DIR/metrics.csv"
mkdir -p "$DIR"

if [ ! -s "$FILE" ]; then
  echo "ts,load1,mem_used_pct" > "$FILE"
fi

ts=$(date -Iseconds)
load1=$(cut -d' ' -f1 /proc/loadavg)
mem_total_kb=$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)
mem_avail_kb=$(awk '/^MemAvailable:/ {print $2}' /proc/meminfo)
mem_used_pct=$(awk -v t="$mem_total_kb" -v a="$mem_avail_kb" 'BEGIN{printf "%.2f", (t-a)/t*100}')

echo "$ts,$load1,$mem_used_pct" >> "$FILE"

Make it executable:

chmod +x /opt/sre-ai/collect_metrics.sh

Create a systemd service and timer:

/etc/systemd/system/sre-metrics.service

[Unit]
Description=SRE AI - Collect metrics

[Service]
Type=oneshot
ExecStart=/opt/sre-ai/collect_metrics.sh

/etc/systemd/system/sre-metrics.timer

[Unit]
Description=Run SRE metrics collector every minute

[Timer]
OnCalendar=*:0/1
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now sre-metrics.timer
systemctl status sre-metrics.timer --no-pager

You should see /var/lib/sre-ai/metrics.csv grow each minute.

---

2) Detect Anomalies with Isolation Forest

When recent behavior is “learned,” outliers jump out. This watchdog reads recent metrics and flags unusual spikes.

/opt/sre-ai/detect_anomalies.py

#!/usr/bin/env python3
import sys, json, pathlib
import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest

DATA = pathlib.Path("/var/lib/sre-ai/metrics.csv")

def main():
    if not DATA.exists():
        print("no metrics yet", file=sys.stderr)
        return 0

    df = pd.read_csv(DATA, parse_dates=["ts"])
    if len(df) < 200:
        print("insufficient data for anomaly detection", file=sys.stderr)
        return 0

    df = df.tail(2880)  # ~2 days if 1-min sampling
    X = df[["load1", "mem_used_pct"]].astype(float).values

    model = IsolationForest(n_estimators=200, contamination=0.01, random_state=42)
    model.fit(X)

    latest = X[-1].reshape(1, -1)
    pred = model.predict(latest)[0]   # 1 normal, -1 anomaly
    score = float(model.decision_function(latest)[0])

    result = {
        "ts": df["ts"].iloc[-1].isoformat(),
        "load1": float(df["load1"].iloc[-1]),
        "mem_used_pct": float(df["mem_used_pct"].iloc[-1]),
        "is_anomaly": pred == -1,
        "anomaly_score": score
    }
    print(json.dumps(result))

    # Exit code 2 on anomaly so systemd/journal can highlight it
    return 2 if pred == -1 else 0

if __name__ == "__main__":
    sys.exit(main())

Make it executable and a thin wrapper to log anomalies:

/opt/sre-ai/run_anomaly_watchdog.sh

#!/usr/bin/env bash
set -euo pipefail
source /opt/sre-ai/venv/bin/activate
OUT=$(/opt/sre-ai/detect_anomalies.py || true)
echo "$OUT"
if echo "$OUT" | jq -e '.is_anomaly == true' >/dev/null 2>&1; then
  logger -t sre-ai "ANOMALY: $(echo "$OUT" | jq -c '.')"
  # Optional: trigger webhooks/alerts here
  exit 2
fi

chmod +x /opt/sre-ai/detect_anomalies.py /opt/sre-ai/run_anomaly_watchdog.sh

Systemd unit and timer:

/etc/systemd/system/sre-anomaly.service

[Unit]
Description=SRE AI - Anomaly watchdog

[Service]
Type=oneshot
ExecStart=/opt/sre-ai/run_anomaly_watchdog.sh

/etc/systemd/system/sre-anomaly.timer

[Unit]
Description=Run anomaly watchdog every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now sre-anomaly.timer
journalctl -u sre-anomaly.service -n 5 --no-pager

You’ll get JSON in logs and a syslog line when an anomaly is detected.

---

3) De‑duplicate Noisy Alerts with TF‑IDF + Clustering

Group similar errors so one incident doesn’t fire a dozen near-duplicates. We’ll cluster recent high-priority logs from journald.

/opt/sre-ai/dedupe_alerts.py

#!/usr/bin/env python3
import sys, json
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import DBSCAN

lines = [ln.strip() for ln in sys.stdin if ln.strip()]
if not lines:
    sys.exit(0)

# Vectorize log messages
vec = TfidfVectorizer(stop_words="english", ngram_range=(1,2), max_features=4000)
X = vec.fit_transform(lines)

# Cluster by cosine similarity
clu = DBSCAN(eps=0.5, min_samples=2, metric="cosine").fit(X)
labels = clu.labels_

clusters = {}
for i, lbl in enumerate(labels):
    clusters.setdefault(int(lbl), []).append(lines[i])

summary = []
for lbl, msgs in clusters.items():
    if lbl == -1:  # noise
        continue
    # Pick a representative message (shortest)
    rep = min(msgs, key=len)
    summary.append({"cluster": lbl, "count": len(msgs), "example": rep})

print(json.dumps({"groups": summary, "total": len(lines)}, indent=2))

Wrapper to read last 5 minutes of errors, cluster, and write a report:

/opt/sre-ai/run_dedupe.sh

#!/usr/bin/env bash
set -euo pipefail
source /opt/sre-ai/venv/bin/activate

OUT=$(
  journalctl -p err..alert -S -5m -o short-iso -q \
  | sed 's/^[^]]*] *//' \
  | /opt/sre-ai/dedupe_alerts.py
)

echo "$OUT" | tee /var/log/sre-ai/alert_groups.json
logger -t sre-ai "DEDUPE: $(echo "$OUT" | jq -c '{groups: [.groups[] | {count, example}], total}')"

chmod +x /opt/sre-ai/dedupe_alerts.py /opt/sre-ai/run_dedupe.sh

Systemd unit and timer:

/etc/systemd/system/sre-dedupe.service

[Unit]
Description=SRE AI - Alert de-duplication

[Service]
Type=oneshot
ExecStart=/opt/sre-ai/run_dedupe.sh

/etc/systemd/system/sre-dedupe.timer

[Unit]
Description=Group recent alerts every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now sre-dedupe.timer
tail -n +1 /var/log/sre-ai/alert_groups.json

This creates compact summaries you can forward to Slack/PagerDuty instead of raw floods.

---

4) Forecast Capacity Daily (SARIMA)

Forecast the next 24 hours of load and warn if you’ll cross safe levels (e.g., 80% of CPU capacity).

/opt/sre-ai/forecast_capacity.py

#!/usr/bin/env python3
import sys, json, pathlib, math
import pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX

DATA = pathlib.Path("/var/lib/sre-ai/metrics.csv")

def cpu_count():
    try:
        with open("/proc/cpuinfo") as f:
            return sum(1 for ln in f if ln.startswith("processor"))
    except:
        return 1

def main():
    if not DATA.exists():
        print("no metrics", file=sys.stderr)
        return 0

    df = pd.read_csv(DATA, parse_dates=["ts"])
    if len(df) < 300:
        print("insufficient history", file=sys.stderr)
        return 0

    s = df.set_index("ts")["load1"].astype(float).resample("1H").mean().bfill().astype(float)
    # Simple model; tune orders as needed
    model = SARIMAX(s, order=(1,1,1), seasonal_order=(0,1,1,24), enforce_stationarity=False, enforce_invertibility=False)
    res = model.fit(disp=False)

    steps = 24
    fc = res.get_forecast(steps=steps).predicted_mean
    cap = cpu_count()
    warn_level = 0.8 * cap

    breaches = [(str(ts), float(val)) for ts, val in fc.items() if val >= warn_level]
    out = {
        "cpu_cores": cap,
        "warn_threshold_load": round(warn_level, 2),
        "forecast": [{ "ts": str(ts), "load1": float(val)} for ts, val in fc.items()],
        "breaches": [{"ts": ts, "pred_load": val} for ts, val in breaches]
    }
    print(json.dumps(out, indent=2))

    return 2 if breaches else 0

if __name__ == "__main__":
    sys.exit(main())

Wrapper and systemd timer:

/opt/sre-ai/run_forecast.sh

#!/usr/bin/env bash
set -euo pipefail
source /opt/sre-ai/venv/bin/activate
OUT=$(/opt/sre-ai/forecast_capacity.py || true)
echo "$OUT" | tee /var/log/sre-ai/forecast.json
if echo "$OUT" | jq -e '.breaches | length > 0' >/dev/null 2>&1; then
  logger -t sre-ai "FORECAST-WARN: $(echo "$OUT" | jq -c '{breaches, warn_threshold_load, cpu_cores}')"
  exit 2
fi

chmod +x /opt/sre-ai/forecast_capacity.py /opt/sre-ai/run_forecast.sh

/etc/systemd/system/sre-forecast.service

[Unit]
Description=SRE AI - Daily capacity forecast

[Service]
Type=oneshot
ExecStart=/opt/sre-ai/run_forecast.sh

/etc/systemd/system/sre-forecast.timer

[Unit]
Description=Run capacity forecast daily at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now sre-forecast.timer
journalctl -u sre-forecast.service -n 5 --no-pager

You now get a daily JSON forecast at /var/log/sre-ai/forecast.json and a syslog warning if future load is likely to exceed safe levels.

---

Real‑World Notes and Tips

• Start simple. Even basic features (load1, mem_used_pct) catch many issues. You can add disk IO, network, and app metrics later.

• Keep state close. CSV in /var/lib/sre-ai is easy to inspect, back up, and rotate.

• Tune thresholds. Isolation Forest contamination=0.01 is a conservative start. Raise it if you never see anomalies; lower if too chatty.

• Pipe to your tools. Replace “logger” lines with curl to your alert manager of choice.

• Governance. Keep a changelog of models and parameters. It helps explain why a particular alert fired.

---

Conclusion and Next Steps (CTA)

You just added practical AI helpers to your SRE workflow—without new servers, SaaS lock‑in, or opaque magic. From here:

• Wire outputs to your incident channel (Slack/PagerDuty) and weekly ops review.

• Add features: disk saturation, application error rate, p99 latency.

• Put the dedupe summary into your alert pipeline to trim noisy pages.

• Iterate model params monthly based on feedback.

If you want a deeper dive (e.g., Prometheus scraping + PromQL + model training on richer signals), or to integrate with Kubernetes node metrics, say the word—I can extend this playbook with distro‑specific install steps and manifests.