Posted on
• Artificial Intelligence

Artificial Intelligence SRE Automation Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence SRE Automation Projects: 4 Bash‑First Ideas You Can Ship This Week

Woken up at 03:14 by an alert storm that turned out to be a noisy outlier? Drowning in logs after an incident, manually summarizing what went wrong? You’re not alone. Modern SREs are managing sprawling estates, and traditional, rule‑only automation is cracking under scale and variance. The good news: small, focused AI automations can deflect pages, surface signal in noise, and accelerate remediation—without re‑architecting your stack.

This post gives you four practical, Linux/Bash‑first AI automation projects you can pilot quickly. Each one is built from composable CLI and Python tools, can run locally, and comes with copy‑paste snippets.

  • Why this is valid now: open‑source models and libraries are good enough for narrow SRE tasks; Linux pipelines make data wrangling trivial; Python brings the ML; Bash glues it together. You don’t need a huge platform—just a terminal and a plan.

Prerequisites: Set up your toolbox

Install base tools and Python build deps:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-venv python3-pip jq git curl gcc g++ make pkg-config
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-virtualenv python3-pip jq git curl gcc gcc-c++ make pkgconf-pkg-config
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y python3 python3-venv python3-pip jq git curl gcc gcc-c++ make pkg-config

Create a Python virtual environment for ML bits:

python3 -m venv ~/.venvs/sre-ai
source ~/.venvs/sre-ai/bin/activate
pip install -U pip setuptools wheel
pip install scikit-learn pandas numpy statsmodels rapidfuzz joblib pyyaml

Optional (local LLM for summarization): install Ollama

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
ollama pull mistral

Note: If you prefer containers, you can run Ollama or any OpenAI‑compatible API behind a reverse proxy and keep the Bash snippets identical (just change the URL).

Project 1: Log Anomaly Detection That Actually Reduces Pages

Goal: learn “normal” for a service’s logs and highlight weird lines before they flood Slack.

Approach: use Isolation Forest on simple TF‑IDF features of log lines. Train daily on the last 24h of logs; stream‑score new lines and emit anomalies.

1) Extract training data from journald (adapt the unit and window):

journalctl -u myservice --since "24 hours ago" --no-pager \
  | sed 's/^[A-Za-z0-9:.\- ]\{,32\}//' > ~/myservice-logs-train.txt

2) Train and persist a model:

cat > train_isoforest.py <<'PY'
import sys, joblib, pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import IsolationForest
from sklearn.pipeline import Pipeline

if len(sys.argv) != 3:
    print("Usage: train_isoforest.py <train.txt> <model.pkl>", file=sys.stderr); sys.exit(1)

train_path, model_path = sys.argv[1], sys.argv[2]
lines = [l.strip() for l in open(train_path, 'r', errors='ignore') if l.strip()]

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1,2), min_df=2)),
    ("iso", IsolationForest(n_estimators=200, contamination=0.005, random_state=42)),
])
pipe.fit(lines)
joblib.dump(pipe, model_path)
print(f"Model saved to {model_path}, trained on {len(lines)} lines")
PY

python train_isoforest.py ~/myservice-logs-train.txt ~/myservice-logs.model

3) Stream new logs and flag anomalies in real time:

cat > predict_isoforest.py <<'PY'
import sys, joblib
pipe = joblib.load(sys.argv[1])
for line in sys.stdin:
    s = line.strip()
    if not s: continue
    pred = pipe.named_steps['iso'].predict(pipe.named_steps['tfidf'].transform([s]))[0]
    if pred == -1:
        print(f"[ANOMALY] {s}", flush=True)
PY

journalctl -u myservice -f --no-pager \
  | sed 's/^[A-Za-z0-9:.\- ]\{,32\}//' \
  | python predict_isoforest.py ~/myservice-logs.model \
  | tee -a ~/myservice-anomalies.log

Tips:

  • Start with a low contamination (0.005–0.02) and tune.

  • Route anomalies to Alertmanager or a Slack webhook only after you measure precision.

  • Retrain nightly via cron or a systemd timer.

Project 2: Incident Summaries From Your Terminal (Local LLM)

Goal: reduce MTTR and postmortem time by auto‑summarizing the last N minutes of alerts/logs.

Approach: collect recent events, ask a small local model (Ollama “mistral”) for a crisp summary and top suspects.

1) Quick incident summarizer:

cat > summarize-incidents.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail

SINCE_MINUTES=${1:-30}
MODEL=${MODEL:-mistral}
OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11434}

EVENTS=$(journalctl --since "${SINCE_MINUTES} minutes ago" -p warning..alert --no-pager \
  | tail -n 1000 | sed 's/^[A-Za-z0-9:.\- ]\{,32\}//' | sed 's/[[:cntrl:]]//g')

read -r -d '' PROMPT <<'P'
You are an SRE assistant. Given raw events:
---
{EVENTS}
---
Produce:

- 5-bullet incident summary

- Top 3 likely root causes (hypotheses)

- Suggested next 3 commands to run (Bash)
Be concise and technical.
P

PAYLOAD=$(jq -n --arg m "$MODEL" --arg p "${PROMPT//\{EVENTS\}/$EVENTS}" '{model:$m, prompt:$p, stream:false}')
curl -s "$OLLAMA_HOST/api/generate" -H "Content-Type: application/json" -d "$PAYLOAD" \
 | jq -r '.response'
SH
chmod +x summarize-incidents.sh

# Run it
./summarize-incidents.sh 45

Not using Ollama? Point the script at any OpenAI‑compatible endpoint by transforming the payload accordingly, or switch to a cloud LLM by replacing the curl call.

Guardrails:

  • Redact secrets before sending to any model.

  • Keep summaries ephemeral unless required for audit.

Project 3: Auto‑Remediation Playbooks Ranked By Similarity

Goal: when an alert fires, immediately suggest (or auto‑execute) the most probable runbook/script.

Approach: maintain a YAML of runbooks and simple “symptoms”; use fuzzy matching to rank suggestions. It’s fast, explainable, and easy to approve.

1) Define runbooks:

cat > runbooks.yml <<'YML'

- id: restart_nginx
  symptom: "nginx 502 or 5xx upstream errors"
  command: "sudo systemctl restart nginx"
  safe: true

- id: clear_disk_cache
  symptom: "high iowait, disk pressure, cache reclaim"
  command: "echo 3 | sudo tee /proc/sys/vm/drop_caches"
  safe: false

- id: scale_api
  symptom: "api p95 latency regression and cpu saturation"
  command: "kubectl scale deploy api --replicas=+$INCREMENT"
  safe: false
YML

2) Rank recommendations:

cat > recommend_runbooks.py <<'PY'
import sys, yaml
from rapidfuzz import fuzz, process

if len(sys.argv) < 3:
    print("Usage: recommend_runbooks.py <runbooks.yml> <alert text>", file=sys.stderr); sys.exit(1)
rb = yaml.safe_load(open(sys.argv[1]))
alert = " ".join(sys.argv[2:]).lower()

choices = {r['id']: r['symptom'].lower() for r in rb}
scores = process.extract(alert, choices, scorer=fuzz.token_set_ratio, limit=5)
for (rb_id, _symptom, score) in scores:
    r = next(x for x in rb if x['id']==rb_id)
    print(f"{score}\t{r['id']}\t{r['command']}\t{'safe' if r.get('safe') else 'risky'}")
PY

3) Wire into your alert pipeline (example Bash wrapper):

cat > suggest-remediation.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
source ~/.venvs/sre-ai/bin/activate

ALERT_TEXT=${1:-"nginx 502 on /checkout from upstream"}
python recommend_runbooks.py runbooks.yml "$ALERT_TEXT" | head -n 3
SH
chmod +x suggest-remediation.sh

./suggest-remediation.sh "Upstream 5xx surge on nginx ingress, checkout path"

Productionize:

  • Auto‑execute only “safe: true” with guardrails and logging.

  • Require human ACK for “risky” items.

  • Record recommendation → outcome to iteratively refine symptoms.

Project 4: Capacity Forecasting You Can Schedule in Cron

Goal: warn days ahead when a key metric (load, memory, requests) will exceed a threshold.

Approach: pull a metric from Prometheus, fit a simple ARIMA with statsmodels, predict next 7 days, alert if a breach is likely.

1) Pull a metric to CSV:

END=$(date -u +%s)
START=$((END - 14*24*3600))
STEP=300  # 5m
QUERY='node_load1{instance="nodeA"}'

curl -sG 'http://prometheus:9090/api/v1/query_range' \
  --data-urlencode query="$QUERY" \
  --data-urlencode start="$START" \
  --data-urlencode end="$END" \
  --data-urlencode step="$STEP" \
| jq -r '.data.result[0].values[] | @csv' > load.csv

2) Forecast and print a heads‑up:

cat > forecast_arima.py <<'PY'
import sys, csv, pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX
import numpy as np

if len(sys.argv) < 4:
    print("Usage: forecast_arima.py <csv> <step_seconds> <threshold>", file=sys.stderr); sys.exit(1)

csv_path, step, threshold = sys.argv[1], int(sys.argv[2]), float(sys.argv[3])
ts = []
with open(csv_path) as f:
    for t, v in csv.reader(f):
        ts.append((pd.to_datetime(float(t), unit='s'), float(v)))
s = pd.Series({t:v for t,v in ts}).asfreq(f'{step}s').interpolate()

model = SARIMAX(s, order=(1,1,1), seasonal_order=(1,1,1, int((24*3600)/step)), enforce_stationarity=False, enforce_invertibility=False)
res = model.fit(disp=False)
pred = res.get_forecast(steps=int((7*24*3600)/step))
mean = pred.predicted_mean

exceed = mean[mean > threshold]
if not exceed.empty:
    first = exceed.index[0]
    print(f"WARNING: forecast exceeds {threshold} at ~{first} (value={mean[first]:.2f})")
else:
    print("OK: forecast stays below threshold for 7 days.")
PY

python forecast_arima.py load.csv $STEP 3.0

Operationalize:

  • Run daily via cron/systemd; create an Alertmanager “external check” or push to a metric (e.g., pushgateway) that Grafana can visualize.

  • Keep it simple: one metric per job. Expand once you trust results.

How These Fit Together

  • Detect anomalies (Project 1) → summarize them (Project 2) → suggest/run fixes (Project 3) → avoid repeats by forecasting stress before it hits (Project 4).

  • Each project is decoupled. Start with one; add others as confidence grows.

  • Everything runs with Bash + Python; no proprietary lock‑in required.

Real‑World Tips

  • Start narrow: one service, one log stream, one metric. Measure precision/recall.

  • Keep a feedback loop: track false positives and edit parameters or runbooks weekly.

  • Security: sanitize logs before LLMs, avoid sending secrets/PII, and pin model versions.

  • Governance: log every auto‑action (who/what/when/why) to a tamper‑evident store.

Conclusion and Next Steps

Small AI automations can save you hours per week and real incident minutes. You don’t need a platform migration—just a terminal and intent. Pick one project:

  • If you’re drowning in logs: ship Project 1 today.

  • If postmortems drag: wire up Project 2.

  • If pages repeat: add Project 3.

  • If capacity surprises you: schedule Project 4.

Next: drop these snippets into a repo, add a Makefile and systemd units, and pilot on a non‑critical service. After one week, review outcomes and iterate.

If you want a follow‑up post with systemd units, cron samples, and Grafana panels for each project, tell me which one you’re starting with.

Related posts