4 Practical AI Projects DevOps Engineers Can Ship This Month

If you’re on call, you already know the drill: noisy alerts, flakey builds, surprise traffic spikes, and runbooks you can’t find fast enough. The good news is that a little bit of applied AI can take a lot of that pain away—without boiling the ocean or hiring a research team.

This post shows you four focused, Bash-friendly AI projects you can implement quickly to reduce toil, cut costs, and improve reliability. Each one uses open-source tools, is scriptable, and plays nicely with your existing Linux toolchain.

• Why this matters now:
  • You already have the data (logs, metrics, builds, runbooks).
  • Python + Bash glue code is enough to deliver value.
  • Open-source ML libraries make “AI for Ops” pragmatic and cheap.

Prerequisites (install via apt, dnf, or zypper)

You’ll need Python, build tools, and a few CLI utilities.

• Debian/Ubuntu (apt):

  sudo apt-get update
  sudo apt-get install -y python3 python3-venv python3-pip build-essential python3-dev git curl jq
  

• Fedora/RHEL/CentOS (dnf):

  sudo dnf install -y python3 python3-virtualenv python3-pip gcc gcc-c++ make python3-devel git curl jq
  

• openSUSE/SLES (zypper):

  sudo zypper refresh
  sudo zypper install -y python3 python3-pip python3-virtualenv gcc gcc-c++ make python3-devel git curl jq
  

Optional: container runtime (pick one)

• Debian/Ubuntu (apt):

  # Docker (community package)
  sudo apt-get install -y docker.io
  # or Podman (daemonless, rootless-friendly)
  sudo apt-get install -y podman
  

• Fedora/RHEL/CentOS (dnf):

  sudo dnf install -y podman
  # or moby-engine (Docker) on Fedora
  sudo dnf install -y moby-engine
  

• openSUSE/SLES (zypper):

  sudo zypper install -y docker
  # or
  sudo zypper install -y podman
  

Create a fresh Python virtual environment for each project:

python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install --upgrade pip wheel

Project 1: Log Anomaly Detector (Sidecar-friendly)

Detect weird spikes or outliers in your web logs and alert before users complain.

• What it does: tails JSON-formatted NGINX logs, computes light-weight features, learns a baseline, flags outliers with Isolation Forest, and sends a Slack webhook alert (or logs to stderr).

• Where it fits: as a sidecar on web services or as a systemd service on a log collector host.

Step 1 — Configure NGINX to emit JSON logs:

# /etc/nginx/nginx.conf (inside http {})
log_format json escape=json '{ "time_local":"$time_local", "remote_addr":"$remote_addr", '
                            '"request":"$request", "status":$status, '
                            '"body_bytes_sent":$body_bytes_sent, '
                            '"request_time":$request_time, '
                            '"upstream_response_time":"$upstream_response_time" }';
access_log /var/log/nginx/access.json.log json;

Then reload NGINX:

sudo nginx -t && sudo systemctl reload nginx

Step 2 — Install Python deps:

. .venv/bin/activate
pip install scikit-learn pandas ujson requests

Step 3 — Anomaly detection script:

#!/usr/bin/env python3
import sys, ujson as json, time, os
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
import requests

SLACK_WEBHOOK = os.getenv("SLACK_WEBHOOK", "")  # optional

def send_alert(msg):
    if SLACK_WEBHOOK:
        try:
            requests.post(SLACK_WEBHOOK, json={"text": msg}, timeout=3)
        except Exception as e:
            print(f"[warn] slack send failed: {e}", file=sys.stderr)
    else:
        print(f"[alert] {msg}", file=sys.stderr)

model = IsolationForest(n_estimators=100, contamination=0.01, random_state=42)
buffer = []
trained = False
last_alert_ts = 0

def features(obj):
    status = obj.get("status", 0)
    rtime = float(obj.get("request_time", 0) or 0)
    bytes_sent = float(obj.get("body_bytes_sent", 0) or 0)
    return [status, rtime, np.log1p(bytes_sent)]

for line in sys.stdin:
    try:
        obj = json.loads(line)
        x = features(obj)
        buffer.append(x)
        if len(buffer) >= 500 and not trained:
            X = np.array(buffer[-500:])
            model.fit(X)
            trained = True
        if trained and len(buffer) % 10 == 0:
            X = np.array(buffer[-200:])
            scores = model.decision_function(X)
            # last point score: lower => more anomalous
            if scores[-1] < -0.15 and time.time() - last_alert_ts > 30:
                last_alert_ts = time.time()
                send_alert(f"Log anomaly: score={scores[-1]:.3f} latest={obj.get('request','?')} status={obj.get('status')}")
    except Exception:
        continue

Run it:

tail -F /var/log/nginx/access.json.log | ./log_anomaly.py

Systemd unit (optional):

# /etc/systemd/system/log-anomaly.service
[Unit]
Description=Log Anomaly Detector
After=network.target

[Service]
Environment=SLACK_WEBHOOK=https://hooks.slack.com/services/XXX/YYY/ZZZ
ExecStart=/bin/bash -lc 'tail -F /var/log/nginx/access.json.log | /opt/anomaly/.venv/bin/python /opt/anomaly/log_anomaly.py'
Restart=always

[Install]
WantedBy=multi-user.target

Why it works: unsupervised models like Isolation Forest detect “out-of-pattern” requests, status spikes, or payload anomalies without labeled data. You get early signals with minimal tuning.

Project 2: Predictive Autoscaling With Time-Series Forecasts

Scale before the traffic spike hits rather than after the HPA panics.

• What it does: pulls a Prometheus time series (e.g., RPS or CPU), forecasts the next 15 minutes with AutoARIMA, converts demand into desired replicas, and optionally runs kubectl scale.

Install Python deps:

. .venv/bin/activate
pip install requests pmdarima numpy

Script:

#!/usr/bin/env python3
import os, time, math, requests, numpy as np
from datetime import datetime, timedelta
from pmdarima import auto_arima
from subprocess import run, CalledProcessError

PROM_URL = os.getenv("PROM_URL", "http://prometheus.monitoring.svc:9090")
QUERY = os.getenv("QUERY", 'sum(rate(http_requests_total[5m]))')
LOOKBACK_MIN = int(os.getenv("LOOKBACK_MIN", "180"))
HORIZON_MIN = int(os.getenv("HORIZON_MIN", "15"))
TARGET_RPS_PER_POD = float(os.getenv("TARGET_RPS_PER_POD", "100"))
SCALE_DEPLOY = os.getenv("SCALE_DEPLOY", "")  # e.g., "deploy/my-app -n prod"

def prom_range_query(q, start, end, step="60s"):
    r = requests.get(f"{PROM_URL}/api/v1/query_range", params={
        "query": q, "start": start, "end": end, "step": step
    }, timeout=5)
    r.raise_for_status()
    data = r.json()["data"]["result"]
    if not data:
        return []
    # pick first series
    return [float(v[1]) for v in data[0]["values"]]

now = datetime.utcnow()
start = int((now - timedelta(minutes=LOOKBACK_MIN)).timestamp())
end = int(now.timestamp())
series = prom_range_query(QUERY, start, end)
if len(series) < 30:
    print("not enough data")
    raise SystemExit(0)

model = auto_arima(series, seasonal=True, m=12, error_action='ignore', suppress_warnings=True)
future = model.predict(n_periods=HORIZON_MIN)  # per-minute forecast
peak = float(np.max(future))
desired = max(1, math.ceil(peak / TARGET_RPS_PER_POD))

print(f"Forecast peak: {peak:.2f} rps; desired replicas: {desired}")
if SCALE_DEPLOY:
    try:
        run(["kubectl", "scale", SCALE_DEPLOY, f"--replicas={desired}"], check=True)
        print("Scaled successfully")
    except CalledProcessError as e:
        print(f"kubectl scale failed: {e}")

Run it as a CronJob or systemd timer every 5 minutes. Start conservatively; set TARGET_RPS_PER_POD to your safe SLO capacity per pod. This is a low-risk “pre-scaling hint” that complements an HPA/KEDA setup.

Project 3: CI Failure Triage (Automatically Label Build Breakages)

Turn piles of build logs into fast, actionable categories (flake, infra, test, lint, dependency).

• What it does: trains a simple text classifier on historical CI logs and then labels new failures, helping you route issues quickly or file tickets.

Install Python deps:

. .venv/bin/activate
pip install scikit-learn joblib

Training data layout (example):

data/
  flake/
    001.txt
    002.txt
  infra/
    003.txt
  test/
    004.txt
  lint/
    005.txt

Train:

#!/usr/bin/env python3
import pathlib, sys
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import joblib

data_dir = pathlib.Path(sys.argv[1] if len(sys.argv)>1 else "data")
texts, labels = [], []
for cls_dir in data_dir.iterdir():
    if cls_dir.is_dir():
        for f in cls_dir.glob("*.txt"):
            texts.append(f.read_text(errors="ignore"))
            labels.append(cls_dir.name)

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=50000, ngram_range=(1,2))),
    ("clf", LinearSVC())
])
Xtr, Xte, ytr, yte = train_test_split(texts, labels, test_size=0.2, random_state=42, stratify=labels)
pipe.fit(Xtr, ytr)
print(classification_report(yte, pipe.predict(Xte)))
joblib.dump(pipe, "ci_triage_model.joblib")
print("Saved model: ci_triage_model.joblib")

Predict on a failing build log:

#!/usr/bin/env python3
import sys, joblib, pathlib
model = joblib.load("ci_triage_model.joblib")
txt = pathlib.Path(sys.argv[1]).read_text(errors="ignore")
pred = model.predict([txt])[0]
print(pred)

Hook into CI by uploading the failing log artifact and running the predictor. You can post labels to GitHub/GitLab via their CLI or APIs, or just add a Slack/ChatOps message. Start with 50–100 labeled examples per class; improve over time.

Project 4: Terminal Runbook Search With Embeddings

Your runbooks are gold—but only if you can find the right command in seconds.

• What it does: embeds your Markdown runbooks using a small sentence transformer and performs semantic search from the terminal. No LLM required.

Install Python deps:

. .venv/bin/activate
pip install sentence-transformers scikit-learn rich

Index your runbooks:

#!/usr/bin/env python3
import pathlib, json, sys
from sentence_transformers import SentenceTransformer

root = pathlib.Path(sys.argv[1] if len(sys.argv)>1 else "runbooks")
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

docs = []
for p in root.rglob("*.md"):
    text = p.read_text(errors="ignore")
    # chunk by paragraphs for better retrieval
    for para in [t.strip() for t in text.split("\n\n") if t.strip()]:
        docs.append({"path": str(p), "text": para})

emb = model.encode([d["text"] for d in docs], show_progress_bar=True, normalize_embeddings=True)
with open("runbook_index.json", "w") as f:
    json.dump({"docs": docs, "emb": emb.tolist()}, f)
print(f"Indexed {len(docs)} chunks")

Query from the terminal:

#!/usr/bin/env python3
import json, sys, numpy as np
from rich.markdown import Markdown
from rich.console import Console
from sentence_transformers import SentenceTransformer
from sklearn.neighbors import NearestNeighbors

idx = json.load(open("runbook_index.json"))
docs, emb = idx["docs"], np.array(idx["emb"])
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
query = " ".join(sys.argv[1:]) or "restart a crashed pod stuck in CrashLoopBackOff"
qv = model.encode([query], normalize_embeddings=True)
nn = NearestNeighbors(n_neighbors=5, metric="cosine").fit(emb)
dist, knn = nn.kneighbors(qv)
console = Console()
for rank, i in enumerate(knn[0], 1):
    d = docs[i]
    console.rule(f"[bold]#{rank} {d['path']} (cosine dist={dist[0][rank-1]:.3f})")
    console.print(Markdown(d["text"]))

Now you can do:

./rbq.py "rotate nginx logs without downtime"

and jump straight to the relevant snippet and file path. Wrap this in a Bash function and you’ve got a powerful TUI helper.

Tips for Running These in Production

• Start shadow mode: log actions without effect, then graduate to alerting, then to automated remediation.

• Version everything: keep your ML scripts and models in Git; ship them as containers if possible.

• Observability for AI: log model version, thresholds, and decisions to make rollback safe.

• Security: validate and sanitize inputs, restrict permissions for any automation (least privilege).

Conclusion and Next Step

AI for DevOps doesn’t have to be complicated. Start with one hotspot:

• If logs are noisy: ship the Log Anomaly Detector.

• If scaling lags traffic: deploy Predictive Autoscaling.

• If builds waste your time: add CI Failure Triage.

• If runbooks are hard to search: enable Terminal Runbook Search.

Pick one project, spin up a .venv, and run the first script today. Once you see the value, containerize it, add a systemd unit or Kubernetes CronJob, and iterate. If you want a starter repository that bundles these patterns, reply and I’ll share a minimal template you can fork.