Posted on
• Artificial Intelligence

Artificial Intelligence DevOps Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Driven DevOps Monitoring on Linux (with Bash-friendly tooling)

Incidents don’t announce themselves. They creep in as small drifts in CPU, a spike in GPU temperature, or a slow bleed in request latency—and by the time your static thresholds fire, customer impact has begun. AI-assisted monitoring changes that equation. With a small amount of Bash glue and open tooling on Linux, you can surface anomalies earlier, reduce alert noise, and catch regressions right after deploys.

This guide shows you how to bolt an AI-assisted anomaly detector onto a familiar, open-source monitoring stack (Prometheus + Grafana), using lightweight Python and standard Linux tools. You’ll get:

  • A clear, minimal architecture that runs locally or in prod

  • Installation commands for apt, dnf, and zypper where applicable

  • 3–5 actionable steps, including an anomaly detector you can extend

Why AI for DevOps Monitoring?

  • Dynamic baselines beat static thresholds. Microservices, autoscaling, and day/night cycles change “normal” constantly. AI models can learn (or approximate) normal from recent history and flag outliers reliably.

  • Earlier signals, fewer pages. Anomaly detection captures subtle drifts before they turn into outages, improving MTTR and often MTTD.

  • Works with what you have. OpenMetrics and Prometheus expose the data; lightweight Python does the math. No vendor lock-in, no heavy infrastructure required.

  • Good for humans. Turn “thousands of metrics” into “three unusual patterns,” giving SREs and developers time to focus.

Step 1 — Install the fundamentals (metrics first)

We’ll start with:

  • node_exporter for host metrics

  • Prometheus for scraping and querying

  • Grafana for visualization

First, install common prerequisites:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y curl jq python3 python3-pip podman

  • Fedora/RHEL/CentOS Stream (dnf):

    sudo dnf install -y curl jq python3 python3-pip podman

  • openSUSE Leap/Tumbleweed (zypper):

    sudo zypper refresh
sudo zypper install -y curl jq python3 python3-pip podman

Install and start node_exporter (host metrics):

  • Debian/Ubuntu (apt):

    sudo apt install -y prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter

  • Fedora/RHEL/CentOS Stream (dnf):

    sudo dnf install -y node_exporter
sudo systemctl enable --now node_exporter

  • openSUSE Leap/Tumbleweed (zypper):

    sudo zypper install -y golang-github-prometheus-node_exporter
sudo systemctl enable --now node_exporter

Note: Package names may vary slightly by distro version. If your repository lacks Grafana or newer Prometheus, use containers below.

Step 2 — Stand up Prometheus and Grafana quickly (containers)

We’ll use Podman, but you can swap podman with docker if you prefer.

Create a Prometheus config that scrapes node_exporter:

mkdir -p ~/ai-devops/monitoring
cat > ~/ai-devops/monitoring/prometheus.yml << 'YAML'
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
YAML

Run Prometheus:

podman run -d --name prometheus \
  -p 9090:9090 \
  -v ~/ai-devops/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:Z \
  quay.io/prometheus/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml

Run Grafana:

podman volume create grafana-storage
podman run -d --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD='admin' \
  -v grafana-storage:/var/lib/grafana \
  docker.io/grafana/grafana:latest

  • Open Prometheus: http://localhost:9090

  • Open Grafana: http://localhost:3000 (user: admin, pass: admin as set above)

  • In Grafana, add Prometheus as a data source: URL http://host.docker.internal:9090 (macOS/Windows) or http://localhost:9090 (Linux). For Podman on Linux, localhost works.

Tip: Prefer native packages? Try:

  • Debian/Ubuntu: sudo apt install -y prometheus grafana (if available), else use containers.

  • Fedora: sudo dnf install -y prometheus grafana

  • openSUSE: sudo zypper install -y prometheus grafana

If a package isn’t in your repo, stick with the container method above for predictable versions.

Step 3 — Add AI-assisted anomaly detection (lightweight Python)

We’ll pull a PromQL time series from Prometheus and flag anomalies using a robust Z-score (Median Absolute Deviation). This avoids heavy ML dependencies while catching real outliers.

Install Python deps:

python3 -m pip install --user numpy requests

Create the detector script:

mkdir -p ~/ai-devops/ai
cat > ~/ai-devops/ai/ai_mon.py << 'PY'
#!/usr/bin/env python3
import os, time, json, math
import requests
import numpy as np
from datetime import datetime, timedelta, timezone
from urllib.parse import quote_plus

PROM_URL = os.getenv("PROM_URL", "http://localhost:9090")
# PromQL for host CPU utilization (percentage):
# 100 - avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
PROMQL = os.getenv(
    "PROMQL",
    '100 - avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100'
)
WINDOW_MINUTES = int(os.getenv("WINDOW_MINUTES", "60"))
STEP = os.getenv("STEP", "30s")
Z_THRESHOLD = float(os.getenv("Z_THRESHOLD", "4.0"))  # robust z-score threshold
ALERT_WEBHOOK = os.getenv("ALERT_WEBHOOK", "")  # e.g., Slack Incoming Webhook URL
ALERT_PREFIX = os.getenv("ALERT_PREFIX", "[AI-MON]")

def robust_z_scores(values):
    # Return z-scores using median and MAD (median absolute deviation)
    arr = np.array(values, dtype=float)
    med = np.median(arr)
    mad = np.median(np.abs(arr - med))
    if mad == 0:
        # fallback to std to avoid divide by zero
        std = np.std(arr) or 1.0
        return (arr - np.mean(arr)) / std
    return 0.6745 * (arr - med) / mad

def query_range(prom_url, promql, start_ts, end_ts, step):
    url = f"{prom_url}/api/v1/query_range?query={quote_plus(promql)}&start={start_ts}&end={end_ts}&step={step}"
    r = requests.get(url, timeout=15)
    r.raise_for_status()
    data = r.json()
    if data.get("status") != "success":
        raise RuntimeError(f"Prometheus error: {data}")
    return data["data"]["result"]

def post_alert(webhook, text):
    if not webhook:
        print(text)
        return
    try:
        requests.post(webhook, json={"text": text}, timeout=10)
    except Exception as e:
        print(f"Alert post failed: {e}")

def main():
    now = datetime.now(timezone.utc)
    start = now - timedelta(minutes=WINDOW_MINUTES)
    series = query_range(PROM_URL, PROMQL, int(start.timestamp()), int(now.timestamp()), STEP)

    anomalies = []
    for s in series:
        instance = s["metric"].get("instance", "unknown")
        # Each value: [timestamp, value_str]
        values = [float(v[1]) for v in s["values"] if v[1] not in ("NaN", "Inf")]
        if len(values) < 10:
            continue
        z = robust_z_scores(values)
        latest = values[-1]
        latest_z = float(z[-1])
        if math.isnan(latest_z):
            continue
        if abs(latest_z) >= Z_THRESHOLD:
            anomalies.append((instance, latest, latest_z))

    if anomalies:
        lines = [f"{ALERT_PREFIX} Anomalies detected for query:\n{PROMQL}"]
        for inst, val, z in anomalies:
            lines.append(f"- instance={inst} value={val:.2f}% z={z:.2f}")
        post_alert(ALERT_WEBHOOK, "\n".join(lines))
    else:
        print("No anomalies.")

if __name__ == "__main__":
    main()
PY
chmod +x ~/ai-devops/ai/ai_mon.py

Run it ad-hoc:

export PROM_URL="http://localhost:9090"
export ALERT_WEBHOOK="" # optional Slack Incoming Webhook URL
~/ai-devops/ai/ai_mon.py

Tune it:

  • PROMQL: Point to any high-signal SLI (e.g., 99th latency, error rate, queue depth)

  • Z_THRESHOLD: Lower for more sensitivity; raise to reduce noise

  • WINDOW_MINUTES and STEP: Match your signal’s cadence

Optional: Send to Slack (Incoming Webhook):

export ALERT_WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"

Alternative: Send to Alertmanager

  • Post JSON to Alertmanager’s /api/v2/alerts with standard labels/annotations (requires Alertmanager accessible). Slack is simpler to start.

Step 4 — Automate and harden (systemd timers; Bash-first)

Use a user-level systemd timer to run detection every minute.

Create a service:

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/ai-mon.service << 'UNIT'
[Unit]
Description=AI anomaly detector for Prometheus

[Service]
Type=oneshot
Environment=PROM_URL=http://localhost:9090
# Example robust CPU utilization PromQL:
Environment=PROMQL=100 - avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
Environment=Z_THRESHOLD=4.0
# Uncomment to enable Slack alerts:
# Environment=ALERT_WEBHOOK=https://hooks.slack.com/services/XXX/YYY/ZZZ
ExecStart=%h/ai-devops/ai/ai_mon.py
UNIT

Create a timer:

cat > ~/.config/systemd/user/ai-mon.timer << 'UNIT'
[Unit]
Description=Run AI anomaly detector every minute

[Timer]
OnCalendar=*:0/1
Persistent=true

[Install]
WantedBy=timers.target
UNIT

systemctl --user daemon-reload
systemctl --user enable --now ai-mon.timer
systemctl --user list-timers | grep ai-mon

This runs your AI check every minute and only pages when behavior strays off the learned baseline.

Optional: GPU and logs

  • GPU monitoring (NVIDIA DCGM Exporter) via container:

    podman run -d --name dcgm-exporter --privileged -p 9400:9400 \
nvcr.io/nvidia/k8s/dcgm-exporter:latest

    Then add it to Prometheus targets and point PROMQL to GPU utilization or temperature.

  • Logs: Add Loki + Promtail for AI-on-logs later. Start with metrics first to keep scope tight; add logs once dashboards are stable.

Real-world examples where this shines

  • Post-deploy latency regression: p99 jumped only 12%—below static thresholds—but anomaly detection flagged it within 2 minutes, enabling an immediate rollback.

  • GPU thermal runaway: Training job caused intermittent GPU temperature spikes. MAD-based detector alerted on the spike pattern, not just occasional high temps, preventing throttling.

  • Slowly increasing error rate: An upstream timeout drifted from 0.1% to 0.6% over an hour; anomaly detection caught the trend before SLOs were breached.

Practical tips for success

  • Start with high-signal metrics: latency, error rate, queue depth, CPU saturation, JVM GC pauses, GPU utilization/temperature.

  • Keep label cardinality in check: too many labels balloon series count and degrade Prometheus performance.

  • Baseline per service/instance: anomalies that aggregate over instances can hide outliers.

  • Review weekly: tune thresholds and queries based on actual incidents and false positives.

  • Version your queries: store PROMQL in Git next to your services for reproducibility.

Conclusion and next steps

You don’t need a giant ML platform to get AI benefits in monitoring. With node_exporter, Prometheus, Grafana, and a 100-line Python script, you can detect anomalies earlier and reduce noise—today.

Next steps:

  • Add a second detector for your top SLI (e.g., 99th percentile latency)

  • Wire Slack or Alertmanager, then tune sensitivity for one week

  • Expand coverage: add GPU metrics or a critical business KPI

  • Automate dashboards and detectors via Git, and review weekly

Have questions or want a follow-up post on OpenTelemetry + anomaly detection for traces? Tell me what you’re instrumenting, and I’ll tailor the next guide.

Related posts