Posted on
• Artificial Intelligence

Beginner's Guide to AI-Powered Observability

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Beginner’s Guide to AI-Powered Observability (for Linux and Bash Users)

You can’t fix what you can’t see—and modern systems generate more logs, metrics, and traces than any human can parse. The result: alert fatigue, missed incidents, and long MTTR. AI-powered observability helps you separate signal from noise by learning normal behavior and surfacing anomalies automatically.

This guide shows you how to stand up a minimal, open-source observability stack on Linux and add a simple AI-driven anomaly detector with nothing more than Bash, Python, and HTTP APIs.

  • You’ll collect metrics with Prometheus and Node Exporter

  • Visualize with Grafana

  • Detect anomalies with a lightweight Python script (Isolation Forest)

  • All installation commands for apt, dnf, and zypper included

Why AI-Powered Observability is Worth Your Time

  • Scale and cardinality: As services scale, dashboards multiply and labels explode—manual eyeballing just can’t keep up.

  • Seasonality and drift: Weekday vs weekend traffic, monthly batch jobs, code rollouts—AI can learn patterns humans forget to account for.

  • Faster triage: AI narrows your search space to the most “abnormal” timelines, helping you ask better questions, faster.

Real-world example: After a deploy, p95 latency spikes but CPU and memory look “fine.” An anomaly detector flags a sudden jump in TCP retransmits and context switches—pointing you to a kernel-level or NIC problem you might have missed.

Step 1: Install the Core Stack (Prometheus, Node Exporter, Grafana)

You’ll run Node Exporter on each machine you want to observe and scrape it with Prometheus. Grafana visualizes the data.

Packages you’ll install:

  • Prometheus (server)

  • Node Exporter (agent)

  • Grafana (UI)

  • Python3 and pip (for the AI step)

Note: Package names can vary slightly across distros. When in doubt, search first (e.g., zypper search node_exporter).

Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y prometheus prometheus-node-exporter python3 python3-pip curl gnupg
# Add Grafana OSS repo
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

Fedora/RHEL/CentOS (dnf)

# On RHEL/CentOS, enable EPEL for node_exporter if needed
# sudo dnf install -y epel-release

sudo dnf install -y prometheus node_exporter python3 python3-pip curl

# Add Grafana OSS repo
sudo tee /etc/yum.repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
EOF

sudo dnf install -y grafana

openSUSE/SLE (zypper)

sudo zypper refresh
# Node Exporter package name can differ; try one of:
# sudo zypper install -y prometheus-node_exporter
# or:
sudo zypper install -y prometheus golang-github-prometheus-node_exporter python3 python3-pip curl

# Add Grafana OSS repo
sudo zypper addrepo https://packages.grafana.com/oss/rpm grafana
sudo rpm --import https://packages.grafana.com/gpg.key
sudo zypper refresh
sudo zypper install -y grafana

Enable and start services:

sudo systemctl enable --now prometheus
sudo systemctl enable --now node_exporter || sudo systemctl enable --now prometheus-node-exporter
sudo systemctl enable --now grafana-server

Default ports:

  • Prometheus: 9090

  • Node Exporter: 9100

  • Grafana: 3000

Quick check:

curl -s localhost:9100/metrics | head
curl -s localhost:9090/-/ready

Step 2: Wire Up Scraping and Dashboards

Point Prometheus at Node Exporter. Edit /etc/prometheus/prometheus.yml (or the distro-specific path) and add a scrape config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

Then reload Prometheus:

sudo systemctl restart prometheus

Add a Grafana data source (Prometheus at http://localhost:9090):

  • In the UI: http://localhost:3000 (default admin/admin; change it)

  • Or via API:

# Requires an API token with Admin or Editor role:
# export GRAFANA_TOKEN="..."
curl -s -X POST http://localhost:3000/api/datasources \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name":"Prometheus",
    "type":"prometheus",
    "url":"http://localhost:9090",
    "access":"proxy",
    "basicAuth":false
  }'

Import a ready-made dashboard (e.g., “Node Exporter Full” ID 1860):

  • In the UI: Dashboards > Import > enter 1860

  • Or via API with the JSON body (exported from grafana.com)

You now have system metrics (CPU, memory, disk, network) streaming into graphs.

Step 3: Add AI Anomaly Detection (Python + Isolation Forest)

We’ll fetch a Prometheus time series (CPU usage) and run a simple anomaly detector. This is intentionally minimal so you can extend it later.

Install Python packages to your user site:

python3 -m pip install --user --upgrade pip
python3 -m pip install --user numpy pandas scikit-learn requests
# Ensure ~/.local/bin is on PATH:
export PATH="$HOME/.local/bin:$PATH"

Create ai_anomaly.py:

#!/usr/bin/env python3
import os, time, math
import requests
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from datetime import datetime, timedelta, timezone

PROM_URL = os.environ.get("PROM_URL", "http://localhost:9090")
INSTANCE = os.environ.get("INSTANCE", "localhost:9100")  # node_exporter target
WINDOW_MIN = int(os.environ.get("WINDOW_MIN", "180"))    # analyze last 3h
STEP = os.environ.get("STEP", "30s")                     # sampling step
GRAFANA_URL = os.environ.get("GRAFANA_URL", "http://localhost:3000")
GRAFANA_TOKEN = os.environ.get("GRAFANA_TOKEN")          # optional for annotations

# PromQL: CPU usage % per instance (100 - idle%)
QUERY = f'100 - (avg by (instance)(rate(node_cpu_seconds_total{{mode="idle",instance="{INSTANCE}"}}[5m])) * 100)'

def prom_range_query(query, start, end, step):
    r = requests.get(f"{PROM_URL}/api/v1/query_range", params={
        "query": query,
        "start": start.isoformat(),
        "end": end.isoformat(),
        "step": step
    }, timeout=15)
    r.raise_for_status()
    data = r.json()
    assert data["status"] == "success", data
    results = data["data"]["result"]
    if not results:
        return pd.DataFrame(columns=["ts","value"])
    # Use first series (by instance)
    values = results[0]["values"]  # [ [ts, "val"], ... ]
    rows = [(datetime.fromtimestamp(float(ts), tz=timezone.utc), float(val)) for ts,val in values if val != "NaN"]
    return pd.DataFrame(rows, columns=["ts","value"]).dropna()

def detect_anomalies(df):
    if len(df) < 30:
        return df.assign(anom=False, score=0.0)
    X = df["value"].values.reshape(-1,1)
    # IsolationForest for unsupervised anomalies; tune contamination if too many/few
    model = IsolationForest(n_estimators=200, contamination=0.02, random_state=42)
    model.fit(X)
    scores = model.decision_function(X)
    preds = model.predict(X)  # -1 anomaly, 1 normal
    out = df.copy()
    out["anom"] = (preds == -1)
    out["score"] = -scores  # higher = more anomalous
    return out

def annotate_grafana(text, when):
    if not GRAFANA_TOKEN:
        return
    payload = {
        "text": text,
        "tags": ["ai", "anomaly", "cpu"],
        "time": int(when.timestamp() * 1000)
    }
    headers = {
        "Authorization": f"Bearer {GRAFANA_TOKEN}",
        "Content-Type": "application/json"
    }
    try:
        r = requests.post(f"{GRAFANA_URL}/api/annotations", json=payload, headers=headers, timeout=10)
        r.raise_for_status()
    except Exception as e:
        print(f"warn: failed to create grafana annotation: {e}")

def main():
    end = datetime.now(timezone.utc)
    start = end - timedelta(minutes=WINDOW_MIN)
    df = prom_range_query(QUERY, start, end, STEP)
    if df.empty:
        print("no data from Prometheus; check PROM_URL/INSTANCE settings")
        return
    out = detect_anomalies(df)
    anomalies = out[out["anom"]].sort_values("score", ascending=False)
    if anomalies.empty:
        print("OK: no anomalies detected in CPU usage")
        return
    # Report top anomalies
    print("Anomalies detected (most severe first):")
    for _, row in anomalies.head(5).iterrows():
        t = row["ts"].astimezone().strftime("%Y-%m-%d %H:%M:%S%z")
        print(f"- {t}: cpu={row['value']:.2f}% score={row['score']:.3f}")
        annotate_grafana(f"AI anomaly: CPU {row['value']:.1f}% on {INSTANCE}", row["ts"])

if __name__ == "__main__":
    main()

Run it:

chmod +x ai_anomaly.py
./ai_anomaly.py
# Customize:
# PROM_URL, INSTANCE, WINDOW_MIN, STEP, GRAFANA_URL, GRAFANA_TOKEN
# Example:
# PROM_URL=http://localhost:9090 INSTANCE=localhost:9100 GRAFANA_TOKEN=... ./ai_anomaly.py

Automate via cron (runs every 5 minutes):

crontab -e
*/5 * * * * PROM_URL=http://localhost:9090 INSTANCE=localhost:9100 /usr/bin/env python3 $HOME/ai_anomaly.py >> $HOME/ai_anomaly.log 2>&1

What you get:

  • Console output listing anomalous CPU points

  • Optional Grafana annotations marking anomalies on your dashboards

Tip: Swap the PromQL query to target other signals:

  • Disk saturations: rate(node_disk_io_time_seconds_total[5m]) * 100

  • Network errors: rate(node_network_receive_errs_total[5m])

  • App metrics (e.g., HTTP 5xx): your app’s counter via Prometheus client libraries

Step 4: Make It Useful in the Real World

  • Correlate signals: Add panels for CPU, load, TCP retransmits, disk I/O, and app error rates on the same time axis. Anomalies that pop on multiple panels warrant faster action.

  • Baselines per environment: Run the detector per instance or service. Weekday/weekend behavior can differ; a single global threshold is rarely enough.

  • Alerting: Use Prometheus Alertmanager for known bad conditions; keep the AI detector for unknown-unknowns. Example rule for high 5xx rate:

    groups:
- name: app.rules
rules:
- alert: HighErrorRate
  expr: rate(http_server_errors_total[5m]) > 0.05
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "High 5xx rate"
    description: "Error rate >5% for 5m"

  • Post-incident learning: Save anomaly windows and root causes. Tune contamination and add features (e.g., combine CPU, load, GC pauses) to reduce false positives over time.

Real-world example: In a k8s node with intermittent pod OOMKills, the anomaly detector on container_memory_working_set_bytes flagged sharp, short-lived spikes hours before OOMs. Adding a Prometheus alert on the derivative caught the issue earlier; autoscaling limits were then adjusted, eliminating the incident class.

Troubleshooting

  • Grafana won’t start: sudo journalctl -u grafana-server -f

  • No Prometheus data: curl -s localhost:9100/metrics and check Prometheus targets at http://localhost:9090/targets

  • Python module errors: ensure ~/.local/bin is on PATH and pip installed packages under your user

Conclusion and Next Steps (CTA)

You’ve built a lean, AI-assisted observability loop:

  • Metrics flow from Node Exporter to Prometheus

  • Grafana visualizes

  • A Python detector flags anomalies and can annotate dashboards

Next steps:

  • Point Prometheus at more instances and app metrics

  • Extend the detector to multiple signals (CPU + errors + latency)

  • Add simple remediation scripts when certain anomalies repeat

When you’re ready, explore logs (Loki/Promtail) and traces (OpenTelemetry + Jaeger) and feed their aggregates into the same anomaly pipeline.

Have a favorite metric you want to “AI-ify”? Paste the PromQL here and I’ll help you adapt the detector.

Related posts