Posted on
• Artificial Intelligence

Artificial Intelligence Uptime Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Uptime Monitoring with Bash: Smarter Alerts, Fewer False Positives

If your status page says “All systems operational” while users are still complaining, you don’t have an uptime problem—you have an observability problem. Traditional checks answer “Is it up?” AI-enhanced checks answer “Is it healthy, normal, and meeting SLOs?” This post shows how to combine Bash with a lightweight AI detector to catch real issues earlier and silence the noise.

What you’ll get:

  • A tiny, fast Bash probe that logs latency and availability for HTTP endpoints and hosts (ping)

  • A systemd timer for reliable minute-by-minute checks

  • A Python “AI” anomaly detector (IsolationForest) to spot unusual slowness and genuine incidents

  • Optional Slack notifications with context

No heavy agents. No vendor lock-in. Your data stays on your box.

Why AI for uptime?

  • Static thresholds break with traffic patterns. AI baselines adjust to time-of-day, deployments, and seasonality.

  • Uptime ≠ health. AI can flag degraded states (e.g., 200 OK but 4× slower) that traditional checks miss.

  • Alert fatigue kills response time. Anomaly scoring and deduplication reduce noise.

Prerequisites and installation

We’ll use:

  • curl and ping for checks

  • jq and bc for light processing

  • Python 3 with scikit-learn for anomaly detection (small and fast)

Install the packages using your distro’s package manager:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq bc python3 python3-sklearn python3-numpy
# Optional if you prefer cron: sudo apt install -y cron

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y curl jq bc python3 python3-scikit-learn python3-numpy
# Optional if you prefer cron: sudo dnf install -y cronie

openSUSE Leap/Tumbleweed (zypper):

sudo zypper install -y curl jq bc python3 python3-scikit-learn python3-numpy
# Optional if you prefer cron: sudo zypper install -y cronie

Note: We’ll use systemd timers (recommended). Cron is optional.

Step 1 — Define your targets

Create directories and a targets file.

sudo mkdir -p /etc/ai-uptime /var/log/ai-uptime
sudo tee /etc/ai-uptime/targets.txt >/dev/null <<'EOF'
# type,name,value
http,api,https://example.com/health
http,homepage,https://example.com/
ping,dns,1.1.1.1
ping,gateway,192.168.1.1
EOF

Format: type,name,value

  • http: curl the URL and record status + latency

  • ping: ICMP a host and record success + latency

Step 2 — Bash probe to collect metrics

Create the probe script.

sudo tee /usr/local/bin/ai-uptime.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

LOG_DIR="/var/log/ai-uptime"
LOG_FILE="$LOG_DIR/metrics.jsonl"
TARGETS_FILE="/etc/ai-uptime/targets.txt"

mkdir -p "$LOG_DIR"
touch "$LOG_FILE"

# Prevent overlap
exec 9>/var/run/ai-uptime.lock
flock -n 9 || exit 0

ts=$(date -Is)

while IFS=, read -r type name value; do
  # skip comments/blank lines
  [[ -z "${type:-}" || "${type:0:1}" = "#" ]] && continue

  case "$type" in
    http)
      # time_total seconds, http_code
      read -r t code < <(curl -sS -o /dev/null -w '%{time_total} %{http_code}' --max-time 10 "$value" || echo "10.000 000")
      if [[ "$code" =~ ^[0-9]+$ ]] && (( code >= 200 && code < 400 )); then up=1; else up=0; fi
      lat_ms=$(awk -v t="$t" 'BEGIN{printf "%.0f", t*1000}')
      printf '{"ts":"%s","target":"%s","type":"http","status":%d,"latency_ms":%d,"up":%d,"url":"%s"}\n' \
        "$ts" "$name" "${code:-0}" "$lat_ms" "$up" "$value" >>"$LOG_FILE"
      ;;
    ping)
      if out=$(ping -n -c 1 -W 2 "$value" 2>/dev/null); then
        up=1
        lat=$(printf "%s" "$out" | sed -n 's/.*time=\([0-9.]\+\) ms.*/\1/p' | head -n1)
        lat_ms=${lat%.*}
        [[ -z "$lat_ms" ]] && lat_ms=0
      else
        up=0
        lat_ms=0
      fi
      printf '{"ts":"%s","target":"%s","type":"ping","status":%d,"latency_ms":%d,"up":%d,"host":"%s"}\n' \
        "$ts" "$name" "$up" "$lat_ms" "$up" "$value" >>"$LOG_FILE"
      ;;
    *)
      echo "Unknown type $type for $name" >&2
      ;;
  esac
done < "$TARGETS_FILE"
EOF
sudo chmod +x /usr/local/bin/ai-uptime.sh

Quick test:

/usr/local/bin/ai-uptime.sh
tail -n3 /var/log/ai-uptime/metrics.jsonl

Step 3 — Run it every minute with systemd timers

Service and timer:

sudo tee /etc/systemd/system/ai-uptime.service >/dev/null <<'EOF'
[Unit]
Description=AI Uptime probe

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ai-uptime.sh
EOF

sudo tee /etc/systemd/system/ai-uptime.timer >/dev/null <<'EOF'
[Unit]
Description=Run AI Uptime probe every minute

[Timer]
OnBootSec=1min
OnUnitActiveSec=60s
AccuracySec=5s
Unit=ai-uptime.service

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ai-uptime.timer
systemctl list-timers ai-uptime.timer

Prefer cron? Example:

(crontab -l 2>/dev/null; echo "* * * * * /usr/local/bin/ai-uptime.sh") | crontab -

Step 4 — AI anomaly detector (IsolationForest)

This small Python script learns a baseline per target and flags unusual slowness or drops. It requires python3 and scikit-learn (installed above).

sudo tee /usr/local/bin/ai-uptime-anomaly.py >/dev/null <<'EOF'
#!/usr/bin/env python3
import json, os, sys, time
from datetime import datetime, timedelta
from collections import defaultdict, deque
from urllib.request import Request, urlopen
from urllib.error import URLError
import numpy as np
from sklearn.ensemble import IsolationForest

LOG_FILE = "/var/log/ai-uptime/metrics.jsonl"
ANOMALY_LOG = "/var/log/ai-uptime/anomalies.log"
ENV_FILE = "/etc/ai-uptime/env"  # optional AI_UPTIME_SLACK_WEBHOOK="..."

def load_env(path):
    env = {}
    if not os.path.exists(path): return env
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith("#") or "=" not in line: continue
            k, v = line.split("=", 1)
            env[k.strip()] = v.strip().strip('"').strip("'")
    return env

def tail_lines(path, n=1000):
    if not os.path.exists(path): return []
    with open(path, 'r') as f:
        return list(deque(f, maxlen=n))

def pctl(a, q):
    if len(a) == 0: return 0.0
    return float(np.percentile(a, q))

def post_slack(webhook, text):
    if not webhook: return
    data = json.dumps({"text": text}).encode("utf-8")
    req = Request(webhook, data=data, headers={"Content-Type":"application/json"})
    try:
        urlopen(req, timeout=5).read()
    except URLError:
        pass

def main():
    env = load_env(ENV_FILE)
    webhook = env.get("AI_UPTIME_SLACK_WEBHOOK", os.getenv("AI_UPTIME_SLACK_WEBHOOK", ""))

    rows = []
    for line in tail_lines(LOG_FILE, n=2000):
        try:
            rows.append(json.loads(line))
        except Exception:
            continue

    groups = defaultdict(list)
    for r in rows:
        key = f"{r.get('type')}::{r.get('target')}"
        ts = r.get("ts")
        try:
            t = datetime.fromisoformat(ts)
        except Exception:
            continue
        lat = float(r.get("latency_ms", 0.0))
        up = int(r.get("up", 0))
        groups[key].append((t, lat, up, r))

    anomalies = []

    for key, items in groups.items():
        items.sort(key=lambda x: x[0])
        if len(items) < 30:
            continue  # need minimal history

        # features: adjusted latency (penalize failures), up flag
        lats = np.array([it[1] for it in items], dtype=float)
        ups  = np.array([it[2] for it in items], dtype=int)
        # replace failed latency with a high value to make it stand out
        base = np.median(lats[lats > 0]) if np.any(lats > 0) else 1000.0
        adj_lats = np.where(ups == 1, lats, np.maximum(lats, base * 2.0))

        # train on older window, score latest point
        X = np.vstack([adj_lats, ups]).T
        if X.shape[0] < 30: 
            continue
        train_X = X[:-1]
        test_x = X[-1].reshape(1, -1)

        # contamination ~ 5% expected anomalies
        clf = IsolationForest(n_estimators=100, contamination=0.05, random_state=42)
        clf.fit(train_X)
        pred = clf.predict(test_x)[0]  # -1 anomaly, 1 normal
        score = clf.decision_function(test_x)[0]
        last_t, last_lat, last_up, last_raw = items[-1]

        # secondary heuristic: spike vs recent baseline
        recent = adj_lats[-21:-1] if adj_lats.shape[0] > 21 else adj_lats[:-1]
        p90 = pctl(recent, 90)
        spike = (adj_lats[-1] > max(p90, 1.2 * (np.median(recent) if recent.size else base)))

        if pred == -1 or spike or last_up == 0:
            msg = f"[{last_t.isoformat()}] Anomaly: {key} up={last_up} latency_ms={int(last_lat)} score={score:.3f} p90={int(p90)}"
            anomalies.append((msg, last_raw))

    if anomalies:
        with open(ANOMALY_LOG, "a") as f:
            for msg, raw in anomalies:
                f.write(json.dumps({"ts": datetime.utcnow().isoformat(), "msg": msg, "row": raw}) + "\n")
        text = "*AI Uptime* detected anomalies:\n" + "\n".join(f"- {m}" for m, _ in anomalies[:10])
        post_slack(webhook, text)

if __name__ == "__main__":
    main()
EOF
sudo chmod +x /usr/local/bin/ai-uptime-anomaly.py

Optional Slack webhook:

sudo tee /etc/ai-uptime/env >/dev/null <<'EOF'
# Optional Slack Incoming Webhook URL
AI_UPTIME_SLACK_WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
EOF

Run once to verify:

/usr/bin/python3 /usr/local/bin/ai-uptime-anomaly.py
tail -n3 /var/log/ai-uptime/anomalies.log

Step 5 — Schedule the detector

sudo tee /etc/systemd/system/ai-uptime-anomaly.service >/dev/null <<'EOF'
[Unit]
Description=AI Uptime anomaly detector

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/local/bin/ai-uptime-anomaly.py
EOF

sudo tee /etc/systemd/system/ai-uptime-anomaly.timer >/dev/null <<'EOF'
[Unit]
Description=Run AI Uptime anomaly detector every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=15s
Unit=ai-uptime-anomaly.service

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ai-uptime-anomaly.timer

Check status:

systemctl list-timers | grep ai-uptime
journalctl -u ai-uptime.service -u ai-uptime-anomaly.service --since "10 min ago" -n 100 -e

Real-world usage, tuning, and examples

  • Start with SLOs, not guesses

    • Example: “Homepage p95 < 400ms, API health < 250ms.” Adjust contamination (0.05 → 0.02) to be stricter or looser.

  • Combine signals for confidence

    • Keep both HTTP and ping checks. A ping OK + HTTP slow suggests app issue; both failing suggests network.

  • Reduce noise with context

    • The detector compares to a rolling baseline (recent p90 + IsolationForest score). This turns deploy spikes and night traffic changes into fewer pages.

  • Add another check in seconds

    • Append to /etc/ai-uptime/targets.txt:
    http,checkout,https://example.com/checkout/health
ping,public-dns,8.8.8.8
    • Your next run picks it up automatically.

  • Send the alert to the right place

    • Use the Slack webhook, or modify the Python to POST to PagerDuty, Matrix, or email. The small, self-describing JSON lines make integration easy.

Optional log rotation (recommended):

sudo tee /etc/logrotate.d/ai-uptime >/dev/null <<'EOF'
/var/log/ai-uptime/metrics.jsonl /var/log/ai-uptime/anomalies.log {
  daily
  rotate 7
  compress
  missingok
  notifempty
  copytruncate
}
EOF

Troubleshooting

  • No logs? Run the probe manually:

    bash -x /usr/local/bin/ai-uptime.sh

  • Permissions or SELinux denials? Check:

    journalctl -xe

  • Too many anomalies? Lower sensitivity by:

    • Increasing training history (edit script to require more samples)
    • Reducing contamination to 0.02
    • Requiring both an IsolationForest anomaly and a spike to alert

Conclusion and next steps

You now have a lean, portable uptime monitor that goes beyond red/green, using AI to learn your baseline and flag what actually matters. From here:

  • Roll it out to staging, then production

  • Add synthetic transactions (e.g., “login” flow) as additional http targets

  • Ship metrics.jsonl to your time-series or log system for dashboards

  • Extend the Python to deduplicate repeated alerts and tag with deployment info

If this helped, try adding three of your most business-critical endpoints to the targets file and watch the anomaly log for a day. You’ll likely find “degraded-but-not-down” moments that were previously invisible.

Related posts