AI-Driven Cloud Monitoring from the Bash Prompt

Cloud apps scale faster than humans can watch dashboards. Nighttime traffic spikes, noisy alerts, rogue autoscaling, and stealthy memory leaks all conspire to hide real incidents until users complain. What if your monitoring could learn normal behavior and raise its hand only when something is truly off?

In this article, you’ll build a vendor-neutral, AI-assisted monitoring workflow that runs on any Linux cloud VM. It’s simple, scriptable, and friendly to Bash-first teams. You’ll collect telemetry, perform anomaly detection, forecast capacity, and push enriched alerts to chat—no heavy agents required.

Why AI for Cloud Monitoring?

• Scale and complexity: Microservices + ephemeral nodes = too many signals to handle manually.

• Signal-to-noise: Traditional static thresholds either spam or miss novel failures.

• Cost and performance: Early detection of anomalies and capacity trends prevents outages and waste.

• Time-to-diagnosis: Enriched, deduplicated alerts reduce pager fatigue and MTTR.

AI excels at learning baselines, spotting outliers, and projecting the near future. Combined with Bash and Linux-native telemetry, you get a pragmatic stack that works anywhere.

What you’ll build

• A tiny, cron-able collector that emits CPU, memory, disk, and network metrics to CSV.

• An anomaly detector using Isolation Forest (scikit-learn) to flag unusual behavior.

• A lightweight capacity forecaster to warn before you hit CPU or disk limits.

• A Bash wrapper that enriches and sends alerts to Slack (or any webhook), with dedupe.

All code runs locally on your Linux VM/container and can be rolled out fleet-wide with your favorite automation.

---

Prerequisites and installation

Install core tools (curl, jq, Python 3, and pip). Use the package manager for your distro:

• Debian/Ubuntu (apt):

  sudo apt update
  sudo apt install -y curl jq python3 python3-pip
  

• Fedora/RHEL/CentOS Stream (dnf):

  sudo dnf install -y curl jq python3 python3-pip
  

• openSUSE/SLES (zypper):

  sudo zypper refresh
  sudo zypper install -y curl jq python3 python3-pip
  

Upgrade pip and install the Python libraries we’ll use:

python3 -m pip install --user --upgrade pip
python3 -m pip install --user numpy pandas scikit-learn

Optional: Use a virtual environment instead of --user:

python3 -m venv ~/.venvs/aimon
source ~/.venvs/aimon/bin/activate
pip install --upgrade pip
pip install numpy pandas scikit-learn

---

Step 1: Collect and normalize telemetry (Bash-only)

Create a minimal collector that samples system health and appends to CSV. This script:

• Measures CPU busy over 1 second

• Calculates memory used percent

• Reads root disk utilization

• Aggregates network RX/TX bytes (excluding loopback)

Save as collector.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

OUT_FILE="${1:-/var/log/aimon/metrics.csv}"
OUT_DIR="$(dirname "$OUT_FILE")"
mkdir -p "$OUT_DIR"

if [[ ! -f "$OUT_FILE" ]]; then
  echo "timestamp,cpu_busy,mem_used_percent,disk_used_percent,net_rx_bytes,net_tx_bytes" > "$OUT_FILE"
fi

ts="$(date -u +%FT%TZ)"

# CPU busy% over ~1 second
r1="$(grep '^cpu ' /proc/stat)"
sleep 1
r2="$(grep '^cpu ' /proc/stat)"
cpu_busy="$(printf "%s\n%s\n" "$r1" "$r2" | awk '
  NR==1{for(i=2;i<=NF;i++) a[i]=$i}
  NR==2{for(i=2;i<=NF;i++) b[i]=$i}
  END{
    for(i=2;i in a;i++){t1+=a[i]; t2+=b[i]}
    idle1=a[5]+a[6]; idle2=b[5]+b[6]
    dt=t2-t1; di=idle2-idle1
    if (dt>0) printf "%.2f", (dt-di)*100/dt; else printf "0.00"
  }')"

# Memory used %
mem_used="$(awk '
  /MemTotal:/ {t=$2}
  /MemAvailable:/ {a=$2}
  END{ if(t>0) printf "%.2f", (1 - a/t)*100; else printf "0.00" }
' /proc/meminfo)"

# Disk used % on /
disk_used="$(df -P / | awk 'NR==2 {gsub("%","",$5); print $5}')"

# Network RX/TX bytes (sum over all non-lo interfaces)
read rx tx < <(awk -F'[: ]+' 'NR>2 && $1!="lo"{rx+=$3; tx+=$11} END{print rx,tx}' /proc/net/dev)

echo "$ts,$cpu_busy,$mem_used,$disk_used,$rx,$tx" >> "$OUT_FILE"

Usage:

sudo ./collector.sh /var/log/aimon/metrics.csv

Schedule it every minute (cron):

( sudo crontab -l 2>/dev/null; echo '* * * * * /usr/local/bin/collector.sh /var/log/aimon/metrics.csv' ) | sudo crontab -

Tip: You can also use a systemd timer for tighter control, but cron works fine for a first pass.

---

Step 2: Detect anomalies with Isolation Forest (AI)

We’ll train an Isolation Forest on a rolling window of recent metrics and flag the latest sample if it looks unusual compared to the baseline.

Save as anomaly_iforest.py:

#!/usr/bin/env python3
import argparse, json, sys
import pandas as pd
from sklearn.ensemble import IsolationForest

parser = argparse.ArgumentParser()
parser.add_argument("--csv", required=True, help="Path to metrics.csv")
parser.add_argument("--window", type=int, default=200, help="Number of recent rows to learn from")
args = parser.parse_args()

try:
    df = pd.read_csv(args.csv)
except Exception as e:
    print(json.dumps({"error": f"failed_to_read_csv: {e}"}))
    sys.exit(1)

if len(df) < max(50, args.window // 2):
    print(json.dumps({"ready": False, "reason": "not_enough_data", "rows": len(df)}))
    sys.exit(0)

window_df = df.tail(args.window).copy()
features = ["cpu_busy","mem_used_percent","disk_used_percent","net_rx_bytes","net_tx_bytes"]
X = window_df[features].values

model = IsolationForest(contamination="auto", random_state=42)
model.fit(X)

pred = model.predict(X)              # 1 = normal, -1 = anomaly
score = model.decision_function(X)   # higher = more normal, negative = more anomalous

last = window_df.iloc[-1]
is_anom = (pred[-1] == -1)
result = {
    "ready": True,
    "timestamp": str(last["timestamp"]),
    "anomaly": bool(is_anom),
    "decision_score": float(score[-1]),
    "features": {k: float(last[k]) for k in features}
}
print(json.dumps(result))

Install dependencies if you haven’t yet:

python3 -m pip install --user numpy pandas scikit-learn

Now create a tiny Bash wrapper that runs the analysis right after collection and sends a Slack message if an anomaly is detected.

Save as analyze.sh:

#!/usr/bin/env bash
set -euo pipefail

CSV="${1:-/var/log/aimon/metrics.csv}"
WEBHOOK="${SLACK_WEBHOOK_URL:-}"
HOST_TAG="${HOST_TAG:-$(hostname -f)}"
DEDUP_FILE="/tmp/aimon_last_alert"
MIN_INTERVAL_SEC="${MIN_INTERVAL_SEC:-900}"  # 15 minutes

# Run the model
out="$(python3 /usr/local/bin/anomaly_iforest.py --csv "$CSV" --window 200 || true)"
echo "$out" | jq . >/dev/null 2>&1 || { echo "$out"; exit 0; }

ready="$(echo "$out" | jq -r '.ready // false')"
anomaly="$(echo "$out" | jq -r '.anomaly // false')"

if [[ "$ready" != "true" || "$anomaly" != "true" ]]; then
  exit 0
fi

now=$(date +%s)
if [[ -f "$DEDUP_FILE" ]]; then
  last=$(cat "$DEDUP_FILE" || echo 0)
else
  last=0
fi
if (( now - last < MIN_INTERVAL_SEC )); then
  # Skip duplicate alerts within the window
  exit 0
fi

cpu=$(echo "$out" | jq -r '.features.cpu_busy')
mem=$(echo "$out" | jq -r '.features.mem_used_percent')
disk=$(echo "$out" | jq -r '.features.disk_used_percent')
score=$(echo "$out" | jq -r '.decision_score')

# Top processes for quick triage
top_procs="$(ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head -n 6 | tail -n +2 | sed 's/"/\"/g')"

msg="ANOMALY on ${HOST_TAG}
score=${score}
cpu_busy=${cpu}%  mem_used=${mem}%  disk_used=${disk}%
top_cpu:
${top_procs}
runbook: set SLACK_WEBHOOK_URL and tune thresholds/window if needed."

if [[ -n "$WEBHOOK" ]]; then
  curl -sS -X POST -H 'Content-type: application/json' \
    --data "$(jq -n --arg text "$msg" '{text: $text}')" \
    "$WEBHOOK" >/dev/null || true
else
  echo "[INFO] SLACK_WEBHOOK_URL not set; printing alert locally"
  echo "$msg"
fi

echo "$now" > "$DEDUP_FILE"

Make scripts executable and place them in your PATH:

sudo install -m 0755 collector.sh /usr/local/bin/collector.sh
sudo install -m 0755 anomaly_iforest.py /usr/local/bin/anomaly_iforest.py
sudo install -m 0755 analyze.sh /usr/local/bin/analyze.sh

Chain collection + analysis in cron:

( sudo crontab -l 2>/dev/null; \
  echo '* * * * * /usr/local/bin/collector.sh /var/log/aimon/metrics.csv && /usr/local/bin/analyze.sh /var/log/aimon/metrics.csv' \
) | sudo crontab -

Export your Slack webhook:

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"

---

Step 3: Forecast capacity to prevent incidents

Use a simple linear trend to forecast CPU and disk. If the future crosses a threshold, alert proactively (e.g., scale up or expand disk).

Save as forecast_capacity.py:

#!/usr/bin/env python3
import argparse, json, sys
import pandas as pd
import numpy as np

parser = argparse.ArgumentParser()
parser.add_argument("--csv", required=True)
parser.add_argument("--lookback", type=int, default=180)   # minutes
parser.add_argument("--horizon", type=int, default=60)     # minutes ahead
parser.add_argument("--cpu_thresh", type=float, default=80.0)
parser.add_argument("--disk_thresh", type=float, default=85.0)
args = parser.parse_args()

try:
    df = pd.read_csv(args.csv)
except Exception as e:
    print(json.dumps({"error": f"read_csv_failed: {e}"})); sys.exit(1)

if len(df) < args.lookback:
    print(json.dumps({"ready": False, "reason": "not_enough_data", "rows": len(df)})); sys.exit(0)

w = df.tail(args.lookback).reset_index(drop=True)
x = np.arange(len(w))

alerts = []

def lin_forecast(series, ahead):
    y = series.values.astype(float)
    # robust to flat lines
    if np.all(y == y[0]):
        return y[-1]
    a, b = np.polyfit(x, y, 1)  # y = a*x + b
    return a * (len(w) + ahead) + b

cpu_f = lin_forecast(w["cpu_busy"], args.horizon)
disk_f = lin_forecast(w["disk_used_percent"], args.horizon)  # simplistic; disk usually monotonic

if cpu_f >= args.cpu_thresh:
    alerts.append({"signal": "cpu_busy", "forecast": round(float(cpu_f),2), "action": "scale_up | optimize workload"})

# For disk, forecast further out (e.g., 1 day) because it grows slower
disk_day_f = lin_forecast(w["disk_used_percent"], 1440)
if disk_day_f >= args.disk_thresh:
    alerts.append({"signal": "disk_used_percent", "forecast_day": round(float(disk_day_f),2), "action": "expand_volume | cleanup"})

print(json.dumps({
    "ready": True,
    "alerts": alerts,
    "latest": {
        "cpu_busy": float(w["cpu_busy"].iloc[-1]),
        "disk_used_percent": float(w["disk_used_percent"].iloc[-1])
    }
}))

Run it hourly via cron and post to Slack if an action is recommended:

#!/usr/bin/env bash
set -euo pipefail
CSV="/var/log/aimon/metrics.csv"
WEBHOOK="${SLACK_WEBHOOK_URL:-}"

out="$(python3 /usr/local/bin/forecast_capacity.py --csv "$CSV" --lookback 360 --horizon 60 || true)"
cnt="$(echo "$out" | jq '.alerts | length')"
if [[ "$cnt" -gt 0 ]]; then
  if [[ -n "$WEBHOOK" ]]; then
    curl -sS -X POST -H 'Content-type: application/json' \
      --data "$(jq -n --arg t "Capacity forecast on $(hostname -f): $out" '{text: $t}')" \
      "$WEBHOOK" >/dev/null || true
  else
    echo "$out"
  fi
fi

Add to cron (hourly):

( sudo crontab -l 2>/dev/null; echo '0 * * * * /usr/local/bin/forecast_capacity.py --csv /var/log/aimon/metrics.csv --lookback 360 --horizon 60 | logger -t aimon-forecast' ) | sudo crontab -

---

Step 4: Enrich and deduplicate alerts

You already saw a simple dedupe window in analyze.sh. Add lightweight enrichment so responders get context immediately:

• Include top CPU/memory processes (already added).

• Attach recent deltas: network spikes are clearer with per-minute change. You can extend collector.sh to compute deltas by storing last counters in /var/tmp/aimon_state.

• Tag alerts with metadata (HOST_TAG, SERVICE, ENV) via environment variables and include them in the Slack message.

Example: set tags in /etc/default/aimon and source it in your scripts:

HOST_TAG="cart-api-3.use1a"
SERVICE="cart"
ENV="prod"
export HOST_TAG SERVICE ENV

Then add to message body to group-by in chat.

---

Step 5: Real-world examples

• E-commerce spike: Within minutes of a surprise campaign, CPU and network anomalies fire with top processes listed, letting you add capacity before cart latency degrades.

• Memory leak in a rollout: Isolation Forest flags a pattern-break in memory vs historical baselines; you roll back just one pod/node instead of paging the whole team later.

• Disk growth: Linear trend predicts the root volume will breach 85% within 24 hours—ops extends the volume during business hours.

---

Notes, tuning, and next steps

• Tuning: Adjust --window, --lookback, and thresholds. Too many alerts? Increase dedupe window, decrease contamination, or remove noisy features.

• Training data: More (clean) history yields better baselines. Let the system warm up after deploys before acting on anomalies.

• Security: Store webhooks as secrets. Limit script permissions and log scrubbing.

• Observability stack integration:

  • Emit Prometheus text format and scrape with Prometheus.
  • Use OpenTelemetry Collector to unify metrics/logs/traces.
  • Ship anomalies into your incident system (PagerDuty, Opsgenie) via webhook.

• More AI: Try seasonal models (Holt-Winters), Prophet, or tree-based regressors for richer forecasts. For logs, cluster similar errors with TF-IDF + k-means to reduce noise.

---

Conclusion and Call to Action

You now have a Bash-first, AI-assisted monitoring loop:

• Collect normalized telemetry

• Detect anomalies automatically

• Forecast capacity to get ahead of incidents

• Enrich and dedupe alerts for faster resolution

Start by deploying collector.sh and analyze.sh on one production node today. Once you’re happy with the signal, roll it out across your fleet and wire alerts into your on-call workflow. From there, iterate: add features, improve models, and integrate with your existing Prometheus/Grafana or OpenTelemetry pipelines.

If you want a follow-up post with Prometheus scraping + AI scoring, or OpenTelemetry Collector configs, tell me which stack you use and I’ll tailor the next guide.