Posted on
• Artificial Intelligence

Artificial Intelligence Virtual Machine Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Virtual Machine Monitoring (for Bash-first Linux users)

Are your VMs “fine”…until they aren’t? Intermittent CPU steal, creeping memory leaks, and noisy-neighbor I/O can go unnoticed until users complain. Traditional threshold alerts (CPU > 80%) are too coarse for today’s dynamic, virtualized fleets. The fix: combine proven Linux observability tools with a light layer of AI-driven anomaly detection so you can catch drift and degradation early—before they become incidents.

This post shows how to build a simple, open-source, Bash-first pipeline:

  • Instrument your VMs with guest-aware metrics and exporters

  • Centralize metrics with Prometheus and visualize with Grafana

  • Add a small AI detector that flags anomalies

  • Automate with systemd so it runs itself

All installation commands include apt, dnf, and zypper where applicable.

Why AI for VM monitoring?

  • Baselines beat static thresholds: VMs behave differently over time and under different tenants. AI models can learn a per-VM “normal,” then surface deviations with fewer false positives.

  • Early warning on subtle issues: Slow memory leaks, increasing I/O wait, and rising CPU steal time may never cross a hard threshold—until it’s too late.

  • Works with what you already use: You don’t need a proprietary platform. Prometheus + Grafana + a small Python model wrapped in Bash is enough to get signal.

1) Instrument your VMs (guest agent, sysstat, exporter)

Install core utilities inside each VM. These commands are safe to run on most Linux guests.

Packages:

  • qemu-guest-agent: guest ↔ host insights (FS freeze/thaw, IP reporting)

  • sysstat: sar/iostat/vmstat for OS-level performance

  • python3/pip: for a small AI script

  • podman (or docker): to run exporters and monitoring stack as containers

  • jq and curl: to glue things together in Bash

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y qemu-guest-agent sysstat python3 python3-venv python3-pip podman jq curl

Fedora/RHEL/CentOS (dnf):

# For RHEL/CentOS, you may need: sudo dnf install -y epel-release
sudo dnf install -y qemu-guest-agent sysstat python3 python3-pip podman jq curl

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y qemu-guest-agent sysstat python3 python3-pip podman jq curl

Enable the guest agent and sysstat:

sudo systemctl enable --now qemu-guest-agent
sudo systemctl enable --now sysstat

Quick sanity checks:

# CPU, run queue:
vmstat 1

# Disk depth, service time:
iostat -xz 1

# Historical stats:
sar -q 1 5

Run a Node Exporter in each VM to expose metrics:

# Runs on host network so it listens on :9100
sudo podman run -d --name node_exporter --net=host --pid=host \
  -v /:/host:ro,rslave \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

(If you prefer Docker, replace podman with docker.)

Tip: If you can’t or don’t want to run a container, install from packages instead:

  • apt: sudo apt install -y prometheus-node-exporter

  • dnf: sudo dnf install -y node_exporter

  • zypper: sudo zypper install -y prometheus-node-exporter

2) Centralize metrics with Prometheus + Grafana (containers)

Create a Prometheus config on your monitoring VM/host:

mkdir -p ~/prom-stack
cat > ~/prom-stack/prometheus.yml <<'YAML'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      # Replace with your VM IPs/hosts
      - targets: ['10.0.0.11:9100','10.0.0.12:9100']
YAML

Run Prometheus and Grafana via containers:

# Prometheus
sudo podman run -d --name prometheus -p 9090:9090 \
  -v ~/prom-stack/prometheus.yml:/etc/prometheus/prometheus.yml:ro \
  -v prometheus-data:/prometheus \
  docker.io/prom/prometheus:latest

# Grafana
sudo podman run -d --name grafana -p 3000:3000 \
  -v grafana-data:/var/lib/grafana \
  docker.io/grafana/grafana-oss:latest

  • Prometheus UI: http://YOUR-MONITOR:9090

  • Grafana UI: http://YOUR-MONITOR:3000 (default admin/admin)

Import community dashboards for Node Exporter to get CPU, memory, disk, and network overviews fast.

Useful PromQL examples you’ll use below:

# CPU usage %
100 - (avg by (instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage %
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Disk read latency (ms) per device
(rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])) * 1000

# CPU steal time %
avg by (instance)(irate(node_cpu_seconds_total{mode="steal"}[5m]) * 100)

3) Add a lightweight AI anomaly detector

We’ll pull time series from Prometheus, train a simple Isolation Forest model, and flag anomalies in near-real time.

Create a Python virtual environment and install dependencies:

python3 -m venv ~/ai-vm
source ~/ai-vm/bin/activate
pip install --upgrade pip
pip install scikit-learn pandas requests

Save this script as ai_vm_anomaly.py:

#!/usr/bin/env python3
import os, sys, time, requests, pandas as pd
from datetime import datetime, timedelta, timezone
from sklearn.ensemble import IsolationForest

PROM = os.environ.get("PROM_URL", "http://127.0.0.1:9090")
INSTANCE = os.environ.get("INSTANCE")
WINDOW_MIN = int(os.environ.get("WINDOW_MIN", "360"))  # 6 hours
STEP = os.environ.get("STEP", "60s")
CONTAM = float(os.environ.get("CONTAM", "0.02"))       # ~2% anomalies

if not INSTANCE:
    print("ERROR: set INSTANCE, e.g. export INSTANCE='10.0.0.11:9100'", file=sys.stderr)
    sys.exit(2)

def q_range(expr, start, end, step):
    r = requests.get(f"{PROM}/api/v1/query_range", params={"query": expr, "start": start, "end": end, "step": step}, timeout=20)
    r.raise_for_status()
    data = r.json()["data"]["result"]
    if not data:
        return pd.Series([], dtype=float)
    # Use the first series (group by instance in expr keeps single series)
    ts = data[0]["values"]
    idx = pd.to_datetime([float(t[0]) for t in ts], unit="s", utc=True)
    vals = [float(t[1]) for t in ts]
    return pd.Series(vals, index=idx)

now = datetime.now(timezone.utc)
start = now - timedelta(minutes=WINDOW_MIN)

# PromQL expressions (per instance)
cpu_expr = f'100 - (avg by (instance)(irate(node_cpu_seconds_total{{instance="{INSTANCE}",mode="idle"}}[5m]))*100)'
mem_expr = f'100 * (1 - (node_memory_MemAvailable_bytes{{instance="{INSTANCE}"}} / node_memory_MemTotal_bytes{{instance="{INSTANCE}"}}))'
disk_lat_expr = f'(rate(node_disk_read_time_seconds_total{{instance="{INSTANCE}"}}[5m]) / rate(node_disk_reads_completed_total{{instance="{INSTANCE}"}}[5m])) * 1000'

cpu = q_range(cpu_expr, int(start.timestamp()), int(now.timestamp()), STEP)
mem = q_range(mem_expr, int(start.timestamp()), int(now.timestamp()), STEP)
lat = q_range(disk_lat_expr, int(start.timestamp()), int(now.timestamp()), STEP)

df = pd.DataFrame({"cpu": cpu, "mem": mem, "lat": lat}).dropna()
if df.empty or len(df) < 30:
    print("WARN: not enough data yet")
    sys.exit(0)

# Train unsupervised on the window
model = IsolationForest(contamination=CONTAM, random_state=42)
model.fit(df)

scores = model.decision_function(df)           # higher is more normal
pred = model.predict(df)                       # -1 = anomaly, 1 = normal

last_is_anom = pred[-1] == -1
last_score = scores[-1]
last_row = df.iloc[-1]

status = "ANOMALY" if last_is_anom else "OK"
print(f"{status} instance={INSTANCE} cpu={last_row.cpu:.1f}% mem={last_row.mem:.1f}% rlat={last_row.lat:.1f}ms score={last_score:.4f}")
sys.exit(1 if last_is_anom else 0)

Example run:

export PROM_URL="http://10.0.0.50:9090"
export INSTANCE="10.0.0.11:9100"
~/ai-vm/bin/python ~/ai-vm/ai_vm_anomaly.py

Output looks like:

OK instance=10.0.0.11:9100 cpu=21.3% mem=58.9% rlat=4.2ms score=0.0312

or

ANOMALY instance=10.0.0.11:9100 cpu=7.5% mem=97.2% rlat=85.7ms score=-0.1123

Wrap it in Bash for alerts or self-healing:

#!/usr/bin/env bash
set -euo pipefail

PROM_URL="${PROM_URL:-http://10.0.0.50:9090}"
INSTANCE="${1:-10.0.0.11:9100}"

export PROM_URL INSTANCE
OUT="$("$HOME/ai-vm/bin/python" "$HOME/ai-vm/ai_vm_anomaly.py" || true)"
echo "$(date -Is) $OUT"

if grep -q '^ANOMALY ' <<<"$OUT"; then
  # Example follow-up: gather quick triage data
  echo "--- vmstat snapshot ---"
  vmstat 1 3 || true
  echo "--- iostat snapshot ---"
  iostat -xz 1 3 || true

  # Optional: notify (mail, Slack webhook, etc.)
  # curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$OUT\"}" https://hooks.slack.com/services/XXX/YYY/ZZZ
fi

4) Automate with systemd timers

Create a systemd service to run the detector every 2 minutes.

Service unit:

sudo tee /etc/systemd/system/ai-vm-watch@.service >/dev/null <<'UNIT'
[Unit]
Description=AI VM Watch for %i

[Service]
Type=oneshot
Environment=HOME=%h
Environment=PROM_URL=http://10.0.0.50:9090
User=%i
ExecStart=/bin/bash -lc '$HOME/ai-vm/bin/python $HOME/ai-vm/ai_vm_anomaly.py'
UNIT

Timer unit:

sudo tee /etc/systemd/system/ai-vm-watch@.timer >/dev/null <<'UNIT'
[Unit]
Description=Run AI VM Watch for %i every 2 minutes

[Timer]
OnBootSec=2m
OnUnitActiveSec=2m
AccuracySec=15s
Persistent=true

[Install]
WantedBy=timers.target
UNIT

Enable for a given user or service account that owns the venv:

# Example: run as user 'monitor'
sudo systemctl enable --now ai-vm-watch@monitor.timer
sudo systemctl list-timers | grep ai-vm-watch
journalctl -u ai-vm-watch@monitor.service -f

If you prefer cron:

*/2 * * * * PROM_URL=http://10.0.0.50:9090 INSTANCE=10.0.0.11:9100 /home/monitor/ai-vm/bin/python /home/monitor/ai-vm/ai_vm_anomaly.py >> /var/log/ai-vm-watch.log 2>&1

Real-world signals to watch (and why)

  • CPU steal rising: Indicates hypervisor contention. PromQL:

    avg by (instance)(irate(node_cpu_seconds_total{mode="steal"}[5m]) * 100)

    If AI flags sudden steal spikes, migrate or rebalance tenants.

  • Slow memory leak: Gradual increase in memory usage with unchanged workload usually means a leaking service. AI sees the trend change well before OOM.

  • Read latency spikes: Even with low throughput, rising node_disk_read_time_seconds_total / reads_completed_total points to backend storage issues.

  • Network retransmits: If you expose ethtool or use exporters that capture TCP stats, anomaly spikes reveal saturation or bad NIC offload configs.

Put it all together: 3–5 actionable steps

1) Install the basics on every VM:

# apt
sudo apt install -y qemu-guest-agent sysstat podman python3 python3-pip jq curl
# dnf
sudo dnf install -y qemu-guest-agent sysstat podman python3 python3-pip jq curl
# zypper
sudo zypper install -y qemu-guest-agent sysstat podman python3 python3-pip jq curl

Enable services and run Node Exporter.

2) Stand up Prometheus + Grafana with containers:

sudo podman run -d --name prometheus -p 9090:9090 -v ~/prom-stack/prometheus.yml:/etc/prometheus/prometheus.yml:ro -v prometheus-data:/prometheus docker.io/prom/prometheus
sudo podman run -d --name grafana -p 3000:3000 -v grafana-data:/var/lib/grafana docker.io/grafana/grafana-oss

3) Add the AI detector:

  • Create venv, install scikit-learn, pandas, requests.

  • Use ai_vm_anomaly.py against Prometheus to score current VM state.

  • Wire alerts in Bash.

4) Automate:

  • Use systemd timers or cron.

  • Start with a low anomaly rate (contamination ~2%), then tune.

5) Iterate:

  • Add more signals (I/O wait, TCP retransmits, application latency).

  • Label true/false positives and adjust contamination or add per-metric models.

Conclusion and next steps

You don’t need a heavyweight platform to get AI-powered VM monitoring. With a few packages, containers, and a short Python script orchestrated by Bash, you can detect issues earlier and respond faster.

Your next step:

  • Pick two VMs in staging.

  • Deploy Node Exporter, Prometheus, and Grafana.

  • Drop in the AI detector, run it for a week, and review alerts vs reality.

  • Tune, then roll out across your fleet.

Got stuck or want a deeper, fully Bash-only pipeline? Document your environment (distro, hypervisor, constraints), and I’ll help you tailor the commands and PromQL to your setup.

Related posts