Posted on
• Artificial Intelligence

Building an Artificial Intelligence Linux Monitoring Dashboard

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Building an Artificial Intelligence Linux Monitoring Dashboard (with Bash, SQLite, and a tiny ML brain)

If you only look at CPU or memory graphs after a production incident, you’re already late. Classic dashboards are great at showing “what happened,” but they rarely warn you when something unusual is about to happen. In this guide, you’ll build a lightweight, local-first Linux monitoring dashboard that adds a dash of AI to spot anomalies in real time—using Bash, SQLite, and a small Python model. No cloud required, no vendors, and it runs entirely on your box.

You’ll get:

  • A Bash-based metrics collector writing to SQLite

  • A tiny AI anomaly detector (Isolation Forest) over your recent metrics

  • A refreshable terminal dashboard you can run with watch or tmux

  • Systemd timer setup to automate collection

The result is a practical, hackable setup that you can extend to fit any Linux environment.

Why add AI to a Linux monitoring dashboard?

  • Early warning: Anomaly detection highlights unusual behavior in context (e.g., moderate CPU + unusual TCP egress might mean exfiltration or runaway processes).

  • Noise reduction: Instead of paging you for every spike, an AI model can separate normal bursts from true outliers.

  • Local-first: Everything runs on your machine. No data leaves the host.

  • Composable: Bash + SQLite + Python gives you transparent code you can tweak, fork, and reason about.

Architecture overview (simple and auditable)

  • Data: Collect CPU, load, memory, disk usage, network throughput, and temperature.

  • Storage: SQLite (fast, durable, easy to query).

  • Intelligence: A small Python script with Isolation Forest to label the latest data point as normal vs anomaly and show the top deviating metrics.

  • Display: A Bash dashboard that pulls both the latest metrics and the AI verdict.

Step 1 — Install prerequisites

You’ll need system tools, SQLite, Python, and jq. Choose your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y sqlite3 python3 python3-venv python3-pip sysstat lm-sensors smartmontools jq iproute2 tmux
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y sqlite python3 python3-virtualenv python3-pip sysstat lm_sensors smartmontools jq iproute tmux
  • openSUSE Leap/Tumbleweed (zypper):
sudo zypper refresh
sudo zypper install -y sqlite3 python3 python3-virtualenv python3-pip sysstat lm_sensors smartmontools jq iproute2 tmux

Optional but recommended:

  • Initialize lm-sensors to read temps:
sudo sensors-detect --auto

Create and activate a virtualenv for Python packages (keeps your system clean):

python3 -m venv ~/.venvs/aimon
source ~/.venvs/aimon/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn numpy

Tip: Add source ~/.venvs/aimon/bin/activate to your shell profile if you plan to hack often.

Step 2 — Create the SQLite DB and a Bash metrics collector

We’ll store metrics in /var/lib/aimon/metrics.db and collect every minute. The collector computes:

  • CPU utilization (from /proc/stat deltas)

  • Load average

  • Memory used/free (from /proc/meminfo)

  • Root filesystem usage (%)

  • Network RX/TX KiB/s (primary interface via ip route)

  • Max temperature (from /sys/class/thermal if available)

Create the collector at /usr/local/bin/aimon-collect.sh:

#!/usr/bin/env bash
set -euo pipefail

DB=${DB:-/var/lib/aimon/metrics.db}
mkdir -p "$(dirname "$DB")"

# Ensure table exists
sqlite3 "$DB" 'CREATE TABLE IF NOT EXISTS metrics (
  ts INTEGER,
  cpu_usage REAL,
  load1 REAL,
  mem_free_mb REAL,
  mem_used_mb REAL,
  disk_root_used_pct REAL,
  net_rx_kib REAL,
  net_tx_kib REAL,
  temp_c REAL
);'

now_ts=$(date +%s)

# Determine primary network interface
IFACE=$(ip route get 1.1.1.1 2>/dev/null | awk '{for (i=1;i<=NF;i++) if ($i=="dev") {print $(i+1); exit}}') || true
if [[ -z "${IFACE:-}" || ! -e "/sys/class/net/$IFACE" ]]; then
  IFACE=$(ls -1 /sys/class/net | grep -v '^lo$' | head -n1 || true)
fi

# Read initial CPU and NET counters
read _ u1 n1 s1 id1 iw1 ir1 si1 st1 g1 gn1 < <(grep '^cpu ' /proc/stat)
t1=$((u1+n1+s1+id1+iw1+ir1+si1+st1))
rx1=0; tx1=0
if [[ -n "${IFACE:-}" && -r "/sys/class/net/$IFACE/statistics/rx_bytes" ]]; then
  rx1=$(<"/sys/class/net/$IFACE/statistics/rx_bytes")
  tx1=$(<"/sys/class/net/$IFACE/statistics/tx_bytes")
fi

sleep 1

# Read second CPU and NET counters
read _ u2 n2 s2 id2 iw2 ir2 si2 st2 g2 gn2 < <(grep '^cpu ' /proc/stat)
t2=$((u2+n2+s2+id2+iw2+ir2+si2+st2))
td=$((t2 - t1))
idd=$((id2 - id1))
cpu_usage=$(awk -v td="$td" -v idd="$idd" 'BEGIN{ if (td<=0) print 0; else printf "%.1f", (1 - idd/td)*100 }')

rx2=$rx1; tx2=$tx1
if [[ -n "${IFACE:-}" && -r "/sys/class/net/$IFACE/statistics/rx_bytes" ]]; then
  rx2=$(<"/sys/class/net/$IFACE/statistics/rx_bytes")
  tx2=$(<"/sys/class/net/$IFACE/statistics/tx_bytes")
fi
rx_kib=$(awk -v d="$((rx2 - rx1))" 'BEGIN{ if (d<0) d=0; printf "%.1f", d/1024 }')
tx_kib=$(awk -v d="$((tx2 - tx1))" 'BEGIN{ if (d<0) d=0; printf "%.1f", d/1024 }')

# Load average (1m)
load1=$(awk '{print $1}' /proc/loadavg)

# Memory
mem_total_kb=$(awk '/MemTotal:/ {print $2}' /proc/meminfo)
mem_avail_kb=$(awk '/MemAvailable:/ {print $2}' /proc/meminfo)
mem_used_mb=$(( (mem_total_kb - mem_avail_kb) / 1024 ))
mem_free_mb=$(( mem_avail_kb / 1024 ))

# Disk usage for /
disk_root_used_pct=$(df -P / | awk 'NR==2 {gsub("%","",$5); print 0+$5}')

# Temperature (max of thermal zones if available)
temp_val="NULL"
max_mill=0
shopt -s nullglob
for z in /sys/class/thermal/thermal_zone*/temp; do
  if [[ -r "$z" ]]; then
    v=$(<"$z")
    if [[ "$v" =~ ^[0-9]+$ ]] && (( v > max_mill )); then
      max_mill=$v
    fi
  fi
done
shopt -u nullglob
if (( max_mill > 0 )); then
  temp_val=$(awk -v m="$max_mill" 'BEGIN{ printf "%.1f", m/1000 }')
fi

# Insert
sqlite3 "$DB" <<SQL
INSERT INTO metrics (ts, cpu_usage, load1, mem_free_mb, mem_used_mb, disk_root_used_pct, net_rx_kib, net_tx_kib, temp_c)
VALUES ($now_ts, $cpu_usage, $load1, $mem_free_mb, $mem_used_mb, $disk_root_used_pct, $rx_kib, $tx_kib, ${temp_val});
SQL

Make it executable and create the data directory:

sudo install -m 0755 -o root -g root /usr/local/bin/aimon-collect.sh /usr/local/bin/aimon-collect.sh
sudo mkdir -p /var/lib/aimon
sudo chown root:root /var/lib/aimon

Test a manual run:

sudo /usr/local/bin/aimon-collect.sh
sqlite3 /var/lib/aimon/metrics.db 'SELECT * FROM metrics ORDER BY ts DESC LIMIT 1;'

Step 3 — Add an AI anomaly detector

We’ll use Isolation Forest to judge if the latest row is “unusual” compared to recent history and also compute per-metric z-scores to explain what drove the anomaly.

Create ~/aimon-anomaly.py:

#!/usr/bin/env python3
import json
import os
import sqlite3
import sys
import math
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest

DB = os.environ.get("DB", "/var/lib/aimon/metrics.db")
WINDOW_SEC = int(os.environ.get("WINDOW_SEC", "86400"))  # last 24h

def main():
    if not os.path.exists(DB):
        print(json.dumps({"error": f"DB not found: {DB}"}))
        sys.exit(0)

    conn = sqlite3.connect(DB)
    q = """
    SELECT ts, cpu_usage, load1, mem_free_mb, mem_used_mb,
           disk_root_used_pct, net_rx_kib, net_tx_kib, temp_c
    FROM metrics
    WHERE ts > strftime('%s','now') - ?
    ORDER BY ts ASC
    """
    df = pd.read_sql_query(q, conn, params=[WINDOW_SEC])
    conn.close()

    if len(df) < 20:
        # not enough history
        out = {"is_anomaly": False, "reason": "insufficient_history", "window_points": int(len(df))}
        print(json.dumps(out))
        return

    features = ["cpu_usage", "load1", "mem_used_mb", "disk_root_used_pct", "net_rx_kib", "net_tx_kib"]
    if df["temp_c"].notna().any():
        features.append("temp_c")

    X = df[features].fillna(method="ffill").fillna(0.0).values
    X_hist, x_now = X[:-1], X[-1:]

    clf = IsolationForest(n_estimators=200, contamination="auto", random_state=42)
    clf.fit(X_hist)
    pred = clf.predict(x_now)[0]  # -1 anomaly, 1 normal
    if_score = float(clf.decision_function(x_now)[0])  # higher is less anomalous

    # z-scores to explain drivers
    hist = df[features].iloc[:-1].astype(float)
    mu = hist.mean()
    sigma = hist.std().replace(0, np.nan)  # avoid div/0
    z = ((df[features].iloc[-1] - mu) / sigma).fillna(0.0)
    top_drivers = list(z.abs().sort_values(ascending=False).head(3).index)

    out = {
        "is_anomaly": bool(pred == -1),
        "iforest_score": round(if_score, 4),
        "top_drivers": top_drivers,
        "z": {k: round(float(v), 2) for k, v in z.to_dict().items()},
        "window_points": int(len(df))
    }
    print(json.dumps(out))

if __name__ == "__main__":
    main()

Make it executable:

chmod +x ~/aimon-anomaly.py

Try it:

source ~/.venvs/aimon/bin/activate
python3 ~/aimon-anomaly.py | jq .

You should see an object with is_anomaly, iforest_score, and top_drivers.

Step 4 — Build a Bash terminal dashboard

This dashboard prints the latest metrics and the AI verdict. Run it with watch to auto-refresh.

Create ~/aimon-dashboard.sh:

#!/usr/bin/env bash
set -euo pipefail

DB=${DB:-/var/lib/aimon/metrics.db}
PY=${PY:-$HOME/aimon-anomaly.py}

latest_row=$(sqlite3 -separator '|' "$DB" \
  "SELECT ts,cpu_usage,load1,mem_used_mb,mem_free_mb,disk_root_used_pct,net_rx_kib,net_tx_kib,temp_c
   FROM metrics ORDER BY ts DESC LIMIT 1;") || true

if [[ -z "${latest_row:-}" ]]; then
  echo "No metrics yet. Run the collector first."
  exit 0
fi

IFS='|' read -r ts cpu load mem_used mem_free disk rx tx temp <<< "$latest_row"
ts_human=$(date -d "@$ts" +"%F %T" 2>/dev/null || date -r "$ts" +"%F %T")

# 10-min averages
avg_row=$(sqlite3 -separator '|' "$DB" \
  "SELECT round(avg(cpu_usage),1), round(avg(load1),2), round(avg(net_rx_kib+net_tx_kib),1)
   FROM metrics WHERE ts > strftime('%s','now') - 600;")
IFS='|' read -r cpu_avg load_avg net_avg <<< "$avg_row"

# AI verdict
ai_json=$(DB="$DB" "$PY" 2>/dev/null || echo '{}')
status=$(echo "$ai_json" | jq -r '.is_anomaly // false')
score=$(echo "$ai_json" | jq -r '.iforest_score // empty')
drivers=$(echo "$ai_json" | jq -r '.top_drivers | join(", ") // empty')
points=$(echo "$ai_json" | jq -r '.window_points // empty')

echo "==============================================================="
echo " AI Linux Monitoring Dashboard - $(hostname) "
echo " Timestamp: $ts_human"
echo "---------------------------------------------------------------"
printf " CPU: %5.1f%%   Load: %-5.2f   Mem Used: %-5s MB   Free: %-5s MB\n" "$cpu" "$load" "$mem_used" "$mem_free"
printf " Disk(/): %3s%%   Net: RX %-6s KiB/s  TX %-6s KiB/s\n" "$disk" "$rx" "$tx"
if [[ -n "${temp:-}" && "$temp" != "NULL" ]]; then
  printf " Temp: %s °C\n" "$temp"
fi
echo "---------------------------------------------------------------"
printf " 10m avg -> CPU: %4s%%   Load: %-5s   Net Total: %-6s KiB/s\n" "$cpu_avg" "$load_avg" "$net_avg"
echo "---------------------------------------------------------------"
if [[ "$status" == "true" ]]; then
  echo " AI: ANOMALY DETECTED   score=$score   top drivers: $drivers   (n=$points)"
else
  echo " AI: OK                  score=$score   (n=$points)"
fi
echo "==============================================================="

Make it executable and try it:

chmod +x ~/aimon-dashboard.sh
watch -n 5 "$HOME/aimon-dashboard.sh"

Open another terminal and generate some load or network traffic to see it react:

yes > /dev/null &
sleep 10
kill %1

Step 5 — Automate collection with systemd

Create a oneshot service and a timer to run the collector every minute.

/etc/systemd/system/aimon-collect.service:

[Unit]
Description=AI Monitor Metrics Collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/aimon-collect.sh

/etc/systemd/system/aimon-collect.timer:

[Unit]
Description=Run AI Monitor Collector every minute

[Timer]
OnBootSec=1min
OnUnitActiveSec=60s
AccuracySec=1s
Unit=aimon-collect.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now aimon-collect.timer
systemctl list-timers | grep aimon-collect

Verify records are coming in:

sqlite3 /var/lib/aimon/metrics.db 'SELECT count(*), min(ts), max(ts) FROM metrics;'

Real-world examples this setup can catch

  • Disk pressure before outages: “OK” CPU but steadily rising disk_root_used_pct and load1 can trigger anomalies hours before an out-of-space crash.

  • Thermal throttling: temp_c spikes paired with dropping CPU utilization = cooling or airflow problem, not software.

  • Silent egress: Low CPU but sustained high net_tx_kib with unusual z-score stands out quickly (think misconfig, backup loop, or worse).

Tips, extensions, and safety

  • Extend features: Add IO wait, per-disk stats (iostat), TCP states, container cgroup metrics, or per-process tallies.

  • Keep it local: This guide never sends data off-host. If you export anywhere, scrub secrets first.

  • Retention: Add a daily DELETE job to keep the DB small (e.g., keep 7–30 days).

  • Visualization: Pipe historical queries into gnuplot, or render SVGs and serve via a tiny Python HTTP server.

  • Multi-host: Push rows over SSH to a central SQLite/Timescale/InfluxDB if you need aggregate views.

Conclusion and next steps (CTA)

You now have a working, local-first monitoring pipeline with a minimal AI brain:

  • Bash script gathers system signals

  • SQLite stores structured history

  • Python flags anomalies and explains why

  • A terminal dashboard surfaces exactly what you need

Suggested next steps:

  • Put the dashboard in tmux and keep it on a side pane for a week

  • Tune features and anomaly thresholds for your workloads

  • Add an alert hook (e.g., send mail or a webhook if is_anomaly=true)

  • Share this with your team and fork it for your environment

When your system starts whispering “something’s off,” you’ll hear it—before it screams.

Related posts