Build an AI-augmented Network Dashboard on Linux with Bash, Prometheus, and Grafana

Ever stared at a wall of network graphs and thought, “So… what’s actually wrong?” Traditional dashboards flood you with metrics but rarely surface the needle in the haystack. Artificial Intelligence Network Dashboards flip that dynamic: they highlight anomalies, predict trouble before it hits, and guide you straight to the signal.

In this article, you’ll learn how to stand up a lightweight, production-ready dashboard that adds AI-style anomaly scoring to your Linux network telemetry—using familiar tools (Bash, Prometheus, Grafana) and a tiny Python helper. No heavyweight ML infrastructure required.

The problem and the value

• Problem: Networks generate more telemetry than humans can digest. Spikes, microbursts, and subtle degradations hide in plain sight, leading to long MTTR and missed early warnings.

• Value: AI-augmented dashboards turn raw metrics into “attention guidance.” You get:

  • Automated anomaly scoring on throughput/latency
  • Fast visual cues and alert thresholds
  • Simple plumbing that fits Linux, Bash, and your existing monitoring stack

What is an “AI Network Dashboard”?

It’s a standard dashboard (timeseries panels, alerts) enhanced with machine-learning or statistical detection that:

• Flags unusual behavior based on recent baselines, not just static thresholds

• Surfaces an “anomaly score” so you know when to dig deeper

• Can incorporate predictions over time (capacity, saturation risk)

We’ll implement a compact, robust version using a rolling baseline and z-score anomaly detection. It’s simple, explainable, and deploys anywhere.

---

Step 1: Install the stack (Prometheus, Node Exporter, Grafana, Python)

We’ll collect metrics with Prometheus Node Exporter, compute a small “AI” anomaly score with a Python script, and visualize in Grafana.

Run the set that matches your distro:

• Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y prometheus prometheus-node-exporter grafana \
  python3 python3-venv python3-pip vnstat jq curl git

• Fedora/RHEL/CentOS Stream (dnf)

sudo dnf -y install prometheus node_exporter grafana \
  python3 python3-virtualenv python3-pip vnstat jq curl git

• openSUSE Leap/Tumbleweed (zypper)

sudo zypper refresh
sudo zypper install -y prometheus golang-github-prometheus-node_exporter grafana \
  python3 python3-virtualenv python3-pip vnstat jq curl git

Enable and start services:

• Prometheus

sudo systemctl enable --now prometheus

• Node Exporter

# Service name differs by distro; run the one that exists for you:
sudo systemctl enable --now prometheus-node-exporter || \
sudo systemctl enable --now node_exporter

• Grafana

sudo systemctl enable --now grafana-server

• vnstat (optional but useful baseline)

sudo systemctl enable --now vnstat

Quick-check:

# Node Exporter should listen on :9100
ss -lntp | grep 9100

# Prometheus should listen on :9090
ss -lntp | grep 9090

# Grafana should listen on :3000
ss -lntp | grep 3000

Note: Grafana default login is admin/admin (you’ll be prompted to change it).

---

Step 2: Enable Node Exporter’s textfile collector

We’ll publish our anomaly score as a Prometheus metric via Node Exporter’s textfile collector.

1) Create a directory for custom metrics:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown -R root:root /var/lib/node_exporter/textfile_collector

2) Tell Node Exporter to read from that directory (systemd override):

# Determine the node_exporter binary path
command -v node_exporter || echo "Adjust the ExecStart path below to your system"

# Open an override editor
sudo systemctl edit node_exporter || sudo systemctl edit prometheus-node-exporter

Paste the following, adjusting the ExecStart path if needed (often /usr/bin/node_exporter):

[Service]
ExecStart=
ExecStart=/usr/bin/node_exporter --collector.textfile \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Then:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter || sudo systemctl restart prometheus-node-exporter

Verify the textfile collector is active:

curl -s localhost:9100/metrics | grep node_textfile

You should see something like node_textfile_mtime_seconds.

---

Step 3: Add a tiny “AI” anomaly detector

We’ll compute per-minute network throughput and a rolling z-score that flags unusual spikes. The script:

• Reads /proc/net/dev for your default interface

• Keeps a small rolling window on disk

• Writes Prometheus metrics into the textfile collector directory

Create a workspace and virtual environment:

sudo mkdir -p /opt/ai-netdash
sudo chown -R $USER:$USER /opt/ai-netdash
cd /opt/ai-netdash

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
# No external libs required; stdlib only for this example
deactivate

Create the script:

cat > /opt/ai-netdash/ai_net_anomaly.py <<'PYEOF'
#!/usr/bin/env python3
import json, os, time, re, statistics, subprocess

STATE_DIR = "/var/lib/ai-netdash"
STATE_FILE = os.path.join(STATE_DIR, "state.json")
OUT_DIR = "/var/lib/node_exporter/textfile_collector"
OUT_FILE = os.path.join(OUT_DIR, "ai_netdash.prom")
WINDOW = 60  # keep last 60 samples (e.g., 60 minutes if run via cron per minute)

def get_default_iface():
    # Parse default route
    try:
        out = subprocess.check_output(["ip", "route"], text=True)
        for line in out.splitlines():
            if line.startswith("default "):
                m = re.search(r"dev\s+(\S+)", line)
                if m: return m.group(1)
    except Exception:
        pass
    # Fallback to first non-loopback iface in /proc/net/dev
    with open("/proc/net/dev") as f:
        for line in f:
            if ":" in line:
                iface = line.split(":")[0].strip()
                if iface != "lo": return iface
    return "eth0"

def read_counters(iface):
    with open("/proc/net/dev") as f:
        for line in f:
            if line.strip().startswith(iface + ":"):
                parts = line.split(":")[1].split()
                rx_bytes = int(parts[0])
                tx_bytes = int(parts[8])
                return rx_bytes, tx_bytes
    raise RuntimeError(f"Interface {iface} not found in /proc/net/dev")

def load_state():
    if not os.path.exists(STATE_FILE):
        return {"samples": [], "last": None, "iface": None}
    with open(STATE_FILE) as f:
        return json.load(f)

def save_state(state):
    tmp = STATE_FILE + ".tmp"
    with open(tmp, "w") as f:
        json.dump(state, f)
    os.replace(tmp, STATE_FILE)

def zscore(series, value):
    if len(series) < 10:
        return 0.0  # not enough history
    mean = statistics.mean(series)
    stdev = statistics.pstdev(series) or 1.0
    return (value - mean) / stdev

def main():
    os.makedirs(STATE_DIR, exist_ok=True)

    iface = get_default_iface()
    now = time.time()
    rx, tx = read_counters(iface)

    state = load_state()
    last = state.get("last")

    if last and last["iface"] == iface:
        dt = max(1, now - last["ts"])
        rx_bps = (rx - last["rx_bytes"]) / dt
        tx_bps = (tx - last["tx_bytes"]) / dt
        total_bps = max(0.0, rx_bps + tx_bps)
    else:
        rx_bps = tx_bps = total_bps = 0.0  # first run

    # Update rolling window
    samples = state.get("samples", [])
    samples.append(total_bps)
    samples = samples[-WINDOW:]

    score = zscore(samples[:-1] if len(samples)>1 else samples, total_bps)

    # Save state
    state = {
        "samples": samples,
        "last": {"ts": now, "rx_bytes": rx, "tx_bytes": tx, "iface": iface},
        "iface": iface
    }
    save_state(state)

    # Write Prometheus textfile metrics
    os.makedirs(OUT_DIR, exist_ok=True)
    lines = []
    lines.append('# HELP ai_netdash_bytes_per_sec_rx Estimated receive throughput (bytes/sec)')
    lines.append('# TYPE ai_netdash_bytes_per_sec_rx gauge')
    lines.append(f'ai_netdash_bytes_per_sec_rx{{iface="{iface}"}} {rx_bps:.3f}')
    lines.append('# HELP ai_netdash_bytes_per_sec_tx Estimated transmit throughput (bytes/sec)')
    lines.append('# TYPE ai_netdash_bytes_per_sec_tx gauge')
    lines.append(f'ai_netdash_bytes_per_sec_tx{{iface="{iface}"}} {tx_bps:.3f}')
    lines.append('# HELP ai_netdash_anomaly_score Z-score of total throughput vs rolling baseline')
    lines.append('# TYPE ai_netdash_anomaly_score gauge')
    lines.append(f'ai_netdash_anomaly_score{{iface="{iface}"}} {score:.3f}')
    lines.append('# HELP ai_netdash_total_bytes_per_sec Total throughput (bytes/sec)')
    lines.append('# TYPE ai_netdash_total_bytes_per_sec gauge')
    lines.append(f'ai_netdash_total_bytes_per_sec{{iface="{iface}"}} {total_bps:.3f}')

    tmp = OUT_FILE + ".tmp"
    with open(tmp, "w") as f:
        f.write("\n".join(lines) + "\n")
    os.replace(tmp, OUT_FILE)

if __name__ == "__main__":
    main()
PYEOF

sudo chmod +x /opt/ai-netdash/ai_net_anomaly.py
sudo mkdir -p /var/lib/ai-netdash

Test it:

sudo /opt/ai-netdash/ai_net_anomaly.py
curl -s localhost:9100/metrics | grep ai_netdash

You should see your new metrics. On first run, values may be 0 until a second sample is taken.

Schedule it via cron (once per minute is a good start):

( sudo crontab -l 2>/dev/null; echo "*/1 * * * * /opt/ai-netdash/ai_net_anomaly.py" ) | sudo crontab -

Tip: Prefer a systemd timer for more robust scheduling, but cron is quick and portable.

---

Step 4: Wire Prometheus and Grafana

Point Prometheus at Node Exporter (if not already). Minimal config:

sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak

sudo bash -c 'cat > /etc/prometheus/prometheus.yml' <<'YAML'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]
YAML

sudo systemctl restart prometheus

In Grafana (http://localhost:3000): 1) Add Data Source -> Prometheus -> URL: http://localhost:9090 -> Save & Test 2) Create Dashboard -> Add a panel - Query for anomaly score: ai_netdash_anomaly_score - Set a threshold line at 3.0 (common “unexpected” z-score) 3) Add panels for throughput: ai_netdash_bytes_per_sec_rx ai_netdash_bytes_per_sec_tx ai_netdash_total_bytes_per_sec 4) Create a simple alert (Grafana Alerting or Prometheus alert rules) when anomaly score >= 3 for N minutes.

Example PromQL for alert rule:

ai_netdash_anomaly_score >= 3

---

Step 5: Real-world examples you can replicate

• Sudden DNS spike:

  • Generate traffic: dig @8.8.8.8 google.com +repeat=2000
  • Watch anomaly score spike above 3 as total throughput departs from baseline.

• Link saturation risk during backups:

  • Kick off a large rsync. The dashboard shows rising throughput; anomaly score highlights unexpected time-of-day behavior.

• East-west chatter hinting at lateral movement:

  • If unusual traffic bursts occur off-hours, the anomaly score panel lights up while your baseline panels may look “normal” at a glance.

---

Going further

• Multi-signal models: Include packet rate, retransmissions (ss -s), errors/drops (ethtool -S), and build a composite anomaly score.

• Blackbox latency: Install Prometheus Blackbox Exporter to probe HTTP/DNS/ICMP and add z-score on latency.

• ML libraries: If you want heavier ML (Isolation Forest, Prophet, ONNX), install them into the venv and swap the z-score with your model’s output.

Example: install scikit-learn into the venv:

cd /opt/ai-netdash
source .venv/bin/activate
pip install scikit-learn
deactivate

Then modify the script to fit your model, still writing metrics via the textfile collector.

---

Why this approach works

• Lightweight: Pure Linux primitives + Prometheus/Grafana you likely already use.

• Explainable: Z-scores provide “why” an alert fired (deviation from baseline) without black-box opacity.

• Extensible: Swap in advanced models when needed—keep the plumbing the same.

---

Conclusion and Call to Action

You just built an AI-augmented network dashboard that elevates anomalies, not just metrics. From here:

• Tune the rolling window and threshold for your environment

• Add panels and alerts for the signals you care about most

• Iterate: plug in richer features (errors, latency) and advanced models as your needs grow

Next step: put this on a noisy link (backup network, edge WAN) and watch the anomaly score tell you when “normal” isn’t normal anymore. If you want a ready-to-import Grafana dashboard JSON or a systemd-timerized version of the script, say the word and I’ll share templates you can drop in.