Posted on
• Artificial Intelligence

Artificial Intelligence Reporting Dashboards

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Reporting Dashboards on Linux: From Bash to Insights

If you’re like most engineers, you already have dashboards—lots of them. But most dashboards only show what happened, not why. An AI-augmented reporting dashboard turns raw metrics into explanations: spotting anomalies, surfacing trends, and recommending actions automatically. And you don’t need GPUs or heavy tooling to get started—you can build a lightweight, explainable AI dashboard on any Linux box with Bash, Python, and a few open-source libraries.

This guide shows you how to stand up a minimal, fast, and reliable AI-enabled reporting dashboard using:

  • Bash for data collection

  • pandas and scikit-learn for analysis and anomaly detection

  • Flask and Chart.js for a simple web UI

  • systemd timers for hands-off automation

We’ll cover end-to-end setup (including apt, dnf, and zypper installs), and give you actionable steps you can adapt to your own metrics.

Why AI Dashboards (on Linux) Are Worth Your Time

  • They summarize, not just visualize. Classic dashboards show charts. AI dashboards add context—“CPU load jumped 35% vs. baseline,” “Memory usage is trending up,” “Here are likely causes.”

  • They work with what you already have. A few CSVs or logs are enough. You don’t need a data warehouse to get value from anomaly detection and simple trend models.

  • They’re simple to operate. With Bash + Python + systemd, you can run this on a tiny VM or even a Raspberry Pi—no Kubernetes cluster required.

  • They’re explainable. Using classical ML (e.g., IsolationForest) and rolling statistics, you can keep the logic transparent and tunable.

What You’ll Build

We’ll collect local system metrics (CPU load and memory usage) into a CSV, analyze them for anomalies and trends, and serve a small web dashboard that updates continuously.

Stack:

  • Data collection: Bash script (reads /proc)

  • Analysis: pandas + scikit-learn (IsolationForest)

  • Web UI: Flask (Python) + Chart.js (no Node required)

  • Automation: systemd user timer

1) Install prerequisites

Install system packages.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-venv python3-pip git nginx
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y python3 python3-pip git nginx

(Note: Python’s venv is included in python3 on most Fedora-based systems. If you need virtualenv specifically: sudo dnf install -y python3-virtualenv)

  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y python3 python3-pip git nginx

(Note: Python’s venv is typically included in python3 on openSUSE. If you need virtualenv: sudo zypper install -y python3-virtualenv)

2) Scaffold the project and Python environment

mkdir -p ~/ai-dashboard && cd ~/ai-dashboard

python3 -m venv .venv
source .venv/bin/activate

pip install --upgrade pip
pip install flask pandas scikit-learn

Directory outline:

ai-dashboard/
  .venv/
  data/
  metrics_collect.sh
  analyze_and_serve.py

3) Collect metrics with Bash (every minute)

Create a simple collector that writes a CSV with timestamp, 1-minute load average, and memory usage percent.

Create data dir and script:

mkdir -p data
cat > metrics_collect.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

DATA_DIR="${DATA_DIR:-$(pwd)/data}"
mkdir -p "$DATA_DIR"
CSV="$DATA_DIR/metrics.csv"

if [ ! -f "$CSV" ]; then
  echo "timestamp,load1,mem_used_pct" > "$CSV"
fi

ts="$(date -Iseconds)"
load1="$(awk '{print $1}' /proc/loadavg)"
mem_used_pct="$(awk '
  /MemTotal/ {t=$2}
  /MemAvailable/ {a=$2}
  END {printf "%.2f", (1 - a/t) * 100}
' /proc/meminfo)"

echo "$ts,$load1,$mem_used_pct" >> "$CSV"
EOF
chmod +x metrics_collect.sh

Run it once to generate initial data:

./metrics_collect.sh

Automate with a systemd user timer (recommended; no root required):

Service unit:

mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/ai-metrics-collector.service << 'EOF'
[Unit]
Description=Collect system metrics to CSV

[Service]
Type=oneshot
WorkingDirectory=%h/ai-dashboard
ExecStart=%h/ai-dashboard/metrics_collect.sh
EOF

Timer unit (run every minute):

cat > ~/.config/systemd/user/ai-metrics-collector.timer << 'EOF'
[Unit]
Description=Run metrics collector every minute

[Timer]
OnCalendar=*:0/1
Persistent=true
Unit=ai-metrics-collector.service

[Install]
WantedBy=timers.target
EOF

Enable and start the timer:

systemctl --user daemon-reload
systemctl --user enable --now ai-metrics-collector.timer

# Check status and recent runs
systemctl --user status ai-metrics-collector.timer
journalctl --user -u ai-metrics-collector.service -n 20 --no-pager

Tip: To use user services across SSH sessions, ensure lingering is enabled for your user:

sudo loginctl enable-linger "$USER"

4) Analyze and serve a dashboard (Flask + Chart.js)

Create a single Python app that:

  • Loads CSV data

  • Computes rolling trends and detects anomalies with IsolationForest

  • Serves a small web UI

Create analyze_and_serve.py:

cat > analyze_and_serve.py << 'EOF'
#!/usr/bin/env python3
import os
from pathlib import Path
import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest
from flask import Flask, jsonify, render_template_string

DATA_CSV = Path("data/metrics.csv")

def load_df():
    if not DATA_CSV.exists():
        return pd.DataFrame(columns=["timestamp","load1","mem_used_pct"])
    df = pd.read_csv(DATA_CSV, parse_dates=["timestamp"])
    if not df.empty:
        df = df.sort_values("timestamp")
    return df

def analyze(df: pd.DataFrame):
    if df.empty:
        return df, {
            "samples": 0,
            "trend": {"load1_pct_change": 0.0, "mem_used_pct_change": 0.0},
            "anomalies_recent": [],
            "recommendations": ["No data yet. Waiting for the first samples..."]
        }

    # Prepare features
    X = df[["load1", "mem_used_pct"]].copy()
    X = X.replace([np.inf, -np.inf], np.nan).fillna(method="ffill").fillna(0.0)

    # Anomaly detection (robust, simple)
    if len(X) >= 30:
        iso = IsolationForest(contamination=0.05, random_state=42)
        df["anomaly"] = (iso.fit_predict(X) == -1)
    else:
        df["anomaly"] = False

    # Rolling averages for light smoothing
    df["load1_rolling"] = df["load1"].rolling(window=15, min_periods=3).mean()
    df["mem_rolling"]  = df["mem_used_pct"].rolling(window=15, min_periods=3).mean()

    # Simple trend: compare recent window to prior baseline
    tail = df.tail(15)
    prev = df.tail(90).head(60)  # baseline window
    def pct_change(a, b):
        if len(a) == 0 or len(b) == 0 or b.mean() == 0:
            return 0.0
        return float(((a.mean() - b.mean()) / (b.mean() + 1e-9)) * 100.0)

    trend_load = pct_change(tail["load1"], prev["load1"])
    trend_mem  = pct_change(tail["mem_used_pct"], prev["mem_used_pct"])

    top_anoms = df[df["anomaly"]].tail(5)[["timestamp","load1","mem_used_pct"]]
    insights = {
        "samples": int(len(df)),
        "trend": {
            "load1_pct_change": round(trend_load, 2),
            "mem_used_pct_change": round(trend_mem, 2),
        },
        "anomalies_recent": [
            {
                "timestamp": pd.to_datetime(ts).isoformat(),
                "load1": float(l),
                "mem_used_pct": float(m)
            }
            for ts, l, m in zip(top_anoms["timestamp"], top_anoms["load1"], top_anoms["mem_used_pct"])
        ],
        "recommendations": []
    }

    # Simple, explainable recommendations
    recent_mem = float(df["mem_used_pct"].tail(1).iloc[0])
    recent_anoms = int(df["anomaly"].tail(10).sum())
    if trend_load > 25 or recent_anoms >= 2:
        insights["recommendations"].append(
            "Investigate recent CPU load spikes; check top processes (top/htop) and recent deploys."
        )
    if recent_mem > 85:
        insights["recommendations"].append(
            "High memory usage detected (>85%). Inspect memory-heavy services, leaks, or increase memory."
        )
    if not insights["recommendations"]:
        insights["recommendations"].append("System looks healthy in the last window.")

    return df, insights

app = Flask(__name__)

INDEX_HTML = """<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>AI Reporting Dashboard</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <style>
    body { font-family: system-ui, sans-serif; margin: 20px; }
    .grid { display: grid; grid-template-columns: 1fr; gap: 18px; }
    @media (min-width: 900px) { .grid { grid-template-columns: 1fr 1fr; } }
    .card { border: 1px solid #ddd; border-radius: 8px; padding: 14px; }
    .good { color: #177245; } .warn { color: #9B870C; } .bad { color: #B22222; }
    code { background: #f8f8f8; padding: 2px 4px; }
  </style>
  <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
</head>
<body>
<h1>AI Reporting Dashboard</h1>
<p>Lightweight anomaly detection and trend insights, powered by pandas + scikit-learn.</p>

<div class="grid">
  <div class="card">
    <h3>CPU Load (1m)</h3>
    <canvas id="loadChart" height="160"></canvas>
  </div>
  <div class="card">
    <h3>Memory Usage (%)</h3>
    <canvas id="memChart" height="160"></canvas>
  </div>
  <div class="card">
    <h3>Insights</h3>
    <ul id="insights"></ul>
  </div>
  <div class="card">
    <h3>Recent Anomalies</h3>
    <ul id="anoms"></ul>
  </div>
</div>

<script>
async function fetchJSON(url){ const r = await fetch(url); return await r.json(); }

let loadChart, memChart;

function mkChart(ctx, label, color){
  return new Chart(ctx, {
    type: 'line',
    data: { labels: [], datasets: [
      { label: label, data: [], borderColor: color, tension: 0.2, pointRadius: 0 }
    ]},
    options: {
      responsive: true,
      scales: { x: { display: false }, y: { beginAtZero: false } },
      plugins: { legend: { display: true } }
    }
  });
}

async function refresh(){
  const metrics = await fetchJSON('/api/metrics');
  const ins = await fetchJSON('/api/insights');

  if(!loadChart){
    loadChart = mkChart(document.getElementById('loadChart'), 'load1', '#1f77b4');
    memChart  = mkChart(document.getElementById('memChart'),  'mem %', '#ff7f0e');
  }

  loadChart.data.labels = metrics.timestamps;
  loadChart.data.datasets[0].data = metrics.load1;
  loadChart.update('none');

  memChart.data.labels = metrics.timestamps;
  memChart.data.datasets[0].data = metrics.mem_used_pct;
  memChart.update('none');

  const insightEl = document.getElementById('insights');
  insightEl.innerHTML = '';
  const tLoad = ins.trend.load1_pct_change;
  const tMem  = ins.trend.mem_used_pct_change;
  insightEl.innerHTML += `<li>Samples: <code>${ins.samples}</code></li>`;
  insightEl.innerHTML += `<li>Load trend: <b>${tLoad}%</b></li>`;
  insightEl.innerHTML += `<li>Memory trend: <b>${tMem}%</b></li>`;
  ins.recommendations.forEach(r => {
    const li = document.createElement('li');
    li.textContent = r;
    insightEl.appendChild(li);
  });

  const anoms = document.getElementById('anoms');
  anoms.innerHTML = '';
  ins.anomalies_recent.forEach(a => {
    const li = document.createElement('li');
    li.textContent = `${a.timestamp} — load1=${a.load1.toFixed(2)}, mem=${a.mem_used_pct.toFixed(1)}%`;
    anoms.appendChild(li);
  });
}

setInterval(refresh, 30000);
refresh();
</script>
</body>
</html>
"""

@app.get("/")
def index():
    return render_template_string(INDEX_HTML)

@app.get("/api/metrics")
def api_metrics():
    df = load_df()
    df, _ = analyze(df)
    dfl = df.tail(1000) if len(df) > 1000 else df
    out = {
        "timestamps": dfl["timestamp"].dt.strftime("%Y-%m-%d %H:%M:%S").tolist(),
        "load1": [float(x) for x in dfl["load1"].tolist()],
        "mem_used_pct": [float(x) for x in dfl["mem_used_pct"].tolist()],
        "anomaly": [int(x) for x in dfl["anomaly"].astype(int).tolist()]
    }
    return jsonify(out)

@app.get("/api/insights")
def api_insights():
    df = load_df()
    _, insights = analyze(df)
    return jsonify(insights)

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8000"))
    app.run(host="0.0.0.0", port=port)
EOF
chmod +x analyze_and_serve.py

Run the server:

source .venv/bin/activate
python analyze_and_serve.py
# Visit http://127.0.0.1:8000

Let it run while the systemd timer appends new rows each minute; your dashboard will update on refresh and every 30 seconds.

5) Optional: Serve behind Nginx

If you don’t already have Nginx:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y nginx
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y nginx
sudo systemctl enable --now nginx
  • openSUSE (zypper):
sudo zypper install -y nginx
sudo systemctl enable --now nginx

Add a simple reverse proxy (e.g., /etc/nginx/conf.d/ai-dashboard.conf):

server {
    listen 80;
    server_name dashboard.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Test and reload:

sudo nginx -t
sudo systemctl reload nginx

Actionable Tips and Real-World Extensions

1) Add your business metrics

  • Append rows from your job queues, API counters, or database stats into the same CSV (or another one).

  • You can add more columns (e.g., requests_per_min, error_rate) and update analyze() to include them in anomaly detection.

2) Parse Nginx access logs for error trends

# Example: count 5xx responses in the last minute
awk '($9 ~ /^5/) {print $4}' /var/log/nginx/access.log | wc -l

Better: pre-aggregate by minute with log-format that includes $status and $request_time, then write into CSV columns like error_5xx and p95_latency_ms.

3) Make anomalies actionable

  • Page on repeated anomalies: trigger a webhook when df["anomaly"].tail(10).sum() >= N.

  • Tie insights to runbooks: return URLs to diagnostics (Grafana, Loki queries, CI/CD changes).

4) Keep it explainable and fast

  • Tune IsolationForest’s contamination (e.g., 0.01–0.1) for your data.

  • Prefer rolling statistics and simple baselines before going deep into heavy models.

5) Automate everything

  • Use systemd timers for each ingest (service + timer per source).

  • Add a systemd user service for the Flask app or run it with a process manager (e.g., gunicorn + systemd).

Troubleshooting

  • scikit-learn install issues: Ensure you’re on a supported architecture; upgrade pip; try a clean venv. Wheels are available for mainstream distros; compilation is rarely needed.

  • Empty dashboard: Wait a minute for the first few samples; check journalctl for the collector timer.

  • High CPU load in visuals: Limit points (we capped at 1000) or downsample older data.

Conclusion and Next Steps (CTA)

You just built an AI-enabled reporting dashboard that runs anywhere Linux does—no GPUs, no vendor lock-in. From here:

  • Add your own metrics: business KPIs, request errors, queue depths.

  • Expand the analyzer: include error_rate and p95_latency, add domain rules, and tune anomaly sensitivity.

  • Productionize it: run behind Nginx with TLS, add authentication, and ship metrics and insights to long-term storage.

If this helped you ship more signal and less noise, take 30 minutes to wire in one business metric today—you’ll get immediate value from AI-assisted insights without changing your stack.

Related posts