Artificial Intelligence + Rust Server Monitoring (with Bash Glue)

If you’ve ever been paged at 3 AM only to find “CPU high” and a wall of undifferentiated logs, you know classic threshold-based monitoring can be noisy and blind to novel failures. What if your monitoring could learn a node’s normal behavior and flag only the weird stuff—without chewing CPU or pulling in heavy runtimes?

In this post, we’ll build a tiny, fast Rust agent that:

• Samples CPU and memory,

• Learns a rolling baseline,

• Flags anomalies in real time,

• Streams JSON logs so Bash can route alerts anywhere (mail, Slack, webhooks).

You get AI-powered anomaly detection with the reliability of a single static binary, orchestrated by the Linux tools you already trust: Bash + systemd + journalctl.

Why this approach works

• Rust keeps the agent lean and predictable. No garbage collection pauses or hefty interpreters, which is perfect for busy servers or small VMs.

• Unsupervised anomaly detection (robust z-scores in this example) adapts to your node’s unique baseline—less brittle than static thresholds, and no labeled data required.

• Bash-first integration means observability stays simple: pipe JSON to jq, curl a webhook, or grep for audits.

• You own the stack. No SaaS lock-in. Single binary. Easy to audit.

What we’ll build (high level)

• airmon: A Rust daemon that emits JSON lines like: {"ts": "...", "cpu_pct": 7.1, "mem_pct": 54.3, "score": 0.8, "anomaly": false}

• A rolling baseline model using robust statistics (median/MAD). It’s simple, fast, and resilient to spikes.

• A systemd unit to run it as a service.

• Bash snippets to alert on anomalies.

---

1) Install prerequisites

We’ll install compilers, curl, pkg-config, OpenSSL headers (common for Rust crates), and jq for JSON handling. Then we’ll install Rust via rustup.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl build-essential pkg-config libssl-dev jq

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y curl pkgconfig openssl-devel jq

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y curl gcc make pkg-config libopenssl-devel jq

Install Rust toolchain (works on all distros):

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"
rustc --version
cargo --version

Optional for testing load later (stress-ng):

• apt:

sudo apt install -y stress-ng

• dnf:

sudo dnf install -y stress-ng

• zypper:

sudo zypper install -y stress-ng

---

2) Build the Rust AI monitor

Create a new Rust project:

cargo new airmon
cd airmon

Edit Cargo.toml to add dependencies:

cat > Cargo.toml <<'EOF'
[package]
name = "airmon"
version = "0.1.0"
edition = "2021"

[dependencies]
sysinfo = "0.30"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
EOF

Replace src/main.rs with:

cat > src/main.rs <<'EOF'
use serde::Serialize;
use std::collections::VecDeque;
use std::env;
use std::thread;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use sysinfo::{CpuExt, System, SystemExt};

#[derive(Serialize, Debug)]
struct Record {
    ts: u64,          // epoch seconds
    cpu_pct: f64,     // 0..100
    mem_pct: f64,     // 0..100
    z_cpu: f64,       // robust z-score
    z_mem: f64,       // robust z-score
    score: f64,       // combined anomaly score
    anomaly: bool,    // true if score exceeds threshold
}

fn env_u64(key: &str, default: u64) -> u64 {
    env::var(key).ok().and_then(|s| s.parse().ok()).unwrap_or(default)
}

fn env_f64(key: &str, default: f64) -> f64 {
    env::var(key).ok().and_then(|s| s.parse().ok()).unwrap_or(default)
}

fn median(vals: &mut [f64]) -> f64 {
    if vals.is_empty() { return 0.0; }
    vals.sort_by(|a, b| a.partial_cmp(b).unwrap());
    let mid = vals.len() / 2;
    if vals.len() % 2 == 0 {
        (vals[mid - 1] + vals[mid]) / 2.0
    } else {
        vals[mid]
    }
}

fn robust_z(x: f64, window: &VecDeque<f64>) -> f64 {
    if window.len() < 5 {
        return 0.0;
    }
    let mut w: Vec<f64> = window.iter().cloned().collect();
    let med = median(&mut w);
    let mut devs: Vec<f64> = w.into_iter().map(|v| (v - med).abs()).collect();
    let mad = median(&mut devs);
    // 1.4826 scales MAD to stddev for normal distributions
    let denom = (1.4826 * mad).max(1e-6);
    (x - med).abs() / denom
}

fn main() {
    // Tunables via environment variables:
    // AIRM_INTERVAL: seconds between samples (default 5)
    // AIRM_WINDOW:   rolling window size (default 120 ~ 10min at 5s)
    // AIRM_THRESHOLD: anomaly score threshold (default 5.0)
    let interval = env_u64("AIRM_INTERVAL", 5);
    let window_size = env_u64("AIRM_WINDOW", 120) as usize;
    let threshold = env_f64("AIRM_THRESHOLD", 5.0);

    let sleep = Duration::from_secs(interval);
    let mut sys = System::new_all();

    let mut cpu_hist: VecDeque<f64> = VecDeque::with_capacity(window_size);
    let mut mem_hist: VecDeque<f64> = VecDeque::with_capacity(window_size);

    loop {
        sys.refresh_cpu();
        sys.refresh_memory();

        let cpu = sys.global_cpu_info().cpu_usage() as f64; // 0..100
        let mem_total = sys.total_memory() as f64; // KiB
        let mem_used = sys.used_memory() as f64;   // KiB
        let mem_pct = if mem_total > 0.0 {
            (mem_used / mem_total) * 100.0
        } else { 0.0 };

        // update windows
        if cpu_hist.len() == window_size { cpu_hist.pop_front(); }
        if mem_hist.len() == window_size { mem_hist.pop_front(); }
        cpu_hist.push_back(cpu);
        mem_hist.push_back(mem_pct);

        // compute robust z-scores and combined anomaly score
        let zc = robust_z(cpu, &cpu_hist);
        let zm = robust_z(mem_pct, &mem_hist);
        // Combine features; you can weight them differently if needed
        let score = 0.5 * zc + 0.5 * zm;
        let anomaly = score.is_finite() && score >= threshold;

        let ts = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs();
        let rec = Record {
            ts, cpu_pct: cpu, mem_pct, z_cpu: zc, z_mem: zm, score, anomaly
        };
        println!("{}", serde_json::to_string(&rec).unwrap());

        thread::sleep(sleep);
    }
}
EOF

Build a release binary:

cargo build --release

Install it system-wide:

sudo install -Dm755 target/release/airmon /usr/local/bin/airmon

Try it:

AIRM_INTERVAL=2 AIRM_WINDOW=60 AIRM_THRESHOLD=5 /usr/local/bin/airmon | jq .

You should see a stream of JSON with anomaly=false under normal load. We’ll stress it soon.

---

3) Run it as a service (systemd) and alert with Bash

Create an env file to tweak thresholds without rebuilding:

sudo mkdir -p /etc/airmon
sudo tee /etc/airmon/airmon.env >/dev/null <<'EOF'
AIRM_INTERVAL=5
AIRM_WINDOW=120
AIRM_THRESHOLD=5.0
EOF

Create the systemd unit:

sudo tee /etc/systemd/system/airmon.service >/dev/null <<'EOF'
[Unit]
Description=AI Rust Monitor (CPU/Memory anomaly detection)
After=network.target

[Service]
EnvironmentFile=/etc/airmon/airmon.env
ExecStart=/usr/local/bin/airmon
Restart=always
RestartSec=2
# Optional: run as dedicated user (create first)
# User=airmon
# Group=airmon
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now airmon.service
systemctl status airmon.service --no-pager

Watch JSON logs:

journalctl -u airmon.service -f -o cat

Send alerts from Bash (example: print only anomalies):

journalctl -u airmon.service -f -o cat | jq -r 'select(.anomaly==true) | "ALERT ts=\(.ts) cpu=\(.cpu_pct|round) mem=\(.mem_pct|round) score=\(.score|round)"'

Slack webhook example:

# export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"
journalctl -u airmon.service -f -o cat | \
jq -rc 'select(.anomaly==true) | {text: ("[airmon] Anomaly: cpu=" + (.cpu_pct|tostring) + " mem=" + (.mem_pct|tostring) + " score=" + (.score|tostring))}' | \
while read -r line; do
  curl -sS -X POST -H 'Content-type: application/json' --data "$line" "$SLACK_WEBHOOK_URL" >/dev/null
done

Email via mailx, webhook to your incident tool, or even write to a file—the output is just JSON, so Bash makes routing trivial.

---

4) Validate with a real-world test

Generate a short CPU and memory surge to see anomalies fire.

• apt:

sudo apt install -y stress-ng

• dnf:

sudo dnf install -y stress-ng

• zypper:

sudo zypper install -y stress-ng

Then:

stress-ng --cpu 4 --vm 2 --vm-bytes 70% -t 30s

In another terminal:

journalctl -u airmon.service -f -o cat | jq 'select(.anomaly==true)'

You should see anomaly=true while the surge runs, then return to normal.

---

5) Production tips and extensions

• Tune thresholds per host role:

  • Busy build boxes: raise AIRM_THRESHOLD (e.g., 7–8).
  • Latency-sensitive services: lower it (e.g., 4–5).

• Add more features:

  • Extend the Rust code to include load average, disk IO, or network RX/TX; combine features as a weighted score.

• Quiet known-noise windows:

  • Wrap alerts in a Bash guard during deploys/cron windows, or drop a flag file that suppresses alerting temporarily.

• Ship to your monitoring stack:

  • Pipe JSON into a lightweight collector or pushgateway. Or expose Prometheus text format from the Rust app on a localhost port if you prefer scraping.

• Hardening:

  • Run as non-root user.
  • Resource limits with systemd (CPUQuota, MemoryMax).
  • Log rotation is handled by journald; adjust as needed.

---

Why “AI” here?

We used a robust, unsupervised anomaly detector (median/MAD z-scores) that adapts online. It’s fast, explainable, and avoids overfitting. If you want heavier ML:

• Swap in a more advanced model (e.g., Isolation Forest) via Rust’s ML ecosystem (linfa) and score the latest sample against a rolling fit.

• Keep the interface exactly the same: JSON to stdout.

The key is the pattern: learn the baseline locally, decide locally, and let Bash route the outcome.

---

Conclusion and next steps

You now have a tiny, auditable, AI-enabled Rust agent that flags unusual resource behavior and plays perfectly with Linux tooling. From here:

• Deploy airmon to a staging node.

• Tune AIRM_THRESHOLD and AIRM_WINDOW for your workload.

• Plug the JSON stream into your alert transport of choice.

• Extend features (load, IO, network) as needed.

If you want a step-by-step repo, reply and I’ll package this into a GitHub project with CI and container images. Happy monitoring!