Posted on
• Artificial Intelligence

Intelligent Network Monitoring with Bash

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Intelligent Network Monitoring with Bash

What if your network could warn you before users complain—without a heavyweight monitoring stack? Imagine a 50-line Bash script quietly learning your network’s normal latency, suppressing noise, and only waking you when something’s truly wrong. That’s the promise of intelligent network monitoring with Bash: small, portable, and surprisingly powerful.

In this article you’ll build a pragmatic, “smart” Bash-based monitor that:

  • Learns a latency baseline per host (EWMA smoothing)

  • Flags anomalies using standard deviations and packet loss thresholds

  • Runs on any Linux box with minimal dependencies

  • Auto-captures diagnostics (mtr) and alerts to a webhook

You’ll walk away with a working script, a systemd timer (or cron), and clear steps to extend it.

Why Bash Is a Valid Choice (Yes, Really)

  • Lightweight and portable: Bash + a few common tools run fine on servers, cloud instances, routers, and containers—no agents or databases required.

  • Composable: Combine ping, mtr, and curl to gather metrics, diagnose, and alert.

  • Smart without bloat: A simple EWMA (Exponentially-Weighted Moving Average) plus sigma-based anomaly detection drastically reduces false positives compared to hard thresholds alone.

  • Easy to automate: Systemd timers or cron make it simple to schedule and supervise.

What We’ll Build

  • A small Bash script that:

    • Pings a list of hosts, parses latency, jitter, and loss.
    • Maintains rolling baselines with EWMA.
    • Triggers alerts only when anomalies persist (consecutive breaches).
    • Automatically captures an mtr report on alert.
    • Logs metrics to CSV for later analysis.

  • A minimal config file to tune thresholds and hosts.

  • A systemd timer (or cron) to run it every minute.

1) Install Dependencies

We’ll use ping (from iputils), mtr for on-demand path diagnostics, curl for webhooks, and jq to safely build JSON payloads.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y iputils-ping mtr curl jq

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y iputils mtr curl jq

openSUSE Leap/Tumbleweed (zypper):

sudo zypper install -y iputils mtr curl jq

Create directories:

sudo install -d -m 750 -o root -g root /opt/netmon /var/log/netmon /var/lib/netmon

2) Create a Minimal Config

Save as /opt/netmon/netmon.conf:

# Hosts to monitor (IPs or DNS names)
HOSTS=("1.1.1.1" "8.8.8.8" "github.com")

# Ping sampling
SAMPLES=5          # packets per poll
INTERVAL=0.2       # seconds between packets
TIMEOUT=1          # per-packet timeout (seconds)

# Baseline + detection
ALPHA=0.3          # EWMA smoothing factor (0.1–0.5 is sane)
SIGMA=3            # STDDEV multiplier for anomaly
LOSS_THRESHOLD=20  # % packet loss considered problematic
CONSECUTIVE=2      # require N consecutive anomalies before alerting

# Paths
LOG_DIR="/var/log/netmon"
STATE_DIR="/var/lib/netmon"

# Optional: POST JSON {"text": "..."} to a webhook (Slack/Discord/custom)
WEBHOOK_URL=""

Tips:

  • Start with stable, always-up targets: your gateway, your ISP DNS, a public DNS (1.1.1.1, 8.8.8.8), and a key SaaS endpoint.

  • Tune ALPHA and SIGMA to suppress noise. Higher SIGMA or lower ALPHA = fewer alerts.

3) The Monitoring Script

Save as /opt/netmon/netmon.sh and make it executable:

sudo chmod +x /opt/netmon/netmon.sh

Script:

#!/usr/bin/env bash
set -euo pipefail

CONFIG="/opt/netmon/netmon.conf"
[[ -f "$CONFIG" ]] || { echo "Config not found: $CONFIG" >&2; exit 1; }
# shellcheck disable=SC1090
source "$CONFIG"

mkdir -p "$LOG_DIR" "$STATE_DIR"

sanitize() {
  # Safe filename for a host (keep letters, numbers, dot, dash, underscore)
  echo -n "$1" | tr -c 'A-Za-z0-9._-\' '\n' | tr '\n' '_' | sed 's/_\+/_/g; s/_$//'
}

emit_csv_header_if_needed() {
  local csv="$LOG_DIR/metrics.csv"
  [[ -s "$csv" ]] || echo "iso_time,host,loss_pct,avg_ms,jitter_ms,ewma_avg_ms,ewma_jitter_ms,anomaly" >> "$csv"
}

read_state() {
  local sf="$1"
  EWMA_AVG=""
  EWMA_JIT=""
  CONSEC=0
  [[ -f "$sf" ]] && source "$sf" || true
}

write_state() {
  local sf="$1"
  cat > "$sf" <<EOF
EWMA_AVG=$EWMA_AVG
EWMA_JIT=$EWMA_JIT
CONSEC=$CONSEC
EOF
}

max() { awk -v a="$1" -v b="$2" 'BEGIN{printf "%.6f", (a>b)?a:b}'; }

ping_once() {
  local host="$1"
  # Use numeric output (-n), quiet summary (-q), SAMPLES, INTERVAL, TIMEOUT
  ping -n -q -c "$SAMPLES" -i "$INTERVAL" -W "$TIMEOUT" "$host" 2>&1 || true
}

parse_loss_pct() {
  grep -oE '[0-9.]+% packet loss' | awk '{print $1}' | tr -d '%' || true
}

parse_rtt_avg() {
  awk -F'=' '/rtt/ {split($2,a,"/"); gsub(/ /,"",a[2]); print a[2]}' || true
}

parse_rtt_jit() {
  awk -F'=' '/rtt/ {split($2,a,"/"); gsub(/ ms/,"",a[4]); gsub(/ /,"",a[4]); print a[4]}' || true
}

alert() {
  local host="$1" loss="$2" avg="$3" jit="$4" ew_avg="$5" ew_jit="$6" diag_file="$7"
  local ts msg payload

  ts="$(date -Iseconds)"
  msg="[$ts] ALERT: $host latency/loss anomaly. loss=${loss}% avg=${avg}ms (baseline ~${ew_avg}ms ± ${ew_jit}ms). mtr: $diag_file"

  echo "$msg" | tee -a "$LOG_DIR/alerts.log" >&2

  if [[ -n "${WEBHOOK_URL:-}" ]]; then
    payload=$(jq -n --arg text "$msg" '{text:$text}')
    curl -fsS -X POST -H 'Content-Type: application/json' -d "$payload" "$WEBHOOK_URL" || true
  fi
}

poll_host() {
  local host="$1"
  local san out loss avg jit ts csv="$LOG_DIR/metrics.csv"
  san="$(sanitize "$host")"
  out="$(ping_once "$host")"

  loss="$(echo "$out" | parse_loss_pct)"
  avg="$(echo "$out" | parse_rtt_avg)"
  jit="$(echo "$out" | parse_rtt_jit)"

  # Normalize missing values (e.g., 100% loss => no RTT line)
  loss="${loss:-100}"
  avg="${avg:-0}"
  jit="${jit:-0}"

  local sf="$STATE_DIR/$san.state"
  read_state "$sf"

  # Initialize EWMA if missing
  if [[ -z "${EWMA_AVG:-}" || -z "${EWMA_JIT:-}" ]]; then
    EWMA_AVG="$avg"
    EWMA_JIT="$jit"
  fi

  # Update EWMA
  EWMA_AVG="$(awk -v a="$ALPHA" -v x="$avg" -v m="$EWMA_AVG" 'BEGIN{printf "%.3f", a*x + (1-a)*m}')"
  EWMA_JIT="$(awk -v a="$ALPHA" -v x="$jit" -v m="$EWMA_JIT" 'BEGIN{printf "%.3f", a*x + (1-a)*m}')"

  # Anomaly if: high loss OR avg too far above baseline
  local floor_jit
  floor_jit="$(max "$EWMA_JIT" 1)"  # avoid near-zero jitter
  local thresh
  thresh="$(awk -v m="$EWMA_AVG" -v s="$SIGMA" -v j="$floor_jit" 'BEGIN{printf "%.3f", m + s*j}')"

  local is_anom=0
  awk -v l="$loss" -v lt="$LOSS_THRESHOLD" 'BEGIN{if (l>=lt) exit 0; else exit 1}' && is_anom=1 || true
  awk -v a="$avg" -v t="$thresh" 'BEGIN{if (a>t) exit 0; else exit 1}' && is_anom=1 || true

  # Consecutive confirmation
  if [[ "$is_anom" -eq 1 ]]; then
    CONSEC=$(( CONSEC + 1 ))
  else
    CONSEC=0
  fi

  # Log metrics
  ts="$(date -Iseconds)"
  emit_csv_header_if_needed
  echo "$ts,$host,$loss,$avg,$jit,$EWMA_AVG,$EWMA_JIT,$is_anom" >> "$csv"

  # On confirmed anomaly, capture diagnostics and alert
  if [[ "$is_anom" -eq 1 && "$CONSEC" -ge "$CONSECUTIVE" ]]; then
    local diag="$LOG_DIR/${san}-$(date +%Y%m%dT%H%M%S).mtr.txt"
    mtr -rwzc 10 "$host" > "$diag" 2>&1 || true
    alert "$host" "$loss" "$avg" "$jit" "$EWMA_AVG" "$EWMA_JIT" "$diag"
    CONSEC=0
  fi

  write_state "$sf"
}

main() {
  for h in "${HOSTS[@]}"; do
    poll_host "$h"
  done
}

main "$@"

How it works:

  • EWMA tracks a per-host baseline (EWMA_AVG, EWMA_JIT).

  • An anomaly triggers if:

    • Packet loss >= LOSS_THRESHOLD, or
    • Current avg latency > baseline avg + SIGMA × baseline jitter (with a 1 ms floor).

  • Alerts require CONSECUTIVE consecutive anomalies to avoid flapping.

  • On alert, an mtr report is captured to help localize the issue.

4) Schedule It (systemd or cron)

Option A: systemd timer (recommended)

Create /etc/systemd/system/netmon.service:

[Unit]
Description=NetMon - Intelligent Bash Network Monitor

[Service]
Type=oneshot
ExecStart=/opt/netmon/netmon.sh

Create /etc/systemd/system/netmon.timer:

[Unit]
Description=Run NetMon every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
Unit=netmon.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now netmon.timer
systemctl list-timers | grep netmon

Option B: cron (if you prefer)

sudo crontab -e
# Add:
* * * * * /opt/netmon/netmon.sh >/dev/null 2>&1

5) Real-World Examples and Tips

  • Start small. Example HOSTS set:

    • Your default gateway (often 192.168.1.1)
    • A public DNS (1.1.1.1, 8.8.8.8)
    • A critical SaaS (e.g., github.com)

  • Tune thresholds:

    • If you see noisy alerts on Wi‑Fi, raise SIGMA to 4–5, and/or raise LOSS_THRESHOLD to 30–40.
    • For stable DC links, SIGMA=2 and CONSECUTIVE=1 may detect issues faster.

  • Use webhooks:

    • Set WEBHOOK_URL to a Slack/Discord/Webex/MS Teams Incoming Webhook (they all accept JSON with a text field).

  • Postmortems:

    • Inspect mtr files in /var/log/netmon/ for hop-by-hop loss/latency to identify where issues occur (LAN, ISP edge, backbone).

  • Metrics:

    • The CSV in /var/log/netmon/metrics.csv can be charted with your favorite tool or imported into a spreadsheet quickly.

Troubleshooting

  • Permission issues:

    • Ensure /opt/netmon/netmon.sh is executable and directories are writable by the user running it (root via systemd/cron is simplest).

  • mtr or ping not found:

    • Re-check installation commands for your distro (apt/dnf/zypper above).

  • No alerts:

    • Intentionally set a very low LOSS_THRESHOLD (e.g., 0) or SIGMA=1 temporarily to verify delivery, then revert.

Conclusion and Next Steps

You don’t need a full-blown monitoring platform to get meaningful, low-noise network visibility. With a smart Bash script, EWMA baselines, and just-in-time diagnostics, you can monitor links, catch real problems, and keep your logs tidy.

Next steps:

  • Deploy this on a small VM or your home server.

  • Tailor HOSTS, SIGMA, and CONSECUTIVE for your environment.

  • Wire up WEBHOOK_URL to your team chat and share your first alert.

  • Extend it: write Prometheus textfile metrics, add IPv6 checks, or integrate traceroute-on-recovery for closure.

If this helped, share it with a teammate who’s tired of false-positive pings. Your network will thank you before the next 2 a.m. outage.

Related posts