Posted on
• Artificial Intelligence

Artificial Intelligence Monitoring Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Monitoring Projects with Pure Bash: From “it works” to “it’s observable”

You finally shipped that model. Congrats! But the real test starts now: is it fast? Is it starving for CPU or GPU? Are errors spiking? Are users quiet because they’re happy—or because latency silently doubled overnight? In production AI, the gap between “it runs” and “it’s reliable” is monitoring.

This post shows how to build a practical, low-friction AI monitoring stack using Linux, Bash, and a few tiny tools. You’ll collect latency, system and GPU metrics, detect error anomalies, send alerts to Slack/Telegram, and expose everything to Prometheus/Grafana—all without large agents or heavy SDKs.

Why this matters:

  • AI workloads are resource-hungry and bursty: you need guardrails on CPU, memory, disk, and GPU.

  • Latency and error rates drift with new data, models, or traffic patterns.

  • Lightweight Bash-based monitors fit almost anywhere—VMs, edge nodes, air-gapped boxes—and are easy to audit.

Prerequisites and installation

We’ll use only standard Linux tools and systemd timers. Install the small set of packages we’ll reference:

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y curl jq bc gawk

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y curl jq bc gawk

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y curl jq bc gawk

Notes:

  • GPU metrics are optional and auto-detected via nvidia-smi if your NVIDIA driver stack is installed.

  • We’ll use systemd timers instead of cron, so no extra packages are required.

The plan (4 actionable building blocks)

1) Collect core metrics (latency, CPU/mem/disk, optional GPU, process counts) and expose them in Prometheus format using a Bash exporter.

2) Detect error anomalies from logs using a rolling baseline and z-score, alerting to Slack/Telegram.

3) Run everything on fixed intervals using systemd timers, writing read-only Prometheus textfiles.

4) Visualize with Prometheus node_exporter’s textfile-collector and Grafana (optional).

You can implement all of this in under an hour.

1) Bash exporter: latency + system/GPU/process metrics

This script probes your AI HTTP endpoint, measures latency percentiles, collects host and optional GPU metrics, and writes Prometheus-compatible metrics into a textfile collector directory.

Create directories:

sudo mkdir -p /usr/local/lib/ai-monitor
sudo mkdir -p /var/lib/node_exporter/textfile
sudo chown -R root:root /usr/local/lib/ai-monitor
sudo chown -R root:root /var/lib/node_exporter

Create the exporter script:

sudo tee /usr/local/lib/ai-monitor/ai-metrics.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Config
AI_ENDPOINT="${AI_ENDPOINT:-http://127.0.0.1:8000/health}"   # Any fast, 200-OK endpoint on your model server
CUSTOM_JSON_METRICS_URL="${CUSTOM_JSON_METRICS_URL:-}"       # Optional: returns JSON like {"queue_depth": 7, "qps": 123.4}
AI_PROCESS_NAME="${AI_PROCESS_NAME:-python}"                 # Pattern to count model processes (pgrep -f)
OUT_DIR="/var/lib/node_exporter/textfile"
OUT_FILE="$OUT_DIR/ai-metrics.prom"
TMP_FILE="$(mktemp)"

# Helpers
now_ts() { date +%s; }

latency_samples() {
  local n="${1:-5}"
  for i in $(seq 1 "$n"); do
    # time_total in seconds as float
    curl -s -o /dev/null -w "%{time_total}\n" "$AI_ENDPOINT" || echo "NaN"
  done
}

quantiles_gawk() {
  # Reads floats on stdin; prints "count min p50 p95 max mean"
  gawk '
    function isnan(x){ return (x+0!=x) }
    { if (!isnan($1)) a[++n]=$1; sum+=$1 }
    END{
      if(n==0){ print 0, "NaN","NaN","NaN","NaN","NaN"; exit }
      asort(a)
      p50=a[int(0.5*n+0.5)]
      p95=a[int(0.95*n+0.5)]
      min=a[1]; max=a[n]
      mean=sum/n
      print n, min, p50, p95, max, mean
    }'
}

# Latency
read -r count min p50 p95 max mean < <( latency_samples 5 | quantiles_gawk )

# CPU load (1m), mem, disk
load1=$(cut -d' ' -f1 /proc/loadavg)
mem_total_kb=$(grep -i MemTotal /proc/meminfo | awk '{print $2}')
mem_avail_kb=$(grep -i MemAvailable /proc/meminfo | awk '{print $2}')
disk_root_used_pct=$(df -P / | awk 'NR==2{gsub("%","",$5); print $5}')

# Process count
proc_count=$(pgrep -fc "$AI_PROCESS_NAME" || true)

# Optional GPU via nvidia-smi
gpu_block=""
if command -v nvidia-smi >/dev/null 2>&1; then
  # utilization.gpu (%), memory.used (MiB), memory.total (MiB)
  while IFS=',' read -r util mem_used mem_total; do
    util="$(echo "$util" | tr -dc '0-9')"
    mem_used="$(echo "$mem_used" | tr -dc '0-9')"
    mem_total="$(echo "$mem_total" | tr -dc '0-9')"
    gpu_block+=$'\n'"ai_gpu_utilization_percent $util"
    gpu_block+=$'\n'"ai_gpu_memory_used_mebibytes $mem_used"
    gpu_block+=$'\n'"ai_gpu_memory_total_mebibytes $mem_total"
  done < <(nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits 2>/dev/null || true)
fi

# Optional custom JSON metrics (e.g., from /metrics or /status)
custom_block=""
if [[ -n "${CUSTOM_JSON_METRICS_URL}" ]]; then
  json="$(curl -s --max-time 2 "$CUSTOM_JSON_METRICS_URL" || echo '{}')"
  # Expecting fields "queue_depth" and "qps" if available
  qd=$(echo "$json" | jq -r '.queue_depth // empty' 2>/dev/null || true)
  qps=$(echo "$json" | jq -r '.qps // empty' 2>/dev/null || true)
  [[ -n "$qd" ]]  && custom_block+=$'\n'"ai_queue_depth $qd"
  [[ -n "$qps" ]] && custom_block+=$'\n'"ai_qps $qps"
fi

# Write Prometheus textfile
{
  echo "# HELP ai_latency_seconds AI endpoint latency."
  echo "# TYPE ai_latency_seconds summary"
  [[ "$count" != "0" ]] && {
    echo "ai_latency_seconds_count $count"
    echo "ai_latency_seconds_min $min"
    echo "ai_latency_seconds_p50 $p50"
    echo "ai_latency_seconds_p95 $p95"
    echo "ai_latency_seconds_max $max"
    echo "ai_latency_seconds_mean $mean"
  }
  echo "# HELP ai_host_load1 Host 1-minute load average."
  echo "# TYPE ai_host_load1 gauge"
  echo "ai_host_load1 $load1"

  echo "# HELP ai_mem_available_kib MemAvailable from /proc/meminfo."
  echo "# TYPE ai_mem_available_kib gauge"
  echo "ai_mem_available_kib $mem_avail_kb"

  echo "# HELP ai_mem_total_kib MemTotal from /proc/meminfo."
  echo "# TYPE ai_mem_total_kib gauge"
  echo "ai_mem_total_kib $mem_total_kb"

  echo "# HELP ai_disk_root_used_percent Root filesystem used percent."
  echo "# TYPE ai_disk_root_used_percent gauge"
  echo "ai_disk_root_used_percent $disk_root_used_pct"

  echo "# HELP ai_process_count Count of AI processes matching AI_PROCESS_NAME."
  echo "# TYPE ai_process_count gauge"
  echo "ai_process_count $proc_count"

  [[ -n "$gpu_block" ]] && echo "$gpu_block"
  [[ -n "$custom_block" ]] && echo "$custom_block"

  echo "# HELP ai_exporter_last_scrape Unix timestamp of last exporter run."
  echo "# TYPE ai_exporter_last_scrape gauge"
  echo "ai_exporter_last_scrape $(now_ts)"
} >"$TMP_FILE"

mv "$TMP_FILE" "$OUT_FILE"
EOF
sudo chmod +x /usr/local/lib/ai-monitor/ai-metrics.sh

Run once to test:

sudo AI_ENDPOINT=http://127.0.0.1:8000/health /usr/local/lib/ai-monitor/ai-metrics.sh
cat /var/lib/node_exporter/textfile/ai-metrics.prom

2) Log-based anomaly detection with alerts (Slack/Telegram)

This service counts errors in the past minute from your systemd unit logs, keeps a rolling window, computes a z-score, and alerts if it spikes.

Create the anomaly detector:

sudo tee /usr/local/lib/ai-monitor/ai-log-anomaly.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

SERVICE_NAME="${SERVICE_NAME:-ai.service}"         # Your model systemd unit
WINDOW="${WINDOW:-30}"                             # rolling minutes
THRESH_Z="${THRESH_Z:-3.0}"                        # alert above z-score
ABS_THRESH="${ABS_THRESH:-10}"                     # or if errors > this in one minute
STATE_DIR="/var/lib/ai-monitor"
STATE_FILE="$STATE_DIR/error-counts.txt"
OUT_FILE="/var/lib/node_exporter/textfile/ai-log.prom"
TMP_FILE="$(mktemp)"

# Optional alerts
SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL:-}"
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN:-}"
TELEGRAM_CHAT_ID="${TELEGRAM_CHAT_ID:-}"

mkdir -p "$STATE_DIR"

err_count=$(journalctl -u "$SERVICE_NAME" --since "1 min ago" --no-pager 2>/dev/null \
  | grep -E -c "ERROR|Exception|Traceback|RuntimeError" || true)

# Update rolling window
{ cat "$STATE_FILE" 2>/dev/null; echo "$err_count"; } | tail -n "$WINDOW" > "$STATE_FILE.tmp"
mv "$STATE_FILE.tmp" "$STATE_FILE"

read -r mean sd <<<"$(awk '{s+=$1; ss+=$1*$1} END{n=NR; if(n==0){print "0 0"; exit} mean=s/n; var=(ss/n)-(mean*mean); if(var<0)var=0; sd=sqrt(var); print mean, sd }' "$STATE_FILE")"
z="0"
if awk "BEGIN{exit !($sd>0)}"; then
  z=$(awk -v x="$err_count" -v m="$mean" -v s="$sd" 'BEGIN{ printf("%.3f", (x-m)/s) }')
fi

# Write metrics textfile
{
  echo "# HELP ai_log_errors_last_minute Error lines seen in the last minute."
  echo "# TYPE ai_log_errors_last_minute gauge"
  echo "ai_log_errors_last_minute $err_count"

  echo "# HELP ai_log_error_rate_zscore Rolling z-score of error counts."
  echo "# TYPE ai_log_error_rate_zscore gauge"
  echo "ai_log_error_rate_zscore $z"
} > "$TMP_FILE"
mv "$TMP_FILE" "$OUT_FILE"

alert_needed=0
awk -v z="$z" -v tz="$THRESH_Z" 'BEGIN{ if (z>tz) exit 0; else exit 1 }' && alert_needed=1
if [[ "$err_count" -gt "$ABS_THRESH" ]]; then alert_needed=1; fi

send_alert() {
  local msg="$1"
  if [[ -n "$SLACK_WEBHOOK_URL" ]]; then
    curl -s -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"${msg}\"}" "$SLACK_WEBHOOK_URL" >/dev/null || true
  fi
  if [[ -n "$TELEGRAM_BOT_TOKEN" && -n "$TELEGRAM_CHAT_ID" ]]; then
    curl -s "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
      -d "chat_id=${TELEGRAM_CHAT_ID}" -d "text=${msg}" >/dev/null || true
  fi
}

if [[ "$alert_needed" -eq 1 ]]; then
  hostname="$(hostname -s)"
  send_alert "AI anomaly on ${hostname}/${SERVICE_NAME}: errors_last_min=${err_count}, zscore=${z} (mean=${mean}, sd=${sd})"
fi
EOF
sudo chmod +x /usr/local/lib/ai-monitor/ai-log-anomaly.sh

Test it:

sudo SERVICE_NAME=ai.service /usr/local/lib/ai-monitor/ai-log-anomaly.sh
cat /var/lib/node_exporter/textfile/ai-log.prom

Tip: Set these environment variables on the systemd service to enable alerts:

  • SLACK_WEBHOOK_URL

  • TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID

3) Schedule with systemd timers

Create systemd units for periodic runs:

ai-metrics.timer + service (every 15s):

sudo tee /etc/systemd/system/ai-metrics.service >/dev/null <<'EOF'
[Unit]
Description=AI metrics exporter (Bash)

[Service]
Type=oneshot
Environment=AI_ENDPOINT=http://127.0.0.1:8000/health
Environment=AI_PROCESS_NAME=python
# Optionally: Environment=CUSTOM_JSON_METRICS_URL=http://127.0.0.1:8000/status
ExecStart=/usr/local/lib/ai-monitor/ai-metrics.sh
EOF

sudo tee /etc/systemd/system/ai-metrics.timer >/dev/null <<'EOF'
[Unit]
Description=Run AI metrics exporter every 15s

[Timer]
OnBootSec=10s
OnUnitActiveSec=15s
Unit=ai-metrics.service

[Install]
WantedBy=timers.target
EOF

ai-log-anomaly.timer + service (every minute):

sudo tee /etc/systemd/system/ai-log-anomaly.service >/dev/null <<'EOF'
[Unit]
Description=AI log anomaly detector (Bash)

[Service]
Type=oneshot
Environment=SERVICE_NAME=ai.service
# Optionally: Environment=SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
# Optionally: Environment=TELEGRAM_BOT_TOKEN=123456:ABC...  Environment=TELEGRAM_CHAT_ID=-100123456
ExecStart=/usr/local/lib/ai-monitor/ai-log-anomaly.sh
EOF

sudo tee /etc/systemd/system/ai-log-anomaly.timer >/dev/null <<'EOF'
[Unit]
Description=Run AI log anomaly detector every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
Unit=ai-log-anomaly.service

[Install]
WantedBy=timers.target
EOF

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-metrics.timer ai-log-anomaly.timer
systemctl list-timers | grep ai-

4) Visualize via Prometheus node_exporter’s textfile collector

We’ll run node_exporter and point its textfile collector at our directory. This keeps your scripts tiny and lets Prometheus/Grafana read them alongside standard host metrics.

Install node_exporter (manual, works on any distro):

NODE_EXPORTER_VERSION="1.7.0"
ARCH="$(uname -m)"
case "$ARCH" in
  x86_64)   ARCH="amd64" ;;
  aarch64)  ARCH="arm64" ;;
  ppc64le)  ARCH="ppc64le" ;;
  s390x)    ARCH="s390x" ;;
  *) echo "Unsupported arch: $ARCH"; exit 1 ;;
esac

cd /tmp
curl -fsSLO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}.tar.gz"
tar -xzf "node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}.tar.gz"
sudo mv "node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}/node_exporter" /usr/local/bin/
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter || true
sudo mkdir -p /var/lib/node_exporter/textfile
sudo chown -R node_exporter:node_exporter /var/lib/node_exporter

Systemd unit for node_exporter:

sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.textfile.directory=/var/lib/node_exporter/textfile

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Verify metrics:

curl -s http://127.0.0.1:9100/metrics | grep ^ai_

Add your Prometheus server to scrape 127.0.0.1:9100 (or the host IP). Then build Grafana panels from these time series:

  • ai_latency_seconds_p50 / ai_latency_seconds_p95

  • ai_log_errors_last_minute / ai_log_error_rate_zscore

  • ai_host_load1, ai_mem_available_kib, ai_disk_root_used_percent

  • ai_gpu_utilization_percent, ai_gpu_memory_used_mebibytes (if available)

  • ai_queue_depth, ai_qps (if you expose custom JSON metrics)

Real-world example: FastAPI model with health and custom status

If your FastAPI app exposes:

  • GET /health → 200 OK

  • GET /status → {"queue_depth": 3, "qps": 85.2}

You can set:

sudo systemctl edit ai-metrics.service
# Add under [Service]:
# Environment=AI_ENDPOINT=http://127.0.0.1:8000/health
# Environment=CUSTOM_JSON_METRICS_URL=http://127.0.0.1:8000/status
sudo systemctl daemon-reload
sudo systemctl restart ai-metrics.timer

For logs, if your systemd unit is named ai.service, the anomaly script already targets it. Otherwise:

sudo systemctl edit ai-log-anomaly.service
# [Service]
# Environment=SERVICE_NAME=your-app.service
sudo systemctl daemon-reload
sudo systemctl restart ai-log-anomaly.timer

Why this approach works

  • Minimal and auditable: pure Bash, a couple of common packages, and systemd timers you already use.

  • Extensible: add any metric you can compute in Bash; the textfile collector will pick it up.

  • Portable: runs the same on Debian/Ubuntu, Fedora/RHEL, and SUSE without distro-specific agents.

  • Production-friendly: Prometheus/Grafana ready, and alerts via Slack/Telegram without server plugins.

Conclusion and next steps (CTA)

You now have a lightweight AI monitoring project that:

  • Measures live latency and host/GPU health

  • Flags error anomalies automatically

  • Exposes clean Prometheus metrics for dashboards and alerting

Next steps: 1) Install the prerequisites: - apt: sudo apt update && sudo apt install -y curl jq bc gawk - dnf: sudo dnf install -y curl jq bc gawk - zypper: sudo zypper refresh && sudo zypper install -y curl jq bc gawk 2) Drop in the two scripts, enable the timers, and start node_exporter. 3) Point Prometheus at 127.0.0.1:9100 and build a Grafana dashboard for your ai_* metrics. 4) Add alerts to Slack/Telegram by setting the environment variables on the anomaly service.

Treat this as your starter kit. As your AI stack evolves—multiple models, GPU pools, queues—extend the scripts: capture per-model QPS, batch wait times, cache hit rates, or vector DB latency. Share your tweaks and patterns with the team so every deployment is observable from day one.

Related posts