Posted on
• Artificial Intelligence

Artificial Intelligence Monitoring Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

The Bash-Friendly AI Monitoring Checklist: What to Watch, Why It Matters, and How To Ship It Today

Artificial Intelligence systems rarely fail loudly. They drift. They slow down. They silently chew through GPUs while returning “successful” responses that are subtly wrong. If you run AI in production on Linux, you need a monitoring checklist that’s practical, automatable, and Bash-first.

This post gives you a compact, field-tested checklist for AI monitoring, why each item matters, and ready-to-run Bash snippets to get you started. You’ll also get package installation commands for apt, dnf, and zypper wherever tools are cited.

Why this matters

  • AI workloads are resource-hungry. A single runaway job can starve your cluster and spike costs.

  • Models degrade over time. Inputs shift, distributions drift, and performance dips without tripping errors.

  • Latency and throughput determine user experience. You need SLOs that measure reality, not hope.

  • Reproducibility is your parachute. When a model misbehaves, you’ll want the code/data/version fingerprint immediately.

A solid monitoring checklist lets you catch issues early, prove causality faster, and keep performance predictable.

Quick install: the essentials used below

Install the core tools you’ll use in this post.

  • Debian/Ubuntu (apt):
sudo apt update
# Choose Docker or Podman (either works for the containerized examples)
sudo apt install -y curl jq bc sysstat docker.io
# Or, if you prefer Podman:
# sudo apt install -y curl jq bc sysstat podman
# Optional: Python if you plan to script beyond Bash
sudo apt install -y python3 python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf -y install curl jq bc sysstat docker
# Or, if you prefer Podman:
# sudo dnf -y install curl jq bc sysstat podman
sudo dnf -y install python3 python3-pip
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq bc sysstat docker
# Or, if you prefer Podman:
# sudo zypper install -y curl jq bc sysstat podman
sudo zypper install -y python3 python3-pip

Note on GPUs: nvidia-smi comes with the NVIDIA driver packages, which vary by distro and driver version. If present, the scripts below will use it; if not, they’ll skip GPU collection gracefully.

The AI Monitoring Checklist (with Bash you can copy/paste)

Here are five actionable checks you can run today. Each includes what to watch, why it matters, and a practical Bash snippet.

1) Watch the host and accelerators (CPU, RAM, Disk I/O, GPU, Net)

  • Why: Resource saturation is the root cause of most “my model is slow” incidents.

  • What: CPU load, memory pressure, disk I/O wait, network saturation, and GPU utilization/memory.

Try this minimal sampler. It logs system and (optionally) GPU metrics and can also emit Prometheus textfile metrics if you use node_exporter later.

#!/usr/bin/env bash
# ai-sys-sample.sh
set -euo pipefail

TS="$(date -Is)"
HOST="$(hostname)"

# CPU load averages
read -r LA1 LA5 LA15 _ < /proc/loadavg

# Memory (kB)
MEM_TOTAL=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
MEM_AVAILABLE=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
MEM_USED=$((MEM_TOTAL - MEM_AVAILABLE))

# Disk I/O (requires sysstat for iostat, else skip)
if command -v iostat >/dev/null 2>&1; then
  IO=$(iostat -dx 1 1 | awk '/^[a-z]/ {printf "%s:%s:util=%s,await=%s; ", NR,$1,$NF,$(NF-3)}')
else
  IO="iostat_not_available"
fi

# Network bytes (aggregate)
NET_RX=$(awk '/:/{sum+=$2} END{print sum}' /proc/net/dev)
NET_TX=$(awk '/:/{sum+=$10} END{print sum}' /proc/net/dev)

# GPU (optional nvidia-smi)
GPU_SUMMARY="no_gpu"
if command -v nvidia-smi >/dev/null 2>&1; then
  GPU_SUMMARY=$(nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total --format=csv,noheader | tr '\n' ';' )
fi

LOG_LINE="$TS host=$HOST la1=$LA1 la5=$LA5 la15=$LA15 mem_used_kb=$MEM_USED mem_total_kb=$MEM_TOTAL net_rx_bytes=$NET_RX net_tx_bytes=$NET_TX io=[$IO] gpu=[$GPU_SUMMARY]"
echo "$LOG_LINE"

# Optional: write Prometheus textfile metrics for node_exporter textfile collector
# Uncomment and adjust path if using node_exporter
# OUT="/var/lib/node_exporter/textfile_collector/ai.prom"
# sudo mkdir -p "$(dirname "$OUT")"
# cat > /tmp/ai.prom.$$ <<EOF
# ai_la1 $LA1
# ai_la5 $LA5
# ai_la15 $LA15
# ai_mem_used_kb $MEM_USED
# ai_mem_total_kb $MEM_TOTAL
# ai_net_rx_bytes $NET_RX
# ai_net_tx_bytes $NET_TX
# EOF
# sudo mv /tmp/ai.prom.$$ "$OUT"
  • Run it periodically:
chmod +x ai-sys-sample.sh
# Every minute via cron
( crontab -l 2>/dev/null; echo '* * * * * /path/to/ai-sys-sample.sh >> /var/log/ai-sys.log 2>&1' ) | crontab -

Interpretation tip: rising la15, high %iowait (visible in iostat output), or sustained GPU utilization near 100% can all correlate with latency spikes.

2) Check your inference SLOs (availability and latency) at the edge

  • Why: Users experience availability and latency, not “service is up” booleans.

  • What: HTTP status code, total time to first byte/total time, and payload sanity.

Use this Bash health probe to log status and latency and exit non-zero on SLO violations (handy for alerting):

#!/usr/bin/env bash
# ai-http-check.sh
set -euo pipefail

URL="${URL:-http://localhost:8080/health}"
EXPECT_STATUS="${EXPECT_STATUS:-200}"
MAX_LATENCY_S="${MAX_LATENCY_S:-0.300}"   # 300ms default
TIMEOUT_S="${TIMEOUT_S:-2}"               # curl timeout

OUT=$(mktemp)
trap 'rm -f "$OUT"' EXIT

# -sS: silent but show errors; -m: max time; -o: body to file; -w: write HTTP code + time
read -r CODE TOTAL <<<"$(curl -sS -m "$TIMEOUT_S" -o "$OUT" -w "%{http_code} %{time_total}" "$URL")" || CODE=0

TS="$(date -Is)"
if [ "$CODE" != "$EXPECT_STATUS" ]; then
  echo "$TS url=$URL status=$CODE total_s=$TOTAL ERROR=bad_status"
  exit 2
fi

awk -v t="$TOTAL" -v max="$MAX_LATENCY_S" -v ts="$TS" -v url="$URL" '
BEGIN {
  if (t > max) {
    printf "%s url=%s status=OK total_s=%.3f ERROR=slow\n", ts, url, t;
    exit 1
  } else {
    printf "%s url=%s status=OK total_s=%.3f\n", ts, url, t;
    exit 0
  }
}'
  • Example runs:
URL=http://127.0.0.1:8000/v1/ping EXPECT_STATUS=200 MAX_LATENCY_S=0.200 ./ai-http-check.sh
  • Automate it:
( crontab -l 2>/dev/null; echo '* * * * * URL=https://api.your-ai.com/health MAX_LATENCY_S=0.250 /path/ai-http-check.sh >> /var/log/ai-http.log 2>&1' ) | crontab -

Pro tip: Keep per-endpoint SLOs: e.g., “P95 inference < 250ms for prompt<=1KB.” Your alerts should reflect those.

3) Add a basic data-drift sentinel (feature sanity checks)

  • Why: A healthy service with unhealthy inputs still produces bad outputs. Small, automated checks catch data skew early.

  • What: Track summary stats (min/mean/max) for key numeric inputs and compare to a known-good baseline.

This simple script compares live samples against a JSON baseline, logging a warning on deviations.

#!/usr/bin/env bash
# ai-drift-check.sh
# Expect JSON lines on stdin like: {"feature_a": 12.3, "feature_b": 7}
# Baseline file: {"feature_a":{"mean":10.0,"std":2.0},"feature_b":{"mean":6.5,"std":1.0}}

set -euo pipefail
BASELINE="${BASELINE:-/etc/ai/baseline.json}"
FEATURES="${FEATURES:-feature_a,feature_b}"
WINDOW="${WINDOW:-100}"  # number of samples to aggregate

if ! [ -f "$BASELINE" ]; then
  echo "Missing baseline at $BASELINE" >&2
  exit 1
fi

TMP=$(mktemp)
trap 'rm -f "$TMP"' EXIT

# Read up to WINDOW lines of JSON input
head -n "$WINDOW" > "$TMP"

IFS=',' read -ra FEATS <<< "$FEATURES"
TS="$(date -Is)"
for f in "${FEATS[@]}"; do
  # Compute mean using jq + awk
  COUNT=$(jq -r --arg f "$f" 'select(has($f)) | .[$f]' "$TMP" | wc -l | tr -d ' ')
  if [ "$COUNT" -eq 0 ]; then
    echo "$TS feature=$f WARNING=no_data"
    continue
  fi
  MEAN=$(jq -r --arg f "$f" 'select(has($f)) | .[$f]' "$TMP" | awk '{sum+=$1} END{if (NR>0) printf "%.6f", sum/NR; else print "nan"}')
  BASE_MEAN=$(jq -r --arg f "$f" '.[$f].mean' "$BASELINE")
  BASE_STD=$(jq -r --arg f "$f" '.[$f].std' "$BASELINE")

  # Z-score style check: flag if |mean - base_mean| > 3 * std (adjust as needed)
  DIFF=$(awk -v a="$MEAN" -v b="$BASE_MEAN" 'BEGIN{printf "%.6f", (a-b)>=0?(a-b):(b-a)}')
  THR=$(awk -v s="$BASE_STD" 'BEGIN{printf "%.6f", 3*s}')
  ALERT=$(awk -v d="$DIFF" -v t="$THR" 'BEGIN{print (d>t)?"YES":"NO"}')

  if [ "$ALERT" = "YES" ]; then
    echo "$TS feature=$f mean_now=$MEAN mean_base=$BASE_MEAN std_base=$BASE_STD ALERT=drift_suspected"
  else
    echo "$TS feature=$f mean_now=$MEAN mean_base=$BASE_MEAN std_base=$BASE_STD status=ok"
  fi
done

Usage example:

# Generate or update your baseline from a known-good window
# (example: compute mean/std offline and write /etc/ai/baseline.json)

# Then stream current traffic through the checker:
tail -n +1 -F /var/log/ai/inputs.jsonl | BASELINE=/etc/ai/baseline.json FEATURES=feature_a,feature_b WINDOW=200 ./ai-drift-check.sh >> /var/log/ai-drift.log

Start simple: a couple of high-signal features can catch many upstream issues.

4) Make every run reproducible (version, config, and evidence)

  • Why: When incidents hit, you need to answer “exactly which model/code/config was this request served by?” fast.

  • What: Embed model version, git SHA, and config hash into your process and logs.

Use a systemd unit that pins version metadata into environment variables and makes them queryable via a health endpoint or logs.

# /etc/systemd/system/ai-infer.service
[Unit]
Description=AI Inference Service
After=network.target

[Service]
User=ai
Group=ai
Environment=MODEL_VERSION=2026.07.10
Environment=GIT_SHA=abcdef1234567890
Environment=CONFIG_SHA=cfg-9988aabbcc
Environment=TZ=UTC
WorkingDirectory=/opt/ai
ExecStart=/opt/ai/bin/infer-server --port 8000
Restart=on-failure
RestartSec=2
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Reload and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-infer.service
journalctl -u ai-infer.service -f

Expose these environment variables via your /health or /version endpoint and echo them in startup logs. That alone cuts root-cause time dramatically.

5) Visualize and alert quickly (Prometheus + Grafana in containers)

  • Why: Logs are great; dashboards and alerts catch trends and page humans.

  • What: Use containers to get Prometheus, node_exporter, and Grafana up fast without distro-specific packages.

Start Docker or Podman:

  • Docker:
# apt:   sudo systemctl enable --now docker
# dnf:   sudo systemctl enable --now docker
# zypper:sudo systemctl enable --now docker
  • Or Podman (rootless example):
# apt/dnf/zypper all support:
podman --version

Run node_exporter (host metrics).

  • Docker:
docker run -d --name node_exporter --restart=unless-stopped \
  -p 9100:9100 \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /:/rootfs:ro \
  quay.io/prometheus/node-exporter:latest \
  --path.procfs=/host/proc --path.sysfs=/host/sys --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($|/)"
  • Podman:
podman run -d --name node_exporter --restart=always \
  -p 9100:9100 \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /:/rootfs:ro \
  quay.io/prometheus/node-exporter:latest \
  --path.procfs=/host/proc --path.sysfs=/host/sys --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($|/)"

Create a minimal Prometheus config:

mkdir -p ~/prometheus
cat > ~/prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['host.docker.internal:9100', 'localhost:9100']
  - job_name: 'ai-http-check'
    static_configs:
      - targets: ['localhost:9101']  # example if you later expose custom metrics
EOF

Run Prometheus:

  • Docker:
docker run -d --name prometheus --restart=unless-stopped \
  -p 9090:9090 \
  -v ~/prometheus:/etc/prometheus \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml
  • Podman:
podman run -d --name prometheus --restart=always \
  -p 9090:9090 \
  -v ~/prometheus:/etc/prometheus \
  docker.io/prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml

Run Grafana:

  • Docker:
docker run -d --name grafana --restart=unless-stopped -p 3000:3000 grafana/grafana
  • Podman:
podman run -d --name grafana --restart=always -p 3000:3000 docker.io/grafana/grafana

Then:

  • Open Grafana on http://localhost:3000 (default admin/admin).

  • Add Prometheus at http://localhost:9090 as a data source.

  • Import a Node Exporter dashboard (search gallery) and add your custom panels for:

    • ai_sys: load, memory, I/O wait
    • GPU utilization (via textfile exporter if you enable it)
    • Inference latency from logs (exported through a small sidecar or custom metrics endpoint)

Optional: simple alert with Prometheus rules (save as ~/prometheus/alerts.yml and reference via --rule.files):

groups:

- name: ai-latency
  rules:
  - alert: InferenceLatencyHigh
    expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="ai"}[5m])) by (le)) > 0.25
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High p95 inference latency"
      description: "p95 latency > 250ms for 10m"

Hook this to email/Slack/PagerDuty via Alertmanager when you’re ready.

A compact, real-world AI monitoring checklist

Use this before and after every deploy:

  • Capacity and hardware

    • CPU load and %iowait are within historical bands
    • Memory headroom > 20%, no swapping under load
    • GPU util < 85% sustained unless intended; GPU mem headroom > 10%
    • Disk I/O latency stable; network egress not throttled

  • Service SLOs

    • Health endpoint status=200
    • p95 latency within target for realistic payload sizes
    • Error rate and timeouts near zero

  • Data sanity

    • Key input features within baseline ranges
    • Payload shapes/sizes within expected bounds
    • Spike detection on traffic volume

  • Reproducibility

    • Model version, git SHA, and config hash embedded and visible
    • Artifact checksums recorded
    • Change log links from dashboards to releases

  • Observability plumbing

    • Node and app metrics scraping confirmed in Prometheus
    • Dashboards pinned to SLOs
    • Alerts tested with known-bad scenarios

Conclusion and next step

AI systems don’t fail in one place—they fade across systems, data, and code. A Bash-first checklist, coupled with a few lightweight tools, gives you fast visibility and a reliable early-warning system.

Your next step:

  • Put ai-sys-sample.sh and ai-http-check.sh on one production host today.

  • Stand up Prometheus and Grafana with Docker/Podman.

  • In a week, capture baselines and wire one alert.

Small steps compound. Start monitoring what matters and make silent failures loud—before your users find them.

Related posts