Artificial Intelligence Monitoring Case Studies: A Linux Bash Playbook

When AI systems misbehave, they rarely send a polite email. They just get slow, stall your GPUs, or blow up memory at 3 a.m. The difference between panic and poise is monitoring. In this article, you’ll learn how small, scriptable Linux tools solve big AI problems—via four real-world case studies—and you’ll leave with concrete Bash commands you can run today.

What you’ll get:

• Why AI-specific monitoring matters on Linux boxes

• Four actionable case studies (GPU starvation, latency spikes, memory leaks, I/O bottlenecks)

• Copy-paste Bash snippets and service installs (apt, dnf, zypper)

Why monitor AI workloads differently?

• AI burns expensive resources. A 10% GPU idle rate over a month can waste thousands of dollars.

• Bottlenecks are non-obvious. GPUs sit idle while CPUs block on disk or network I/O.

• Tail latency rules UX. One slow inference ruins the P95, even if the average looks fine.

• Leaks and drifts are sneaky. Long-running trainers and model servers can slowly eat memory or swap.

The good news: 80% of issues surface at the Linux layer. With a few command-line tools and some small scripts, you can catch them early.

Install the essentials

The following tools will be used throughout:

• sysstat (sar, iostat, pidstat) for CPU/memory/I/O

• htop for interactive overviews

• nvtop for NVIDIA GPU live view

• smem for accurate per-process memory (USS/RSS)

• fio for disk benchmarking

• glances and netdata for “quick observability” dashboards

Run the commands for your distro.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat htop nvtop smem fio glances netdata

Fedora/RHEL-compatible (dnf):

sudo dnf install -y sysstat htop nvtop smem fio glances netdata

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y sysstat htop nvtop smem fio glances netdata

Optional quick-start services:

• Netdata (web UI at http://:19999):

sudo systemctl enable --now netdata

• Glances (web mode on port 61208):

glances -w --bind 0.0.0.0

Note: For NVIDIA GPUs, ensure NVIDIA drivers are installed. nvidia-smi should work:

nvidia-smi

If not, install the appropriate NVIDIA driver for your distribution/hardware.

---

Case Study 1 — “Idle GPUs”: Training cluster stalled by slow data pipeline

Symptom:

• Trainers “run,” but nvidia-smi shows low GPU utilization (e.g., <30%) while jobs claim they’re busy.

Hypothesis:

• GPUs are starved waiting for CPU/disk/network I/O. Data loaders or remote storage may be the bottleneck.

What to check:

• GPU utilization snapshot (every 2s):

watch -n2 'nvidia-smi --query-gpu=timestamp,index,utilization.gpu,utilization.memory,memory.used --format=csv,noheader,nounits'

• Disk and CPU wait (%iowait) in near real time:

iostat -xz 2

• Per-process CPU and I/O usage:

pidstat -dur 2

• Live GPU processes:

nvidia-smi pmon -s um -c 1

Action pattern that works: 1) Confirm the bottleneck: If %util on a data disk hits ~100% or %iowait spikes when batches are loaded, it’s I/O-bound. 2) Stage data locally (simple, high-impact):

# Example: stage dataset to fast local NVMe
rsync -a --info=progress2 /mnt/remote/dataset/ /local_fast/dataset/

3) Re-run and re-check GPU utilization.

Bonus: Log GPU + system metrics together for a training run:

#!/usr/bin/env bash
# log-gpu-and-sys.sh
out=${1:-gpu_sys_$(date +%F_%H%M%S).csv}
echo "ts,gpu_index,gpu_util,mem_util,mem_used_MB,cpu_usr,cpu_sys,load1,bi,bo" > "$out"
while true; do
  ts=$(date +%s)
  # GPU line(s)
  while IFS=, read -r name idx gutil mutil mused _rest; do
    # System snapshot (cpu user/sys, load, blocks in/out)
    read -r _ _ _ _ _ _ _ _ _ _ _ _ _ _ cpu_usr cpu_sys _ < <(mpstat 1 1 | awk '/all/ {print}')
    read -r load1 _ < <(awk '{print $1}' /proc/loadavg)
    read -r bi bo < <(awk '{print $6" "$7}' /proc/stat | sed -n 's/^btime.*//p'; iostat -d 1 2 | awk '/^Device/ {f=1; next} f {bi+=$6; bo+=$7} END{print bi" "bo}')
    echo "$ts,$idx,$gutil,$mutil,$mused,$cpu_usr,$cpu_sys,$load1,$bi,$bo" >> "$out"
  done < <(nvidia-smi --query-gpu=name,index,utilization.gpu,utilization.memory,memory.used --format=csv,noheader,nounits)
done

Outcome:

• Teams often discover data loading or remote storage is the limiter. Staging datasets locally, increasing data-loader workers, or precomputing features typically drives GPU utilization up and wall-clock time down.

---

Case Study 2 — “Why is P95 awful?”: Inference latency spikes due to swap pressure

Symptom:

• Intermittent high tail latency (P95/P99), especially under load, while averages look fine.

Hypothesis:

• The node is swapping or incurring major page faults when request spikes align with memory pressure.

What to check:

• Swap and paging in real time:

sar -S 1 10
vmstat 1 10
free -h

• Look for kswapd activity and high si/so (swap in/out). Also compare working set to RAM.

Quick mitigations:

• Reduce swappiness (temporary):

sudo sysctl vm.swappiness=10

• Keep your model and hot tensors resident by right-sizing container/node memory and limiting co-tenants. Keep an eye on memory with:

pidstat -r -p ALL 1

• Simple alert if swap starts moving during inference hours:

while true; do
  si_so=$(vmstat 1 2 | tail -1 | awk '{print $7+$8}')
  if [ "$si_so" -gt 0 ]; then
    echo "$(date) ALERT: Swap in/out detected ($si_so pages/sec)" | systemd-cat -t ai-infer-swap -p warning
  fi
done

Outcome:

• Pinning memory usage below swap thresholds and lowering swappiness stabilizes tail latency without code changes.

---

Case Study 3 — “It gets slower every day”: Memory leak in a model server

Symptom:

• A model API grows from 2 GB to 10+ GB RAM over days, causing OOM kills or prolonged GC pauses.

Hypothesis:

• A leak in app code or a library. You may not fix code today, but you can detect and contain it.

What to check:

• Per-process memory (USS/RSS) accurately:

smem -r -k -p | sort -h -k 4 | tail

• Single PID drilldown:

PID=<your_model_pid>
grep -E 'VmRSS|VmSize' /proc/$PID/status
pmap -x $PID | tail -n 1

• Track growth over time:

#!/usr/bin/env bash
# watch-rss.sh
PID=$1
[ -z "$PID" ] && { echo "Usage: $0 <pid>"; exit 1; }
while true; do
  rss_kb=$(grep VmRSS /proc/$PID/status | awk '{print $2}')
  echo "$(date +%s),$rss_kb" >> rss_$PID.csv
  sleep 60
done

Containment strategies:

• Proactive recycle via systemd if memory crosses a threshold (buy time while fixing the leak):

# Example systemd override for a service
sudo systemctl edit your-model.service
# Add:
# [Service]
# MemoryMax=8G
# Restart=on-failure
# RestartSec=5
sudo systemctl daemon-reload
sudo systemctl restart your-model.service

Outcome:

• Early detection prevents node-wide impact. Teams get a safe window to patch leaks while SLOs remain intact.

---

Case Study 4 — “Throughput caps out”: Feature pipeline bottlenecked by disk I/O

Symptom:

• Feature extraction or TFRecord/Parquet reading can’t saturate GPUs; %util on disks pegs at ~100%.

What to check:

• Device-level view:

iostat -dxz 2

Focus on:

• r/s, w/s, rkB/s, wkB/s

• await and svctm (latency)

• %util (busy time)

• Which process is pounding disk:

pidstat -d 2

Baseline your disk (read/write, sequential vs random) with fio:

# 1GiB sequential read
fio --name=seqread --filename=/data/testfile --size=1G --bs=1M --rw=read --direct=1 --numjobs=1 --iodepth=16

# 4k random read (IOPS)
fio --name=randread --filename=/data/testfile --size=1G --bs=4k --rw=randread --direct=1 --numjobs=4 --iodepth=32

If the workload is bottle-necked by small random reads, consider:

• Pre-sharding and sequentializing access patterns

• Local NVMe staging (as in Case Study 1)

• Increasing parallel readers carefully while watching await and %util

• Verifying filesystem mount options (e.g., noatime) for read-heavy datasets

Outcome:

• Aligning data access with disk characteristics typically yields immediate throughput gains and steadier GPU utilization.

---

Quick wins you can do today

• Baseline the node:

htop
iostat -xz 5
sar -u 5 5
free -h

• Quick GPU reality check:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv,noheader,nounits
nvtop  # interactive

• Enable a 10-minute observability path:

sudo systemctl enable --now netdata
# or
glances -w --bind 0.0.0.0

• Keep a simple log of GPU + system metrics during runs (see log-gpu-and-sys.sh above).

---

Conclusion and next steps

AI monitoring doesn’t have to start with a massive platform. With a few Linux tools and small Bash scripts, you can:

• Prove where the bottleneck really is

• Protect tail latency

• Detect memory leaks early

• Keep GPUs fed and costs under control

Your next step: 1) Install the essentials (sysstat, nvtop, smem, fio, glances, netdata) using apt/dnf/zypper above. 2) Run a training or inference session while capturing metrics (GPU + iostat + pidstat). 3) Archive your findings and set one small automation (e.g., swap alert or memory growth watcher).

If you want a deeper dive (e.g., exporting metrics to Prometheus/Grafana or adding NVIDIA DCGM exporters), say the word and I’ll share a minimal, distro-friendly setup you can roll out this week.