Artificial Intelligence Linux Monitoring Best Practices (with Bash you’ll actually use)

You kicked off a training job at 6 PM. By 9 AM, it’s still crawling, GPUs are at 20%, the CPUs are pegged, and your logs show out-of-memory thrash at 3 AM. Sound familiar? AI workloads are hungry, spiky, and unforgiving. The value of good monitoring isn’t dashboards for their own sake; it’s fewer failed runs, more predictable throughput, and faster feedback loops.

This guide shows you how to monitor AI workloads on Linux the right way—using tools you can install in minutes and automate with Bash. You’ll get concrete steps, commands, and real-world checks that surface the bottlenecks that matter: GPU saturation, memory pressure, I/O stalls, and network choke points.

Why AI monitoring is different on Linux

• GPU-centric bottlenecks: High GPU memory usage with low compute utilization often means your input pipeline is the problem (I/O or CPU-bound data loaders).

• Bursty and multi-resource: Training alternates between computation and synchronization. Spikes in I/O, network, and CPU don’t neatly align.

• Distributed quirks: One “cold” node with slow storage or thermal throttling can drag down the whole cluster.

• Containers and cgroups: Isolation helps, but hides issues unless you monitor cgroup-level resource use.

Below are five actionable practices (with commands) to get from blind spots to insight quickly.

---

0) Install the essentials (5 minutes)

We’ll use a small, proven toolset: sysstat (sar/iostat/pidstat), htop, iotop, iftop, bpftrace, and Prometheus Node Exporter. On RHEL/CentOS, enable EPEL first for some packages.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat htop iotop iftop bpftrace prometheus-node-exporter
# Enable sysstat data collection (Debian/Ubuntu):
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat || true
sudo systemctl enable --now sysstat
sudo systemctl enable --now prometheus-node-exporter

Fedora/RHEL/CentOS (dnf):

# On RHEL/CentOS:
# sudo dnf install -y epel-release

sudo dnf install -y sysstat htop iotop iftop bpftrace golang-github-prometheus-node_exporter
sudo systemctl enable --now sysstat
sudo systemctl enable --now node_exporter

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y sysstat htop iotop iftop bpftrace prometheus-node_exporter
sudo systemctl enable --now sysstat
sudo systemctl enable --now prometheus-node_exporter

GPU note: nvidia-smi comes with the NVIDIA driver (install via NVIDIA’s repo/driver installer). AMD users can use rocm-smi (in ROCm packages).

---

1) Define AI-specific SLOs and “golden signals”

Don’t watch everything—watch what predicts training/inference health.

• Throughput and latency: samples/sec, tokens/sec, per-batch latency.

• GPU: utilization (%), memory used (MiB), power draw, temperature, ECC errors.

• CPU: user/sys %, steal %, run queue length, softirqs.

• Memory: available memory, page faults, swap activity, OOM kills.

• Disk I/O: device utilization (%util), queue size (avgqu-sz), latency (r_await/w_await), read/write throughput.

• Network: throughput, retransmits, latency (p99 if possible).

• Container/cgroup: per-job CPU quota usage, memory usage, throttling.

Quick Bash health check before/while running a job:

echo "=== CPU ==="
mpstat 1 3

echo "=== Memory & Swap ==="
free -h
grep -i oom /var/log/kern.log 2>/dev/null || sudo journalctl -k -g OOM -n 5

echo "=== Disk I/O (extended) ==="
iostat -xz 1 3

echo "=== Network (top 3 interfaces) ==="
sar -n DEV 1 3 | awk '/Average|IFACE|eth|ens|eno/'

echo "=== GPU (NVIDIA) ==="
nvidia-smi --query-gpu=name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu --format=csv

---

2) Baseline with battle-tested tools

Use these to quickly answer “what’s slow and why?” in a single node scenario.

• CPU saturation and per-PID analysis:

htop     # interactive view; press F6 to sort by various metrics
pidstat -durh 1    # per-process CPU (-u), I/O (-d), page faults (-r) every 1s

• I/O pressure and latency:

iostat -xz 1       # extended stats per device; watch %util and r_await/w_await
iotop -oPa         # top I/O-consuming processes (-a: accumulated)

• Network hotspots:

sudo iftop -i $(ip -o -4 route show to default | awk '{print $5}')  # top talkers
ss -tni | awk 'NR==1 || /ESTAB/'  # established TCP sockets with details

• sysstat time series (already collecting):

sar -q 1 5       # run queue length and load averages
sar -r 1 5       # memory usage stats
sar -n TCP,DEV 1 5

• BPF one-liners for deeper insight (requires root):

# Top syscalls by process over 5 seconds
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[comm] = count(); } interval:s:5 { exit(); }'

# File read latency histogram (system-wide)
sudo bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @lat = hist((nsecs-@start[tid])/1000000); delete(@start[tid]); } interval:s:5 { print(@lat); clear(@lat); }'

If you see:

• High GPU memory used but low GPU utilization: likely CPU/I/O pipeline bottleneck.

• High %util and r_await on storage: dataset storage is the limiter.

• High steal in mpstat: noisy neighbor or overcommitted virtualization.

---

3) Export metrics for dashboards and alerts (Prometheus-friendly)

Node Exporter exposes host metrics on port 9100.

Verify it’s running:

curl -s http://localhost:9100/metrics | head

If you want to add custom AI/GPU metrics without writing code, use the textfile collector. Many distros ship Node Exporter with the textfile collector disabled by default. Create a systemd drop-in to enable it.

Debian/Ubuntu:

sudo mkdir -p /etc/systemd/system/prometheus-node-exporter.service.d
cat | sudo tee /etc/systemd/system/prometheus-node-exporter.service.d/textfile.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-node-exporter --collector.textfile.directory=/var/lib/node_exporter/textfile
EOF
sudo mkdir -p /var/lib/node_exporter/textfile
sudo systemctl daemon-reload
sudo systemctl restart prometheus-node-exporter

Fedora/RHEL/CentOS:

sudo mkdir -p /etc/systemd/system/node_exporter.service.d
cat | sudo tee /etc/systemd/system/node_exporter.service.d/textfile.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile
EOF
sudo mkdir -p /var/lib/node_exporter/textfile
sudo systemctl daemon-reload
sudo systemctl restart node_exporter

openSUSE/SLES:

sudo mkdir -p /etc/systemd/system/prometheus-node_exporter.service.d
cat | sudo tee /etc/systemd/system/prometheus-node_exporter.service.d/textfile.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/sbin/prometheus-node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile
EOF
sudo mkdir -p /var/lib/node_exporter/textfile
sudo systemctl daemon-reload
sudo systemctl restart prometheus-node_exporter

Add a simple GPU exporter using nvidia-smi + textfile collector:

sudo tee /usr/local/bin/gpu_textfile.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
out="/var/lib/node_exporter/textfile/gpu.prom"
tmp="$(mktemp)"
now="$(date +%s)"
# Requires NVIDIA driver (nvidia-smi)
if ! command -v nvidia-smi >/dev/null; then
  exit 0
fi
# Query per-GPU metrics
mapfile -t lines < <(nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits)
{
  echo "# HELP nvidia_gpu_utilization GPU utilization percent"
  echo "# TYPE nvidia_gpu_utilization gauge"
  echo "# HELP nvidia_gpu_memory_used_bytes GPU memory used in bytes"
  echo "# TYPE nvidia_gpu_memory_used_bytes gauge"
  echo "# HELP nvidia_gpu_memory_total_bytes GPU memory total in bytes"
  echo "# TYPE nvidia_gpu_memory_total_bytes gauge"
  echo "# HELP nvidia_gpu_temperature_celsius GPU temperature in Celsius"
  echo "# TYPE nvidia_gpu_temperature_celsius gauge"
  echo "# HELP nvidia_gpu_power_draw_watts GPU power draw in watts"
  echo "# TYPE nvidia_gpu_power_draw_watts gauge"
  for l in "${lines[@]}"; do
    IFS=',' read -r idx u_gpu u_mem mem_used mem_total temp_c pwr <<<"$l"
    idx="$(echo "$idx" | xargs)"
    echo "nvidia_gpu_utilization{gpu=\"$idx\"} $u_gpu"
    echo "nvidia_gpu_utilization_memory{gpu=\"$idx\"} $u_mem"
    echo "nvidia_gpu_memory_used_bytes{gpu=\"$idx\"} $(awk "BEGIN{printf \"%.0f\", $mem_used*1024*1024}")"
    echo "nvidia_gpu_memory_total_bytes{gpu=\"$idx\"} $(awk "BEGIN{printf \"%.0f\", $mem_total*1024*1024}")"
    echo "nvidia_gpu_temperature_celsius{gpu=\"$idx\"} $temp_c"
    echo "nvidia_gpu_power_draw_watts{gpu=\"$idx\"} $pwr"
  done
  echo "nvidia_gpu_scrape_timestamp_seconds $now"
} > "$tmp"
mv "$tmp" "$out"
EOF
sudo chmod +x /usr/local/bin/gpu_textfile.sh
echo "*/1 * * * * root /usr/local/bin/gpu_textfile.sh" | sudo tee /etc/cron.d/gpu_textfile >/dev/null

Now Prometheus can scrape Node Exporter plus your GPU metrics. Set alerts like:

• GPU below 30% utilization for 10m while CPU iowait > 10% (pipeline bound).

• Memory available < 2 GiB or recent OOM events.

• Disk device %util > 80% for 5m or queue size > 1 (sustained saturation).

Tip: For NVIDIA DCGM exporter (richer GPU telemetry), install via NVIDIA’s repos and run alongside Node Exporter.

---

4) Troubleshoot hotspots with fast one-liners

• Is CPU the bottleneck in Python data loading?

pidstat -t -C python -ru 1

Look for high %usr/%sys and run-queue growth.

• Are we I/O bound on the dataset volume?

iostat -xz 1

High %util and r_await indicate storage bottlenecks; consider local NVMe caching or parallel prefetching.

• Are OOM kills ruining runs?

sudo journalctl -k -g "Out of memory|Killed process" -n 20

• Is the network link saturated (distributed training)?

sar -n DEV 1 5 | awk '/Average|IFACE|eth|ens|eno/'
ss -ti '( dport = :ssh or dport = :nccl )' 2>/dev/null || true

• Are syscalls or kernel paths expensive?

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); } interval:s:10 { print(@); clear(@); }'

---

5) Capacity planning and isolation (stop noisy neighbors)

Pin, quota, and contain your jobs so they don’t step on each other. This also makes metrics easier to reason about.

• Run a training job with cgroup v2 limits (systemd-run):

sudo systemd-run --unit=trainA --scope \
  -p CPUQuota=400% \
  -p MemoryMax=64G \
  -p MemorySwapMax=0 \
  -p IOReadBandwidthMax=/dev/nvme0n1 1G \
  env CUDA_VISIBLE_DEVICES=0,1 \
  python train.py --config config.yaml

• NUMA-aware placement (if dual-socket systems):

numactl --cpunodebind=0 --membind=0 \
  env CUDA_VISIBLE_DEVICES=0 python train.py

• NVIDIA persistence (reduce init latency):

sudo nvidia-smi -pm 1

Watch Node Exporter’s cgroup metrics (if enabled) or scrape cgroup fs stats for per-job visibility. For containers, ensure you export cgroup and container labels into metrics.

---

Real-world mini-examples

• Symptom: GPUs at 20–40% util, CPU iowait ~15%, disk %util 90%. Fix: Convert dataset to an optimized format (e.g., WebDataset/TFRecord), use larger sequential reads, increase prefetch and workers, and cache on local NVMe.

• Symptom: Spiky OOM kills at midnight. Fix: Set MemoryMax and MemorySwapMax=0 on jobs; add alert on node_memory_MemAvailable_bytes; watch pgfault/pgmajfault via pidstat -r.

• Symptom: One node is always slower in distributed training. Fix: Compare per-node storage latency (iostat -xz), thermal throttling (nvidia-smi -q -d TEMPERATURE,POWER), and network throughput. Replace or re-balance data shards.

---

Conclusion and next steps (your 1-hour plan)

• Install the essentials with your package manager (apt/dnf/zypper).

• Start Node Exporter, enable the textfile collector, and add the gpu_textfile.sh cron.

• Pick three golden signals: GPU util, disk latency, and memory available. Add two alerts.

• Run one training job and keep iostat -xz 1 and pidstat -durh 1 up during the warmup.

• Iterate: fix the highest pain signal (I/O, CPU, or GPU feed), then re-measure.

Monitoring isn’t about more charts—it’s about faster, more reliable AI runs. Start small, automate with Bash, and grow only the parts that give you decisions.