Posted on
• Artificial Intelligence

Artificial Intelligence Linux Monitoring Guide

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Linux Monitoring Guide: Turn GPUs, CPUs, and IO Into Actionable Insights

If your AI training jobs stall at 20% GPU, models crash at epoch 3 with OOM, or your inference latency swings wildly despite “idle” servers, the problem likely isn’t your code—it’s your visibility. AI workloads stress Linux in unique ways: enormous tensors, aggressive IO, heat, and GPUs that can silently throttle. This guide shows you how to monitor Linux for AI the way practitioners do: GPU-first, IO-aware, and alert-ready—with Bash-friendly tools and zero-nonsense steps.

What you’ll get:

  • Why AI monitoring is different (and worth your time)

  • 3–5 focused actions for real results

  • Copy/paste install and run commands (apt, dnf, zypper)

  • Ready-to-use snippets for dashboards and alerts

Why AI monitoring is different

  • GPU-first bottlenecks: A single underutilized GPU wastes hundreds of watts and $$. VRAM pressure or PCIe stalls can drop utilization without obvious logs.

  • IO makes or breaks throughput: Data loaders, feature stores, and checkpoints can choke on disk or network—showing up as “slow GPU.”

  • Thermal and power limits: Hot GPUs throttle silently. CPU throttling and RAM swaps cascade into training instability.

  • Multi-tenant chaos: One noisy neighbor saturates IO or fills page cache, tanking everyone else’s epochs.

  • Containers and schedulers: Kubernetes/Slurm add layers where exporter coverage matters.

The fix: monitor what matters, at the right resolution, with sane alerts.

1) Start GPU-first: know utilization, VRAM, clocks, and temperature

Interactive visibility is the fastest win.

  • NVIDIA: nvidia-smi (ships with the NVIDIA driver)

  • Cross-vendor TUI: nvtop

  • AMD: radeontop (works broadly); rocm-smi if ROCm is installed

Install nvtop and radeontop:

# Debian/Ubuntu
sudo apt update && sudo apt install -y nvtop radeontop

# Fedora/RHEL
sudo dnf install -y nvtop radeontop

# openSUSE
sudo zypper install -y nvtop radeontop

Quick checks during training:

# NVIDIA quick glance (per-second refresh)
watch -n 1 nvidia-smi

# Better live GPU dashboard
nvtop

# AMD live view
radeontop

Log GPU metrics to CSV (handy for post-mortems):

#!/usr/bin/env bash
# gpu_log.sh - logs key NVIDIA GPU metrics to gpu_metrics.csv every second
# Requires: nvidia-smi

echo "timestamp,gpu_index,name,util,mem_util,temp,clock_sm,mem_total,mem_used" > gpu_metrics.csv
while true; do
  nvidia-smi --query-gpu=timestamp,index,name,utilization.gpu,utilization.memory,temperature.gpu,clocks.sm,memory.total,memory.used \
             --format=csv,noheader,nounits >> gpu_metrics.csv
  sleep 1
done

Production-grade GPU metrics (Prometheus-readable) using DCGM exporter in a container:

# Requires Docker (or Podman) and NVIDIA Container Toolkit
docker run -d --name dcgm-exporter --gpus all -p 9400:9400 nvidia/dcgm-exporter:latest

What to look for:

  • Utilization < 40% during training -> data pipeline or CPU-bound

  • VRAM > 90–95% sustained -> fragmentation/OOM risk

  • Temp > 85°C -> likely throttling, check airflow/fans

  • Clocks dropping under load -> power/thermal caps

2) Baseline the box: CPU, RAM, IO, thermals

Install core CLI tools:

# sysstat: iostat, mpstat, sar
# lm-sensors: sensors, sensors-detect
# nvme-cli: NVMe drive stats (latency, health)

# Debian/Ubuntu
sudo apt update && sudo apt install -y sysstat lm-sensors nvme-cli

# Fedora/RHEL
sudo dnf install -y sysstat lm_sensors nvme-cli

# openSUSE
sudo zypper install -y sysstat lm_sensors nvme-cli

Enable and start statistics collection (sysstat):

# Debian/Ubuntu: enable daily collection
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat || true
sudo systemctl enable --now sysstat

# Fedora/RHEL/openSUSE
sudo systemctl enable --now sysstat

Detect and read sensors:

sudo sensors-detect --auto
sensors

Live CPU and IO under load:

# CPU saturation (per-core)
mpstat -P ALL 5

# Disk IO latency and utilization (x: extended, z: omit idle, interval 5s)
iostat -xz 5

# NVMe-specific details
nvme list
nvme smart-log /dev/nvme0

Rules of thumb:

  • iostat r_await/w_await > 20–50ms -> your data loader is choking

  • %iowait > 10% with low GPU util -> storage bottleneck

  • swap usage rising -> increase RAM or reduce batch size

  • CPU steal time > 2–5% (in VMs) -> noisy neighbor or host contention

3) One-minute full-stack dashboards: Netdata or Glances

When you need a quick, attractive view without architecting a stack:

Install Netdata:

# Debian/Ubuntu
sudo apt update && sudo apt install -y netdata

# Fedora/RHEL
sudo dnf install -y netdata

# openSUSE
sudo zypper install -y netdata

Start and browse:

sudo systemctl enable --now netdata
# Visit http://YOUR_HOST:19999

Prefer terminal? Glances gives a rich TUI:

# Debian/Ubuntu
sudo apt update && sudo apt install -y glances

# Fedora/RHEL
sudo dnf install -y glances

# openSUSE
sudo zypper install -y glances

# Run
glances

Use cases:

  • See CPU/IO spikes during dataset unpacking

  • Catch memory creep across epochs (leaks or fragmentation)

  • Confirm net throughput during remote dataset fetches

4) Time-series and alerts with containers: Prometheus + Node Exporter + DCGM + Grafana

If you want history, correlations, and on-call alerts—use a TSDB and dashboards. Containers keep it simple.

Compose file (save as docker-compose.yml):

services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    network_mode: host
    pid: host
    volumes:
      - "/:/host:ro,rslave"
    command: ["--path.rootfs=/host"]
    restart: unless-stopped

  dcgm-exporter:
    image: nvidia/dcgm-exporter:latest
    gpus: all
    ports:
      - "9400:9400"
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alert.rules.yml:/etc/prometheus/alert.rules.yml:ro
    restart: unless-stopped

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    restart: unless-stopped

Prometheus config (prometheus.yml):

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']  # node-exporter (host network)

  - job_name: 'dcgm'
    static_configs:
      - targets: ['localhost:9400']  # NVIDIA GPU metrics

Bring it up:

docker compose up -d
# Prometheus: http://YOUR_HOST:9090
# Grafana:    http://YOUR_HOST:3000  (default admin/admin; set a new password)

Dashboards to import in Grafana:

  • “Node Exporter Full” (ID 1860) for host metrics

  • Search Grafana.com for “DCGM exporter” dashboards for NVIDIA GPUs

Tip: Use labels in Prometheus for hostnames, roles, and GPU types to compare nodes easily.

5) Alerts that actually help (copy/paste rules)

Add alert.rules.yml and reference it in Prometheus as shown above.

Example rules:

groups:

- name: ai-gpu
  rules:
  - alert: GPUSustainedLowUtilization
    expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[10m]) < 30
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "GPU underutilized (<30% for 15m)"
      description: "Likely input pipeline or CPU/IO bottleneck. Check iostat/mpstat and dataloader."

  - alert: GPUMemoryNearFull
    expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "GPU VRAM > 90% for 5m"
      description: "Risk of OOM/fragmentation. Reduce batch size, gradient checkpointing, or free caches."

  - alert: GPUThermalThrottlingRisk
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "GPU > 85°C"
      description: "Improve cooling or lower power limit to avoid throttling."

- name: ai-host
  rules:
  - alert: DiskLatencyHigh
    expr: avg_over_time(node_disk_read_time_seconds_total[5m]) / avg_over_time(node_disk_reads_completed_total[5m]) > 0.05
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Disk read latency > 50ms"
      description: "Your dataloader/checkpoints may be IO-bound. Check iostat and storage."

  - alert: SwapUsageGrowing
    expr: (node_memory_SwapUsed_bytes / node_memory_SwapTotal_bytes) > 0.2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Swap usage > 20%"
      description: "Memory pressure can crash or slow training. Increase RAM or reduce batch size."

Note: DCGM metric names may vary slightly by version. Use Prometheus’s “Explore” to autocomplete exact names after dcgm-exporter is running.

Real-world quick wins

  • Training stuck at 25–35% GPU

    • iostat shows 80–120ms await on data disk. Fix: move datasets to NVMe, add prefetch/cache, or increase data loader workers/pinning.

  • Random CUDA OOMs at epoch 2

    • GPU VRAM charts show sawtooth to 95% then spikes. Fix: smaller batch size, gradient checkpointing, clear torch CUDA cache between eval loops.

  • Inference latency spikes evening hours

    • node-exporter shows CPU steal time increases on shared VM host. Fix: move to dedicated hosts or adjust scheduler quotas.

  • Throughput drops after 20 minutes

    • DCGM shows temps crossing 85°C and clocks dipping. Fix: improve chassis airflow, clean filters, adjust fan curves/power limits.

Conclusion and next steps

Monitoring AI on Linux is about seeing what GPUs, CPUs, disks, and thermals are really doing—then acting fast. You now have:

  • Instant GPU visibility (nvtop, nvidia-smi)

  • Host baselines (sysstat, sensors, nvme-cli)

  • One-minute dashboards (Netdata/Glances)

  • A production-ready telemetry path (Prometheus + dcgm-exporter + Grafana)

  • Alerts that catch real issues before they burn epochs

Your next step: 1) Install the basics (Step 1–3) on one training host and watch a real job. 2) Deploy the containerized Prometheus/Grafana stack (Step 4) and import dashboards. 3) Turn on alerts (Step 5) and iterate thresholds to match your workloads.

Small, steady improvements in visibility compound into big gains in throughput and reliability. Ship dashboards to your team, make alerts someone’s job, and enjoy faster epochs with fewer surprises.

Related posts