Posted on
• Artificial Intelligence

Monitoring Artificial Intelligence GPU Performance

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Monitoring Artificial Intelligence GPU Performance on Linux (with Bash-first workflows)

When your AI job stalls at 0% GPU, or your inference pods tip over with “out of memory,” every second costs money and momentum. The fix often isn’t a bigger GPU; it’s better visibility. In this guide, you’ll learn practical, shell-first ways to monitor GPU performance on Linux so you can spot bottlenecks, prevent OOMs, and extract maximum throughput from the hardware you already have.

We’ll cover vendor tooling (NVIDIA, AMD, Intel), slick CLI dashboards, time-series logging for postmortems, and a quick path to dashboards/alerts. All commands are copy/paste-ready, and we include installation steps for apt, dnf, and zypper where applicable.

Why monitor GPU performance?

  • Cost and capacity: GPU time is expensive. Knowing when your GPUs are waiting on CPUs, I/O, or dataloaders lets you fix the real bottleneck.

  • Reliability: Detect thermal throttling, ECC errors, or power limits before they crash your training run.

  • Utilization vs memory: It’s common to see low compute utilization with high memory usage. Without metrics you can’t know if batch size, fragmentation, or input pipeline is the culprit.

  • Scaling confidence: Baselines and logs tell you whether to scale up (bigger GPU) or out (more GPUs/nodes).

1) Install the right CLI tools for your GPU

First, identify your vendor:

  • NVIDIA: lspci | grep -i nvidia

  • AMD: lspci | grep -i amd or | grep -i radeon

  • Intel GPU: lspci | grep -i intel.*graphics

Then install the vendor tools.

NVIDIA: driver + nvidia-smi

nvidia-smi ships with the proprietary NVIDIA driver. Install the recommended driver for your system. Versions vary; prefer the “recommended” choice.

  • Ubuntu/Debian (apt):

    # Ubuntu: automatically installs the recommended driver
sudo ubuntu-drivers autoinstall
# Or specify a version (replace 535 with what `ubuntu-drivers devices` recommends)
sudo apt update && sudo apt install -y nvidia-driver-535

  • Fedora/RHEL (dnf):

    # Enable RPM Fusion (required for proprietary NVIDIA drivers)
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                  https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf update -y
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda

  • openSUSE (zypper):

    # Add NVIDIA repo (choose your distro on NVIDIA’s site if this URL changes)
sudo zypper addrepo --refresh 'https://download.nvidia.com/opensuse/tumbleweed' NVIDIA
sudo zypper refresh
# Example package; version naming varies (G06/G05). Pick the latest for your GPU generation.
sudo zypper install -y x11-video-nvidiaG06 nvidia-computeG06

Verify:

nvidia-smi

AMD: ROCm tools (rocm-smi)

rocm-smi comes from AMD’s ROCm repositories. Package names and support vary by distro/version; check AMD’s ROCm docs if these change.

  • Ubuntu/Debian (apt):

    # Example for ROCm 6.x; adjust version as needed
echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.0/ ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/rocm.gpg
sudo apt update
sudo apt install -y rocm-smi

  • Fedora/RHEL (dnf):

    # Example repo setup; consult AMD docs for your exact release
sudo tee /etc/yum.repos.d/rocm.repo > /dev/null <<'EOF'
[rocm]
name=ROCm
baseurl=https://repo.radeon.com/rocm/yum/6.0/
enabled=1
gpgcheck=1
gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
EOF
sudo dnf clean all
sudo dnf install -y rocm-smi

  • openSUSE (zypper):

    # Example repo setup; verify for your version in AMD docs
sudo zypper addrepo --refresh https://repo.radeon.com/rocm/zyp/6.0 rocm
sudo rpm --import https://repo.radeon.com/rocm/rocm.gpg.key
sudo zypper refresh
sudo zypper install -y rocm-smi

Verify:

rocm-smi

Intel: intel_gpu_top

  • Ubuntu/Debian (apt):

    sudo apt update && sudo apt install -y intel-gpu-tools

  • Fedora/RHEL (dnf):

    sudo dnf install -y intel-gpu-tools

  • openSUSE (zypper):

    sudo zypper refresh && sudo zypper install -y intel-gpu-tools

Verify:

intel_gpu_top -h

2) Get quick, readable GPU snapshots

When you just want a live view during a run:

  • NVIDIA (basic overview):

    watch -n 1 --color nvidia-smi

  • NVIDIA (more detailed utilization over time):

    nvidia-smi dmon -s pucm
# p: power, u: sm (compute) util, c: memory ctrl util, m: mem usage

  • AMD:

    watch -n 1 --color 'rocm-smi --showuse --showtemp --showpower'

  • Intel:

    sudo intel_gpu_top   # interactive curses view
# Or sampled output every second to stdout:
sudo intel_gpu_top -o - -s 1000

Nice human-readable overlays (NVIDIA):

  • gpustat

    • Install Python pip first:
    • apt: sudo apt update && sudo apt install -y python3-pip
    • dnf: sudo dnf install -y python3-pip
    • zypper: sudo zypper refresh && sudo zypper install -y python3-pip
    • Then:
    pip3 install --user gpustat
~/.local/bin/gpustat --watch

  • nvitop (top-like dashboard for NVIDIA):

    pip3 install --user nvitop
~/.local/bin/nvitop

Tip: Add ~/.local/bin to your PATH:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

3) Record time-series during training for later analysis

Short runs fool the eye. Logging lets you correlate utilization with steps/epochs, batch size, or data I/O.

  • NVIDIA: CSV logging at 1-second intervals

    nvidia-smi \
--query-gpu=timestamp,index,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw \
--format=csv,noheader,nounits -l 1 | tee -a gpu_nvidia.csv

    Or with dmon (per-engine breakdown):

    nvidia-smi dmon -s pucm 1 | tee -a gpu_dmon.log

  • AMD: simple 1-second sampler

    while true; do
date --iso-8601=seconds
rocm-smi --showuse --showtemp --showpower
sleep 1
done | tee -a gpu_amd.log

  • Intel: sample to file every second

    sudo intel_gpu_top -o - -s 1000 | tee -a gpu_intel.log

Pro tip: Run the logger in a tmux/screen pane or background it with nohup ... & so it survives shell exits.

4) From shell to dashboards: Prometheus-friendly metrics in minutes

Option A — Quick-and-dirty with Node Exporter’s textfile collector using a container:

  • Install Podman (rootless-friendly):

    • apt:
    sudo apt update && sudo apt install -y podman
    • dnf:
    sudo dnf install -y podman
    • zypper:
    sudo zypper refresh && sudo zypper install -y podman

  • Create a directory for custom metrics and a simple NVIDIA metrics script:

    mkdir -p $HOME/node_exporter_textfiles
cat > $HOME/node_exporter_textfiles/gpu_metrics_nvidia.sh <<'EOF'
#!/usr/bin/env bash
out="$HOME/node_exporter_textfiles/gpu.prom"
ts=$(date +%s)
if command -v nvidia-smi >/dev/null 2>&1; then
while IFS=, read -r idx util smem used total temp power; do
  idx=$(echo "$idx" | xargs)
  util=$(echo "$util" | xargs)
  smem=$(echo "$smem" | xargs)
  used=$(echo "$used" | xargs)
  total=$(echo "$total" | xargs)
  temp=$(echo "$temp" | xargs)
  power=$(echo "$power" | xargs)
  {
    echo "gpu_utilization_percent{vendor=\"nvidia\",gpu=\"$idx\"} $util $ts"
    echo "gpu_mem_util_percent{vendor=\"nvidia\",gpu=\"$idx\"} $smem $ts"
    echo "gpu_mem_used_mib{vendor=\"nvidia\",gpu=\"$idx\"} $used $ts"
    echo "gpu_mem_total_mib{vendor=\"nvidia\",gpu=\"$idx\"} $total $ts"
    echo "gpu_temp_celsius{vendor=\"nvidia\",gpu=\"$idx\"} $temp $ts"
    echo "gpu_power_watts{vendor=\"nvidia\",gpu=\"$idx\"} $power $ts"
  } >> "$out"
done < <(nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits)
fi
EOF
chmod +x $HOME/node_exporter_textfiles/gpu_metrics_nvidia.sh

  • Run Node Exporter container with the textfile collector:

    podman run -d --name node-exporter -p 9100:9100 \
-v $HOME/node_exporter_textfiles:/textfiles:Z \
quay.io/prometheus/node-exporter:latest \
--collector.textfile.directory=/textfiles

  • Update metrics periodically (cron, systemd timer, or a loop):

    (crontab -l 2>/dev/null; echo "*/1 * * * * $HOME/node_exporter_textfiles/gpu_metrics_nvidia.sh") | crontab -

Scrape http://<host>:9100/metrics in Prometheus and build Grafana panels for the gpu_* series. You can extend the script for AMD by parsing rocm-smi --showuse --showtemp --showpower similarly.

Option B — NVIDIA DCGM Exporter (full-featured, production-grade):

  • If you have a container runtime and NVIDIA drivers, you can run DCGM Exporter to expose rich GPU metrics on port 9400. Example with Podman: podman run -d --name dcgm-exporter --privileged --net=host \ -v /var/run/nvidia-persistenced/socket:/var/run/nvidia-persistenced/socket \ nvcr.io/nvidia/k8s/dcgm-exporter:latest Consult NVIDIA’s DCGM Exporter docs for the exact flags and security context for your environment.

5) Real-world readouts and what they mean

  • Symptom: GPU memory is nearly full but utilization is low.

    • Likely causes: Too-small batches, CPU/dataloader bottleneck, slow storage, or lots of tiny kernels.
    • What to check:
    nvidia-smi dmon -s pucm
iostat -xz 1
vmstat 1
    • What to try: Increase batch size, increase DataLoader workers/prefetch, enable pinned memory, use mixed precision, fuse ops, or increase queue depth on input pipelines.

  • Symptom: Utilization starts high, then drops after a few minutes.

    • Likely causes: Thermal throttling or power limits.
    • What to check:
    watch -n 1 'nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks.sm,clocks.mem --format=csv'
# AMD:
watch -n 1 'rocm-smi --showtemp --showpower --showclocks'
    • What to try: Improve airflow, lower ambient temperature, check power budgets, reduce overclocking, or cap clocks to stable levels.

  • Symptom: Multi-GPU training is uneven; one GPU is always lower.

    • Likely causes: PCIe/NVLink topology, NUMA pinning, data sharding imbalance.
    • What to check:
    nvidia-smi topo -m
numactl --hardware
    • What to try: Rebalance data shards, set process affinity/numa binding, adjust DDP bucket sizes, or change GPU pairing based on topology.

  • Symptom: Out-of-memory near the end of an epoch.

    • Likely causes: Fragmentation, accumulating tensors, or validation spikes.
    • What to check:
    nvidia-smi --query-compute-apps=pid,used_memory --format=csv
    • What to try: Clear caches between phases, disable graph accumulation outside training steps, checkpoint activations, or reduce batch size slightly.

Bonus: Pin work to specific GPUs

Running multiple experiments? Isolate them:

CUDA_VISIBLE_DEVICES=1,2 python train.py

For ROCm:

HIP_VISIBLE_DEVICES=1,2 python train.py

Conclusion and next steps

Monitoring isn’t busywork—it’s your fastest path to cheaper, faster, and more reliable AI runs. You now have:

  • Vendor CLIs for instant visibility (nvidia-smi, rocm-smi, intel_gpu_top)

  • Friendly dashboards in your terminal (gpustat, nvitop)

  • One-liners to log time-series during training

  • A path to Prometheus/Grafana with either Node Exporter + textfile or DCGM Exporter

Call to action:

  • Add a logger to your next training run right now. Start with:

    nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,temperature.gpu,power.draw --format=csv,noheader -l 1 | tee -a gpu.csv

  • Stand up a quick Node Exporter with Podman and watch your first GPU panel in Grafana.

  • Share your favorite one-liners and dashboards with your team and bake them into your training scripts.

Better metrics mean better models—and less time staring at a frozen progress bar.

Related posts