Posted on
• Artificial Intelligence

Artificial Intelligence Container Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Container Monitoring: See Your GPUs, Stop the Fire Drills

AI workloads are hungry. They eat GPUs, memory, I/O, and money—and they do it inside containers that can make root-cause analysis painfully opaque. If you’ve ever wondered “Why did this training job stall?” or “Which model ate the entire GPU?” you already know the problem. Container monitoring tailored to AI bridges the gap between infra metrics and model behavior so you can ship faster, scale cheaper, and sleep better.

This guide shows you how to get meaningful, GPU-aware monitoring for AI containers using plain Bash and battle-tested tools. You’ll get actionable steps, ready-to-run commands, and minimal assumptions about your stack.

Why AI container monitoring is different (and critical)

  • GPUs are the bottleneck and cost center. You need per-container GPU visibility (memory, SM utilization, ECC errors).

  • Failures are expensive. OOM kills, GPU oversubscription, and thermal throttling derail long training runs.

  • Containers add layers. Mapping PIDs to containers, exporting GPU metrics, and tracking ephemeral containers require the right exporters and conventions.

  • Throughput > uptime. For inference, requests/sec and latency matter as much as host health.

Prerequisites: Install a container engine and a few CLI tools

Pick Docker or Podman (examples include both). You’ll also want curl and jq for quick Bash-based checks.

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y docker.io podman curl jq
sudo systemctl enable --now docker

  • Fedora/RHEL/CentOS (dnf):

    sudo dnf install -y docker podman curl jq
sudo systemctl enable --now docker
sudo systemctl start docker

  • openSUSE/SLES (zypper):

    sudo zypper refresh
sudo zypper install -y docker podman curl jq
sudo systemctl enable --now docker

Note:

  • Choose one engine for production to avoid conflicts; both are shown so you can follow along either way.

  • For GPU visibility inside containers, ensure the NVIDIA driver is installed on the host and your runtime supports GPUs (Docker: nvidia-container-toolkit; Podman: NVIDIA container support). Exporters below will still report system-level GPU metrics if the driver is present.

The 5-part recipe for AI container monitoring

1) Establish a quick baseline from the CLI

Start by seeing what’s actually running. These give you an “is it on fire?” snapshot.

  • CPU/mem/IO per container:

    # Docker
docker stats --no-stream

# Podman
podman stats --no-stream

  • GPU status (host-wide):

    nvidia-smi
nvidia-smi pmon -c 1

  • Which containers are busiest right now (top 5 by CPU):

    docker stats --no-stream --format '{{.Name}} {{.CPUPerc}}' | sort -k2 -hr | head -5
# or
podman stats --no-stream --format '{{.Name}} {{.CPUPerc}}' | sort -k2 -hr | head -5

Optional: Drop this minimal Bash logger to grab host GPU processes and map them to containers.

#!/usr/bin/env bash
# ai-gpu-proc-map.sh — map GPU PIDs -> container names (Docker or Podman)
set -euo pipefail
ENGINE="${ENGINE:-}"
if command -v docker >/dev/null 2>&1; then ENGINE="${ENGINE:-docker}"; fi
if command -v podman >/dev/null 2>&1; then ENGINE="${ENGINE:-podman}"; fi
[ -n "$ENGINE" ] || { echo "No docker/podman found"; exit 1; }

# Build a map of containerID -> name
declare -A CNAME
$ENGINE ps --no-trunc --format '{{.ID}} {{.Names}}' | while read -r id name; do
  echo "$id $name"
done | while read -r id name; do CNAME[$id]="$name"; done

# Query GPU compute procs
nvidia-smi --query-compute-apps=pid,used_memory,gpu_uuid --format=csv,noheader,nounits 2>/dev/null | while IFS=',' read -r pid mem gpu; do
  pid="$(echo "$pid" | xargs)"; mem="$(echo "$mem" | xargs)"; gpu="$(echo "$gpu" | xargs)"
  [ -d "/proc/$pid" ] || continue
  cid="$(awk -F/ '/docker|containerd|libpod/ {print $NF}' /proc/"$pid"/cgroup | tail -n1 | cut -c1-12)"
  cname="${CNAME[$cid]:-unknown}"
  cmd="$(tr -d '\0' </proc/"$pid"/cmdline | tr ' ' '\n' | head -n1)"
  printf "%-12s %-30s PID=%-7s GPU=%-38s MEM(MiB)=%s CMD=%s\n" "$cid" "$cname" "$pid" "$gpu" "$mem" "$cmd"
done

Run it:

bash ai-gpu-proc-map.sh

This gives you quick attribution: which container owns which GPU process and how much memory it’s using.

2) Export system and container metrics (cAdvisor + Node Exporter)

We’ll run exporters as containers so you don’t fight distro packages.

  • cAdvisor (container CPU/mem/fs/oom/restarts):

    # Docker
docker run -d --name=cadvisor --restart unless-stopped \
-p 8080:8080 \
-v /:/rootfs:ro \
-v /var/run:/var/run:ro \
-v /sys:/sys:ro \
-v /var/lib/docker/:/var/lib/docker:ro \
gcr.io/cadvisor/cadvisor:latest

# Podman (adjust docker path if using overlayfs)
podman run -d --name=cadvisor --restart=unless-stopped \
-p 8080:8080 \
-v /:/rootfs:ro \
-v /var/run:/var/run:ro \
-v /sys:/sys:ro \
-v /var/lib/containers/:/var/lib/docker:ro \
gcr.io/cadvisor/cadvisor:latest

  • Node Exporter (host CPU/mem/disk/thermal):

    # Docker
docker run -d --name=node-exporter --restart unless-stopped \
-p 9100:9100 --pid=host \
-v "/:/host:ro,rslave" \
quay.io/prometheus/node-exporter:latest \
--path.rootfs=/host

# Podman
podman run -d --name=node-exporter --restart=unless-stopped \
-p 9100:9100 --pid=host \
-v "/:/host:ro,rslave" \
quay.io/prometheus/node-exporter:latest \
--path.rootfs=/host

Now you have Prometheus-friendly endpoints:

  • cAdvisor: http://HOST:8080/metrics

  • Node Exporter: http://HOST:9100/metrics

3) Make it GPU-aware (NVIDIA DCGM Exporter)

This exporter surfaces GPU memory, SM utilization, ECC, clocks, power, and per-GPU health.

# Docker
docker run -d --name=dcgm-exporter --restart unless-stopped \
  --gpus all -p 9400:9400 \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest

# Podman (requires NVIDIA support configured)
podman run -d --name=dcgm-exporter --restart=unless-stopped \
  --hooks-dir=/usr/share/containers/oci/hooks.d \
  --security-opt=label=disable \
  --device nvidia.com/gpu=all -p 9400:9400 \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest

Endpoint: http://HOST:9400/metrics

Key metrics you’ll use:

  • dcgm_fb_used, dcgm_fb_total (GPU memory)

  • dcgm_sm_active (streaming multiprocessors utilization)

  • dcgm_pcie_tx_bytes, dcgm_pcie_rx_bytes (PCIe throughput)

  • dcgm_retired_pages (memory health)

4) Scrape and visualize (Prometheus + Grafana)

Create a minimal Prometheus config to scrape all three exporters.

prometheus.yml:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['localhost:8080']
  - job_name: 'dcgm'
    static_configs:
      - targets: ['localhost:9400']

Run Prometheus and Grafana:

# Prometheus
docker run -d --name=prometheus --restart unless-stopped \
  -p 9090:9090 \
  -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro \
  prom/prometheus:latest

# Grafana
docker run -d --name=grafana --restart unless-stopped \
  -p 3000:3000 \
  grafana/grafana-oss:latest

Open:

  • Prometheus UI: http://HOST:9090

  • Grafana: http://HOST:3000 (default admin/admin)

In Grafana, add Prometheus as a datasource (URL: http://prometheus:9090 if on the same Docker network; otherwise host URL). Import common dashboards:

  • Node Exporter Full (host overview)

  • cAdvisor/Container dashboards (per-container CPU/mem/restarts)

  • NVIDIA DCGM dashboard (GPU utilization/memory/health)

Tip: If running everything on one host with Docker, attach the containers to a shared network so Grafana can reach Prometheus by name:

docker network create mon
docker network connect mon prometheus
docker network connect mon grafana

5) Alert on the stuff that hurts (simple PromQL rules)

Create alerts.yml and mount it into Prometheus to detect high GPU memory, OOMs, and flapping containers.

alerts.yml:

groups:

- name: ai-alerts
  rules:
  - alert: GPUMemoryPressure
    expr: (dcgm_fb_used / dcgm_fb_total) > 0.90
    for: 2m
    labels: { severity: warning }
    annotations:
      summary: "GPU memory > 90%"
      description: "High GPU memory usage on {{ $labels.uuid }}"

  - alert: GPUComputeSaturation
    expr: avg_over_time(dcgm_sm_active[5m]) > 0.95
    for: 5m
    labels: { severity: warning }
    annotations:
      summary: "GPU SM utilization > 95%"
      description: "Sustained high compute utilization on {{ $labels.uuid }}"

  - alert: ContainerOOMKills
    expr: increase(container_oom_events_total[5m]) > 0
    for: 0m
    labels: { severity: critical }
    annotations:
      summary: "Container OOM kill"
      description: "One or more containers hit the OOM killer"

  - alert: ContainerCrashLoop
    expr: increase(container_last_seen[10m]) == 0
    for: 10m
    labels: { severity: warning }
    annotations:
      summary: "Container not seen"
      description: "A previously running container disappeared in the last 10m"

Update prometheus.yml to load alert rules:

rule_files:
  - /etc/prometheus/alerts.yml

Run with both configs:

docker rm -f prometheus
docker run -d --name=prometheus --restart unless-stopped \
  -p 9090:9090 \
  -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro \
  -v $PWD/alerts.yml:/etc/prometheus/alerts.yml:ro \
  prom/prometheus:latest

Hook Alertmanager for notifications (Slack, email, PagerDuty) when you’re ready.

Real-world pattern: Catching a “slow LLM” before it burns a day

Symptom: An inference container running a 13B parameter model shows good CPU but erratic latency.

What the above stack shows:

  • dcgm_sm_active low but dcgm_fb_used ~98%: You’re memory-bound, not compute-bound.

  • cAdvisor shows container restarts climbing during batch size changes: OOM kills from oversizing batches.

  • Node Exporter reveals memory pressure on the host (swap/kswap spikes): container limits are missing or too high.

Quick fix:

  • Lower batch size or enable tensor parallelism across GPUs.

  • Set container memory limits and GPU resource requests.

  • Alert on GPU memory > 90% to catch this proactively next time.

Optional: Lightweight Bash collector for ad-hoc profiling

For quick CSV logs during a training run:

#!/usr/bin/env bash
# ai-sample-logger.sh — log container and GPU snapshot every 10s
set -euo pipefail
ENGINE="${ENGINE:-docker}" # or podman
OUT="${OUT:-ai-sample.csv}"
echo "ts,container,cpu_perc,mem_perc,gpu_used_mib,gpu_sm_active" > "$OUT"

while true; do
  ts="$(date -Iseconds)"
  # CPU/MEM by container
  mapfile -t rows < <($ENGINE stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemPerc}}')
  # GPU host-level snapshot
  gpu_used="$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | awk '{s+=$1} END{print s+0}')"
  sm_active="$(curl -s localhost:9400/metrics 2>/dev/null | awk -F' ' '/^dcgm_sm_active/ {sum+=$2} END{print (sum==""?0:sum)}')"
  for r in "${rows[@]}"; do
    echo "$ts,$r,$gpu_used,$sm_active" >> "$OUT"
  done
  sleep 10
done

Run it while you experiment and graph the CSV later.

Conclusion and next steps

You now have:

  • Fast, Bash-friendly visibility (docker/podman stats + nvidia-smi)

  • Container metrics (cAdvisor), host metrics (Node Exporter), and GPU metrics (DCGM)

  • Central scraping and dashboards (Prometheus + Grafana)

  • Alerts for the issues that actually hurt AI workloads

Next steps:

  • Put Prometheus and Grafana on a shared network and add Alertmanager for notifications.

  • Add app-level metrics (e.g., Triton’s request throughput, model latency) to correlate infra with model performance.

  • Set resource limits/requests for containers to prevent noisy neighbors and OOMs.

  • Roll this to every node in your cluster for a unified view.

If you want a follow-up post, say the word: we can cover Kubernetes-native deployment (DaemonSets for exporters, GPU node labeling), Triton/LLM exporter integration, and cost-aware autoscaling—still all from the comfort of your Linux shell.

Related posts