Posted on
• Artificial Intelligence

Artificial Intelligence Observability for Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Observability for Linux: From GPU To Trace In 30 Minutes

AI workloads don’t fail quietly. They slow down because a GPU is starved by I/O, a kernel throttles CPUs, or a tiny refactor adds a surprise 200 ms to every inference. Without observability on Linux—the platform most of us run for training and inference—you’re flying blind.

This article shows a pragmatic, bash-first path to AI observability on Linux. You’ll set up host and GPU metrics, add app-level telemetry, and learn a couple of eBPF one-liners you’ll actually use when things go sideways. All commands are given for apt, dnf, and zypper where relevant.

Why observability for AI on Linux?

  • AI performance depends on the system: CPU scheduling, NUMA, PCIe bandwidth, disk throughput, and GPU memory. You need metrics beyond your framework logs.

  • Reproducibility: training runs and inference services behave differently across kernels, drivers, and containers. System and app telemetry closes the gap.

  • Cost-control: catching underutilized GPUs or memory thrash early can save real money.

What we’ll build (the 80/20 stack)

  • Host metrics with Prometheus + Node Exporter

  • GPU metrics via a tiny Python “gpu-exporter” that wraps nvidia-smi

  • App-level metrics in your AI code (request latency, batch size, model-specific counters)

  • Deep-dive diagnostics with bpftrace (eBPF) one-liners

  • Optional: basic Prometheus queries you can paste into your dashboard

Everything runs on bare-metal Linux or VMs. No Kubernetes required.

Step 1 — Install Prometheus and Node Exporter

Prometheus scrapes metrics; Node Exporter exposes host metrics (CPU, memory, disk, network).

Install Prometheus:

  • apt (Debian/Ubuntu):
sudo apt update
sudo apt install -y prometheus
  • dnf (Fedora/RHEL/CentOS Stream; enable EPEL on RHEL derivatives if needed):
sudo dnf install -y prometheus
  • zypper (openSUSE/SLE):
sudo zypper -n install prometheus

Install Node Exporter:

  • apt:
sudo apt install -y prometheus-node-exporter
  • dnf:
sudo dnf install -y node_exporter || sudo dnf install -y prometheus-node-exporter
  • zypper:
sudo zypper -n install prometheus-node_exporter || sudo zypper -n install prometheus-node-exporter

Enable services:

# Prometheus
sudo systemctl enable --now prometheus

# Detect Node Exporter service name and enable it
NODE_SVC=$(systemctl list-unit-files | awk '/node.*exporter/ {print $1; exit}')
[ -n "$NODE_SVC" ] && sudo systemctl enable --now "$NODE_SVC" || echo "Node Exporter service not found"

Quick checks:

# Prometheus
curl -fsS localhost:9090/-/ready && echo "prometheus OK"

# Node Exporter (usually 9100)
curl -fsS localhost:9100/metrics | head

Step 2 — Add GPU metrics with a lightweight exporter

NVIDIA GPUs: we’ll publish metrics by parsing nvidia-smi and serving them on http://localhost:9400 for Prometheus to scrape.

Install Python tooling:

  • apt:
sudo apt install -y python3 python3-pip
  • dnf:
sudo dnf install -y python3 python3-pip
  • zypper:
sudo zypper -n install python3 python3-pip

Install dependencies:

sudo pip3 install --upgrade prometheus_client

Create the exporter:

sudo tee /usr/local/bin/gpu_exporter.py >/dev/null <<'PY'
#!/usr/bin/env python3
import os
import time
import subprocess
from http.server import HTTPServer, BaseHTTPRequestHandler
from prometheus_client import CollectorRegistry, Gauge, generate_latest

HOST = os.environ.get("GPU_EXPORTER_HOST", "0.0.0.0")
PORT = int(os.environ.get("GPU_EXPORTER_PORT", "9400"))

def nvidia_metrics():
    cmd = [
        "nvidia-smi",
        "--query-gpu=uuid,index,utilization.gpu,utilization.memory,memory.total,memory.used,temperature.gpu,power.draw",
        "--format=csv,noheader,nounits",
    ]
    try:
        out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, text=True, timeout=2.0)
    except Exception:
        return []
    rows = []
    for line in out.strip().splitlines():
        parts = [p.strip() for p in line.split(",")]
        if len(parts) < 8:
            continue
        rows.append({
            "uuid": parts[0],
            "index": parts[1],
            "util_gpu": float(parts[2]),
            "util_mem": float(parts[3]),
            "mem_total": float(parts[4]),
            "mem_used": float(parts[5]),
            "temp": float(parts[6]),
            "power": float(parts[7]),
        })
    return rows

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/metrics":
            self.send_response(404); self.end_headers(); return
        reg = CollectorRegistry()
        g_util = Gauge("gpu_utilization_percent", "GPU utilization percent", ["gpu", "uuid"], registry=reg)
        g_memu = Gauge("gpu_memory_used_bytes", "GPU memory used (bytes)", ["gpu", "uuid"], registry=reg)
        g_memt = Gauge("gpu_memory_total_bytes", "GPU memory total (bytes)", ["gpu", "uuid"], registry=reg)
        g_temp = Gauge("gpu_temperature_celsius", "GPU temperature C", ["gpu", "uuid"], registry=reg)
        g_pwr  = Gauge("gpu_power_watts", "GPU power draw W", ["gpu", "uuid"], registry=reg)
        g_memu_pct = Gauge("gpu_memory_utilization_percent", "GPU memory utilization percent", ["gpu", "uuid"], registry=reg)

        for row in nvidia_metrics():
            labels = {"gpu": row["index"], "uuid": row["uuid"]}
            g_util.labels(**labels).set(row["util_gpu"])
            g_memu.labels(**labels).set(row["mem_used"] * 1024 * 1024)
            g_memt.labels(**labels).set(row["mem_total"] * 1024 * 1024)
            g_temp.labels(**labels).set(row["temp"])
            g_pwr.labels(**labels).set(row["power"])
            if row["mem_total"] > 0:
                g_memu_pct.labels(**labels).set(100.0 * row["mem_used"] / row["mem_total"])

        content = generate_latest(reg)
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.send_header("Content-Length", str(len(content)))
        self.end_headers()
        self.wfile.write(content)

def main():
    server = HTTPServer((HOST, PORT), Handler)
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        pass

if __name__ == "__main__":
    main()
PY
sudo chmod +x /usr/local/bin/gpu_exporter.py

Add a systemd service:

sudo tee /etc/systemd/system/gpu-exporter.service >/dev/null <<'UNIT'
[Unit]
Description=GPU Exporter (nvidia-smi -> Prometheus)
After=network.target

[Service]
ExecStart=/usr/bin/env python3 /usr/local/bin/gpu_exporter.py
User=nobody
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now gpu-exporter
curl -fsS localhost:9400/metrics | head

Update Prometheus scrape config:

Backup and replace a minimal config (adjust if you already have one):

sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak.$(date +%s) 2>/dev/null || true
sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'YAML'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "gpu"
    static_configs:
      - targets: ["localhost:9400"]
YAML

sudo systemctl restart prometheus

Open Prometheus at http://localhost:9090 and try these queries:

  • GPU utilization: gpu_utilization_percent

  • GPU memory used (GiB): gpu_memory_used_bytes / 1024^3

  • CPU usage: 1 - avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m]))

Tip: If you run Prometheus on a different host, change targets to point at your AI host’s IP.

Step 3 — Add app-level metrics (Python example)

Expose latency, throughput, and batch size from your inference/training code.

Install in your project venv:

pip install prometheus-client

Drop-in snippet for a Flask-ish inference service:

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time

# Start a Prometheus metrics endpoint on :9300
start_http_server(9300)

# Define metrics
inference_requests = Counter("inference_requests_total", "Total inference requests")
inference_errors = Counter("inference_errors_total", "Total inference errors")
inference_latency = Histogram("inference_latency_seconds", "Inference latency (s)", buckets=(0.01,0.05,0.1,0.2,0.5,1,2,5))
batch_size_gauge = Gauge("inference_batch_size", "Current batch size")

def run_inference(model, batch):
    t0 = time.time()
    inference_requests.inc()
    batch_size_gauge.set(len(batch))
    try:
        out = model(batch)  # your model call
        return out
    except Exception:
        inference_errors.inc()
        raise
    finally:
        inference_latency.observe(time.time() - t0)

Prometheus scrape job to add:

  - job_name: "app"
    static_configs:
      - targets: ["localhost:9300"]

If you want distributed tracing later, instrument with OpenTelemetry and ship to Jaeger via a container; you don’t need to change your system packages for that.

Step 4 — Deep dives with eBPF (bpftrace)

When p99 latency spikes, the question is “why right now?” eBPF lets you answer without kernel rebuilds.

Install bpftrace:

  • apt:
sudo apt install -y bpftrace bpfcc-tools
  • dnf:
sudo dnf install -y bpftrace bcc-tools
  • zypper:
sudo zypper -n install bpftrace bcc-tools

Useful one-liners:

  • Which syscalls are hot for your PID (10 seconds):
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == PID/ { @[probe] = count(); } interval:s:10 { exit(); }'
  • Run queue latency (are we CPU-saturated?):
sudo bpftrace -e 'hist:runqlat { @ = hist(arg0); } interval:s:10 { exit(); }'
  • Off-CPU time by blocked stacks (why are threads sleeping? 10s sample):
sudo bpftrace -e 'profile:hz:99 /pid == PID/ { @off[ustack] = count(); } interval:s:10 { exit(); }'

Replace PID with your process ID. Use perf top as a quick alternative if eBPF is restricted.

Step 5 — A real-world triage example

Symptom: Your PyTorch inference p99 jumps from 40 ms to 180 ms under load.

What to check, fast:

1) GPU headroom:

  • Query: max(gpu_utilization_percent) — if it’s under ~70% but latency is high, GPU is waiting on something.

2) CPU saturation and run queue:

  • Query: 1 - avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[1m]))

  • bpftrace: runqlat histogram shows long scheduling delays => CPU bound or noisy neighbors.

3) Memory pressure / swapping:

  • Query: rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m]) — any non-zero is red flag.

4) Disk I/O bottleneck:

  • Query: irate(node_disk_read_bytes_total[5m]) and node_disk_io_time_seconds_total

  • If high, look for dataset or model weights being fetched synchronously.

5) Fix patterns:

  • Increase CPU limits or pin service to isolated CPUs (cgroups).

  • Preload weights into GPU memory, use page-locked host memory.

  • Batch requests (watch your inference_batch_size metric).

  • If NUMA, bind process memory and CPUs to the closest GPU PCIe NUMA node.

You’ll see the impact immediately in your app and GPU metrics, not “after the incident.”

Common Prometheus queries you’ll reuse

  • Node CPU usage:
100 * (1 - avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])))
  • Memory pressure:
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100
  • GPU memory util percent:
gpu_memory_utilization_percent
  • Inference error rate per minute:
rate(inference_errors_total[1m])
  • p95 latency (if you use Histogram):
histogram_quantile(0.95, sum by (le) (rate(inference_latency_seconds_bucket[5m])))

Wrap-up and next steps

You now have:

  • Host metrics (Node Exporter)

  • GPU metrics (nvidia-smi exporter)

  • App metrics (latency, throughput, batch size)

  • eBPF tools for when the graph isn’t enough

Next steps:

  • Add Grafana for dashboards (install via your package manager or container) and wire Prometheus as a data source.

  • Containerize the gpu-exporter and your app for repeatability.

  • Add alerts: start with “GPU utilization < 30% for 10m while requests > X” and “p95 inference latency > target SLO.”

  • If you need traces, run a Jaeger all-in-one container and instrument with OpenTelemetry.

Have questions or want a ready-to-paste dashboard JSON? Tell me your distro and stack and I’ll tailor a Linux-first AI observability starter kit for you.

Related posts