The Artificial Intelligence Prometheus Guide: Turn Your AI Into Observable, Reliable Linux Services

AI systems don’t fail loudly. They degrade. Inference latency creeps up. GPU memory creeps toward 100%. Batch queues silently back up. Without visibility you’re flying blind—and downtime, cost overruns, and poor user experience follow.

Prometheus fixes that. It’s Linux-friendly, pull-based, and battle-tested. With the right exporters and a few lines of code, you can turn your AI workloads from black boxes into measurable, alertable, and optimizable services.

This guide shows you how to:

• Install Prometheus and core exporters on Debian/Ubuntu, Fedora/RHEL, and openSUSE

• Collect GPU, OS, and application metrics from AI model servers

• Configure scraping, create useful alerts, and visualize with Grafana

• Add lightweight Bash-based custom metrics

Why Prometheus for AI workloads?

• You need multi-layer observability: system (CPU, memory, disk, network), accelerators (GPU/VRAM/temperature/SM utilization), and app-level (request rates, latency, errors, queue depth).

• AI SLOs are different: p95/p99 latency, GPU memory headroom, batch throughput, and cost-per-inference all matter.

• Prometheus is a perfect fit for Linux-first MLOps: single binary, systemd services, simple text-based config, and a rich ecosystem of exporters.

• It scales: you can start on one box and grow to federated Prometheus setups or move to remote storage later.

---

1) Install the essentials: Prometheus + Node Exporter (+ Alertmanager)

These commands set up the basics. Use the package manager for your distro.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y prometheus prometheus-node-exporter prometheus-alertmanager
# Optional but recommended: create directories if not present
sudo mkdir -p /etc/prometheus /var/lib/prometheus
# Services typically start automatically; ensure they are enabled
sudo systemctl enable --now prometheus prometheus-node-exporter alertmanager

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y prometheus node_exporter alertmanager
sudo systemctl enable --now prometheus node_exporter alertmanager

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y prometheus node_exporter alertmanager
sudo systemctl enable --now prometheus node_exporter alertmanager

Notes:

• Config lives at /etc/prometheus/prometheus.yml on most distros.

• Prometheus scrapes endpoints that expose simple text metrics over HTTP.

• Node Exporter exposes host metrics on http://localhost:9100/metrics by default.

---

2) Collect the right signals: OS, GPU, and custom AI metrics

You’ll want at least three categories of metrics:

• OS-level: CPU, RAM, disk I/O, network (node_exporter handles this)

• GPU-level: utilization, VRAM, temperature, ECC, throttling

• App-level: request counts, latencies, batch sizes, cache hits/misses, queue depth

2.1 Enable Node Exporter’s textfile collector (for Bash-friendly custom metrics)

This lets you write simple files with “key value” metrics from any script.

sudo mkdir -p /var/lib/node_exporter/textfile_collector

Add the flag to Node Exporter’s systemd service:

Systemd override (works across distros):

sudo systemctl edit node_exporter

Then paste:

[Service]
ExecStart=
ExecStart=/usr/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Apply:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter

Now any file ending in .prom under /var/lib/node_exporter/textfile_collector will be scraped.

Example Bash metric (queue depth from Redis):

#!/usr/bin/env bash
depth=$(redis-cli -h 127.0.0.1 -p 6379 LLEN inference_queue 2>/dev/null || echo 0)
cat <<EOF | sudo tee /var/lib/node_exporter/textfile_collector/ai_custom.prom >/dev/null
ai_inference_queue_depth $depth
EOF

Add to cron:

* * * * * /usr/local/bin/emit_ai_metrics.sh

2.2 GPU metrics via NVIDIA DCGM Exporter

If you run NVIDIA GPUs, dcgm-exporter is the standard path. It runs as a container and exposes GPU metrics on :9400 by default.

Requirements: NVIDIA drivers + container runtime (Docker/Podman) with NVIDIA container toolkit on GPU hosts.

Docker example:

sudo docker run -d --restart=always --name=dcgm-exporter --gpus all \
  -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.5.0-ubuntu22.04

Podman example:

sudo podman run -d --restart=always --name=dcgm-exporter --hooks-dir=/usr/share/containers/oci/hooks.d/ \
  -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.5.0-ubuntu22.04

After starting, verify:

curl -s http://localhost:9400/metrics | head

Common metrics to watch (names may vary slightly with versions):

• DCGM_FI_DEV_GPU_UTIL (percentage)

• DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_TOTAL (bytes)

• DCGM_FI_DEV_MEMORY_TEMP, DCGM_FI_DEV_GPU_TEMP (Celsius)

• DCGM_FI_DEV_POWER_USAGE (watts)

Tip: Run curl to confirm exact metric names on your setup.

---

3) Instrument your AI service for p95/p99 latency and success rates

Prometheus is most powerful when your app exposes domain-aware metrics. Below is a minimal Python example for a FastAPI model server that tracks inference counts and latency histograms.

Python (FastAPI + prometheus_client):

# requirements.txt
fastapi
uvicorn
prometheus-client

from fastapi import FastAPI, Response
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time

app = FastAPI()

INFER_COUNTER = Counter("inference_requests_total", "Total inference requests", ["model", "status"])
INFER_LATENCY = Histogram(
    "inference_latency_seconds",
    "Latency of inference requests",
    ["model"],
    buckets=[0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]
)

@app.get("/infer")
def infer(model: str = "gpt-small"):
    start = time.time()
    try:
        # ... your model inference here ...
        time.sleep(0.03)  # simulate 30ms latency
        INFER_COUNTER.labels(model=model, status="success").inc()
        return {"ok": True}
    except Exception:
        INFER_COUNTER.labels(model=model, status="error").inc()
        return {"ok": False}
    finally:
        INFER_LATENCY.labels(model=model).observe(time.time() - start)

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

Run it:

uvicorn app:app --host 0.0.0.0 --port 8000

Now /metrics exposes inference_requests_total and inference_latency_seconds_bucket metrics for Prometheus to scrape.

---

4) Configure Prometheus to scrape everything

Edit /etc/prometheus/prometheus.yml (paths may vary slightly by distro). Add jobs for node exporter, your app, and dcgm-exporter.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'ai-app'
    static_configs:
      - targets: ['localhost:8000']

  - job_name: 'gpu'
    static_configs:
      - targets: ['localhost:9400']

rule_files:
  - /etc/prometheus/ai-alerts.yml

Reload Prometheus:

sudo systemctl reload prometheus

Validate:

• Prometheus UI: http://localhost:9090

• Status -> Targets should show all jobs UP.

---

5) Alerting and visualization: catch issues before users do

5.1 Alert rules that matter for AI

Create /etc/prometheus/ai-alerts.yml with a few high-signal alerts:

groups:

- name: ai
  rules:
  - alert: HighGPUUtilization
    expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[5m]) > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: GPU utilization high
      description: GPU util > 90% for 10m

  - alert: HighGPUMemoryUsage
    expr: (avg(DCGM_FI_DEV_FB_USED) / avg(DCGM_FI_DEV_FB_TOTAL)) > 0.90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: GPU memory near exhaustion
      description: VRAM usage > 90% for 5m

  - alert: InferenceErrorRate
    expr: rate(inference_requests_total{status="error"}[5m]) / rate(inference_requests_total[5m]) > 0.05
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: Elevated inference error rate
      description: Error rate > 5% over 10 minutes

  - alert: InferenceLatencyP95High
    expr: |
      histogram_quantile(0.95, sum by (le) (rate(inference_latency_seconds_bucket[5m]))) > 0.2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: p95 latency elevated
      description: p95 inference latency > 200ms for 10 minutes

Reload Prometheus to pick up rules:

sudo systemctl reload prometheus

Configure Alertmanager routing (emails, Slack, PagerDuty) in /etc/alertmanager/alertmanager.yml, then:

sudo systemctl restart alertmanager

5.2 Optional: Grafana dashboards (recommended)

Grafana gives you a single-pane view for GPU + app latency + queue depth.

Debian/Ubuntu (apt):

sudo apt install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings
wget -qO- https://packages.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg >/dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y grafana
sudo systemctl enable --now grafana-server

openSUSE/SLES (zypper):

sudo zypper install -y grafana
sudo systemctl enable --now grafana-server

In Grafana:

• Add Prometheus as a data source (http://localhost:9090).

• Import community dashboards for node_exporter and dcgm-exporter.

• Build a panel for histogram_quantile over inference_latency_seconds_bucket.

---

Real-world example: Finding the hidden latency killer

Symptom:

• Users report sporadic slow responses.

• Alert fires: InferenceLatencyP95High.

Investigation:

• Grafana shows GPU util ~40%, but VRAM ~88% and slowly climbing.

• Node Exporter shows disk I/O spikes and increased page cache pressure.

• App metrics reveal rising batch sizes and queue depth.

Root cause:

• An upstream feature doubled input token lengths during certain hours, causing memory fragmentation and larger intermediate tensors.

Fix:

• Introduced input size guardrails and dynamic batching thresholds.

• Alert for queue_depth > 1k added via textfile collector.

• Outcome: p95 back under 120ms, fewer OOM restarts, stable cost per inference.

---

Quick reference: Useful queries

• GPU memory percent:

100 * (avg(DCGM_FI_DEV_FB_USED) / avg(DCGM_FI_DEV_FB_TOTAL))

• p95 latency (global):

histogram_quantile(0.95, sum by (le) (rate(inference_latency_seconds_bucket[5m])))

• Error rate:

rate(inference_requests_total{status="error"}[5m]) / rate(inference_requests_total[5m])

• Queue depth (from textfile):

ai_inference_queue_depth

---

Conclusion and next steps

Prometheus turns AI observability from an afterthought into a first-class capability:

• OS, GPU, and app metrics in one place

• Alerting on what matters (latency, error rate, VRAM)

• A clear path to reliability and performance tuning

Your next steps: 1) Install Prometheus, Node Exporter, and Alertmanager with your package manager. 2) Start dcgm-exporter if you’re on NVIDIA GPUs. 3) Expose /metrics from your AI service and scrape it. 4) Add the alert rules above and wire up Alertmanager. 5) Optional: Install Grafana and import dashboards.

Have questions or want a follow-up guide on multi-node setups, Kubernetes, or remote storage for Prometheus? Tell me what your stack looks like and I’ll tailor the next post.