Posted on
• Artificial Intelligence

Artificial Intelligence Prometheus Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence + Prometheus: Best Practices for Reliable ML in Production

Your model was crushing benchmarks yesterday. Today, users complain about 30-second responses at 2 a.m., GPU bills are spiking, and nobody can tell if the problem is the model, the network, the queue, or a rogue rollout. Without the right metrics and alerts, AI systems turn into a black box.

Prometheus gives you the observability foundation you need—if you capture the right signals and structure them well. This guide shows you practical, Bash-friendly best practices to monitor AI/ML inference and pipelines with Prometheus, from installation to production-grade alerts.

Why Prometheus for AI is worth your time

  • AI services fail in non-obvious ways: drift, queue backlogs, tokenization slowdowns, GPU OOM, model version regressions.

  • Latency, throughput, and error budgets must be tracked by model and version, not just per host.

  • GPU utilization and memory headroom are cost-critical.

  • Prometheus is fast, vendor-neutral, easy to automate, and thrives on Linux. It excels at time-series, alerting, and SLOs—perfect for inference and data pipelines.

Install the core monitoring stack

We’ll install:

  • Prometheus (scrapes and stores metrics)

  • Node Exporter (host/system metrics)

  • Blackbox Exporter (synthetic checks for HTTP/gRPC endpoints)

  • Alertmanager (routes alerts)

  • Grafana (dashboards)

  • Optional: Pushgateway (for short-lived batch/training jobs)

Packages and service names vary by distro. Use the commands for your package manager. After installation, enable and start services with systemd.

Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y \
  prometheus prometheus-node-exporter prometheus-blackbox-exporter \
  prometheus-alertmanager grafana prometheus-pushgateway

# Enable and start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-node-exporter
sudo systemctl enable --now prometheus-blackbox-exporter
sudo systemctl enable --now alertmanager
sudo systemctl enable --now grafana-server
sudo systemctl enable --now prometheus-pushgateway

Fedora/RHEL/Rocky/Alma (dnf)

Tip: On RHEL/Rocky/Alma, enable EPEL first.

# RHEL/Rocky/Alma only
sudo dnf install -y epel-release
sudo dnf upgrade -y

# Fedora (often)
sudo dnf install -y prometheus prometheus-node-exporter prometheus-blackbox_exporter \
  prometheus-alertmanager grafana prometheus-pushgateway

# RHEL/Rocky/Alma (some repos name Prometheus v2 packages with a '2' suffix)
# If the above fails, try:
# sudo dnf install -y prometheus2 prometheus-node-exporter prometheus-blackbox_exporter \
#   prometheus-alertmanager grafana prometheus-pushgateway

# Enable and start
sudo systemctl enable --now prometheus
sudo systemctl enable --now node_exporter
sudo systemctl enable --now blackbox_exporter
sudo systemctl enable --now alertmanager
sudo systemctl enable --now grafana-server
sudo systemctl enable --now pushgateway

openSUSE/SLES (zypper)

sudo zypper refresh
sudo zypper install -y \
  prometheus2 prometheus-node_exporter prometheus-blackbox_exporter \
  prometheus-alertmanager grafana prometheus-pushgateway

# Enable and start
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-node_exporter
sudo systemctl enable --now prometheus-blackbox_exporter
sudo systemctl enable --now alertmanager
sudo systemctl enable --now grafana-server
sudo systemctl enable --now prometheus-pushgateway

Note: GPU metrics are typically exposed via NVIDIA’s DCGM exporter (commonly run via container). See the GPU section below.

Best Practices (with commands and examples)

1) Instrument model-aware metrics from your inference service

System metrics alone won’t tell you which model version is slow. Emit metrics directly from your AI service with clean names, low-cardinality labels, and latency histograms.

  • Use counters for requests and errors

  • Use histograms for latency (with sensible buckets)

  • Attach labels like model, model_version, and route; avoid user_id, prompt, or highly variable values

Python example (FastAPI + prometheus_client):

# pip install prometheus-client fastapi uvicorn
from fastapi import FastAPI, Request, Response
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
import time

REQUESTS = Counter(
    "ai_inference_requests_total",
    "Total inference requests",
    ["model", "model_version", "route", "status"]
)

LATENCY = Histogram(
    "ai_inference_latency_seconds",
    "Inference latency (seconds)",
    ["model", "model_version", "route"],
    buckets=[0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20]
)

TOKENS = Counter(
    "ai_inference_tokens_total",
    "Total tokens generated",
    ["model", "model_version", "route"]
)

QUEUE_DEPTH = Gauge(
    "ai_inference_queue_depth",
    "Number of requests waiting",
    ["model", "model_version"]
)

MODEL_INFO = Gauge(
    "ai_model_info",
    "Static info about the loaded model (value 1)",
    ["model", "model_version"]
)

app = FastAPI()
MODEL = "llm-13b"
VERSION = "2026-07-10"
MODEL_INFO.labels(MODEL, VERSION).set(1)

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

@app.post("/generate")
async def generate(req: Request):
    route = "generate"
    start = time.time()
    QUEUE_DEPTH.labels(MODEL, VERSION).inc()
    status = "200"
    try:
        # ... your inference code ...
        time.sleep(0.12)  # simulate work
        tokens = 120
        TOKENS.labels(MODEL, VERSION, route).inc(tokens)
        return {"tokens": tokens}
    except Exception:
        status = "500"
        return Response(status_code=500)
    finally:
        QUEUE_DEPTH.labels(MODEL, VERSION).dec()
        LATENCY.labels(MODEL, VERSION, route).observe(time.time() - start)
        REQUESTS.labels(MODEL, VERSION, route, status).inc()

Start your service and ensure /metrics is reachable:

curl -fsS http://localhost:8000/metrics | head

Key naming/label tips:

  • snake_case, unit in the name: ai_inference_latency_seconds

  • Keep label values bounded (e.g., 10–20 model versions max)

  • Avoid per-user or per-prompt labels (explosive cardinality)

2) Scrape the right mix: app, system, GPUs, and synthetic checks

Prometheus configuration file is typically at /etc/prometheus/prometheus.yml.

Minimal scrape config for your app, Node Exporter, and Blackbox Exporter:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'ai-inference'
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:8000']
        labels:
          app: 'inference'
          model: 'llm-13b'
          model_version: '2026-07-10'
          env: 'prod'

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://api.example.com/healthz
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox_exporter
      - source_labels: [__param_target]
        target_label: instance

Apply changes:

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

GPU metrics (NVIDIA) via DCGM exporter (container example):

# Requires NVIDIA drivers + container runtime with GPU support
docker run -d --gpus all --restart=always --name dcgm-exporter \
  -p 9400:9400 nvidia/dcgm-exporter:latest

# Add to Prometheus:
# - job_name: 'gpu'
#   static_configs:
#     - targets: ['localhost:9400']

Verify targets:

curl -fsS http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].labels.job'

3) Define SLOs with recording rules and alert on user-visible pain

SLOs align monitoring with experience. Start simple:

  • Availability: error_rate <= 1%

  • Latency: p99 <= 800ms

  • Throughput: tokens/sec above a floor during business hours

  • Backlog: queue depth near zero

Create a rules file /etc/prometheus/rules/ai.yml:

groups:

- name: ai-slo
  interval: 30s
  rules:
  - record: job:ai_requests_total:rate5m
    expr: sum by (model, model_version, route) (rate(ai_inference_requests_total[5m]))

  - record: job:ai_errors_total:rate5m
    expr: sum by (model, model_version, route) (rate(ai_inference_requests_total{status=~"5.."}[5m]))

  - record: job:ai_error_rate:5m
    expr: job:ai_errors_total:rate5m / job:ai_requests_total:rate5m

  - record: job:ai_latency_p99:5m
    expr: histogram_quantile(0.99,
            sum by (le, model, model_version, route) (
              rate(ai_inference_latency_seconds_bucket[5m])
            )
         )

- name: ai-alerts
  rules:
  - alert: InferenceHighLatencyP99
    expr: job:ai_latency_p99:5m{env="prod"} > 0.8
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High p99 latency for {{ $labels.model }} {{ $labels.model_version }}"
      description: "p99 latency is {{ $value }}s for route {{ $labels.route }} (env={{ $labels.env }})"

  - alert: InferenceHighErrorRate
    expr: job:ai_error_rate:5m{env="prod"} > 0.02 and job:ai_requests_total:rate5m{env="prod"} > 1
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Elevated error rate for {{ $labels.model }} {{ $labels.model_version }}"
      description: "Error rate {{ $value | printf \"%.2f\" }} over 5m"

  - alert: InferenceQueueBacklog
    expr: max_over_time(ai_inference_queue_depth{env="prod"}[5m]) > 100
    for: 10m
    labels:
      severity: warn
    annotations:
      summary: "Queue backlog for {{ $labels.model }} {{ $labels.model_version }}"
      description: "Queue depth above 100 for 10m, consider scaling inference workers."

Load rules and reload:

# In /etc/prometheus/prometheus.yml, add under rule_files:
# rule_files:
#   - /etc/prometheus/rules/*.yml

sudo promtool check rules /etc/prometheus/rules/ai.yml
sudo systemctl reload prometheus

Configure Alertmanager routing (/etc/alertmanager/alertmanager.yml):

route:
  receiver: 'devops'
  routes:
    - matchers:
      - severity="page"
      receiver: 'oncall'

receivers:
  - name: 'devops'
    email_configs:
      - to: "devops@example.com"
  - name: 'oncall'
    pagerduty_configs:
      - routing_key: "PAGERDUTY_INTEGRATION_KEY"

Reload Alertmanager:

sudo systemctl reload alertmanager

4) Keep label cardinality under control

Cardinality explosions will hurt performance and cost. Follow these rules:

  • Bound label sets (model, model_version, route, env are safe; do not add user_id, request_id, prompt_hash).

  • Use counters for tokens/requests with a small label set; join with logs for per-user analysis if needed.

  • Drop or relabel noisy labels at scrape time when necessary.

Example relabel to drop an unwanted label:

scrape_configs:

- job_name: 'ai-inference'
  static_configs:
    - targets: ['localhost:8000']
  metric_relabel_configs:
    - source_labels: [unbounded_label]
      regex: ".*"
      action: drop

5) Plan for scale: retention, remote write, and isolation

  • Retention: Store only what you query. For busy AI clusters, 15–30 days is common.
# /etc/default/prometheus or service args
# Example extra args:
--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB
  • Remote write to a long-term store if you need months/years:
remote_write:
  - url: https://your-remote-tsdb/api/v1/write

  • Isolate metrics ports. By default, exporters bind to 0.0.0.0. Restrict with firewalls or bind to localhost and have Prometheus scrape locally.

  • For Kubernetes, prefer ServiceMonitor/PodMonitor (Prometheus Operator) and use namespaces/labels to keep jobs tidy.

Real-world scenario: Safe model rollouts with metrics

You deploy model_version=2026-07-10 alongside 2026-06-01. With metrics labeled by model_version:

  • Grafana shows p99 latency and error rate per version.

  • A dashboard panel using histogram_quantile(0.99, ...) by (model_version) reveals the new version is 30% slower on /generate but equals on /embed.

  • An alert triggers InferenceHighLatencyP99 only for the new version. You roll back or adjust batch size just for that route.

  • Queue depth signals an under-provisioned worker group; autoscaler adds replicas.

All of this is visible and fast to diagnose because your metrics are model-aware and low-cardinality.

Optional: Metrics for short-lived training/batch jobs

For CI/CD data jobs or short training steps, use Pushgateway when the process can’t be scraped:

Install (already covered above), then push at job end:

echo "ai_batch_examples_processed_total{job=\"preprocess\",dataset=\"enwiki\"} 4200" | \
  curl --data-binary @- http://localhost:9091/metrics/job/preprocess/instance/runner-1

Prometheus config:


- job_name: 'pushgateway'
  static_configs:
    - targets: ['localhost:9091']

Note: Pushgateway is for batch jobs, not for regular request metrics.

Quick Grafana start

Grafana runs on port 3000:

sudo systemctl status grafana-server
xdg-open http://localhost:3000 || xdg-open http://$(hostname -I | awk '{print $1}'):3000

  • Default login: admin / admin (you’ll be prompted to change it).

  • Add Prometheus as a data source (http://localhost:9090).

  • Import dashboards: Node Exporter Full, Blackbox Exporter, and your AI custom panels (latency p99, error rate, queue depth, tokens/sec by model_version).

Troubleshooting checklist

  • Prometheus shows “DOWN” targets:

    • curl the target locally: curl -v http://127.0.0.1:8000/metrics
    • Firewalls and bind addresses (--web.listen-address) correct?

  • High cardinality warnings:

    • topk(20, count by (__name__) (scrape_series_added))
    • Inspect labels: label_replace or drop noisy ones.

  • Histogram buckets are wrong:

    • If p99 is always at max bucket, add larger buckets; if flat, add finer low-latency buckets.

Conclusion and next steps

Reliable AI is observable AI. By installing Prometheus and exporters, instrumenting model-aware metrics with tight label hygiene, and enforcing SLOs with recording rules and alerts, you turn black-box behavior into actionable signals.

Your next step: 1) Install the stack with your package manager. 2) Add the instrumentation snippet to your inference service and expose /metrics. 3) Paste the scrape config and rules, reload Prometheus, and watch your first alerts land in Alertmanager. 4) Build a Grafana dashboard that compares p99 latency and error rate across model versions.

Have questions or want a reference dashboard JSON and full configs? Drop a comment or reach out—happy to share a starter kit.

Related posts