Posted on
• Artificial Intelligence

Artificial Intelligence Grafana Dashboards

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Grafana Dashboards: See Your Models Breathe

When AI systems move from notebooks to production, they can feel like sealed black boxes. Are GPUs actually busy? What’s your p95 latency during traffic spikes? Are error rates creeping up with a new model version? Without proper observability, you’re guessing—and guessing is expensive.

Grafana dashboards turn AI workloads into visible, measurable systems you can improve. In this guide, you’ll set up a Linux-friendly stack with Grafana and Prometheus, collect the right AI-centric metrics (GPU, latency, throughput, errors, versions), and build a dashboard that answers the questions your team asks every day.

Why Grafana for AI observability?

  • Time-to-insight: Grafana surfaces real-time and historical signals that reveal hotspots, regressions, and bottlenecks.

  • Metric standardization: Prometheus + Grafana is a proven pattern that fits well with AI services (model servers, batch jobs, preprocessors).

  • Cross-team visibility: SRE, MLOps, and data scientists can share one pane of glass with clear SLOs and alerts.

  • Low friction on Linux: Install, configure, and automate everything with Bash and systemd.

Step 1: Install Grafana and Prometheus

You’ll use Grafana for visualization and Prometheus for metric collection and queries (PromQL). The commands below include apt (Debian/Ubuntu), dnf (Fedora/RHEL), and zypper (openSUSE).

Install Grafana

Apt (Debian/Ubuntu 20.04+):

sudo apt update
sudo apt install -y gnupg curl apt-transport-https software-properties-common
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server

DNF (Fedora/RHEL; on RHEL/CentOS, ensure EPEL if needed):

sudo tee /etc/yum.repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
sudo dnf install -y grafana
sudo systemctl enable --now grafana-server

Zypper (openSUSE):

sudo tee /etc/zypp/repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://packages.grafana.com/oss/rpm
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF
sudo zypper refresh
sudo zypper install -y grafana
sudo systemctl enable --now grafana-server

Grafana UI: http://localhost:3000 (default admin/admin; you’ll be prompted to change it).

Install Prometheus

Apt:

sudo apt update
sudo apt install -y prometheus
sudo systemctl enable --now prometheus

DNF (Fedora; on RHEL/CentOS, enable EPEL first: sudo dnf install -y epel-release):

sudo dnf install -y prometheus
sudo systemctl enable --now prometheus

Zypper (openSUSE):

sudo zypper refresh
sudo zypper install -y prometheus
sudo systemctl enable --now prometheus

Step 2: Collect the right AI metrics

You need three classes of signals:

  • Infrastructure: CPU, memory, disk, network (Node Exporter), GPU (NVIDIA DCGM).

  • Service: latency, throughput, error rate.

  • Model: version, feature-specific counters, drift proxies.

2a) System metrics with Node Exporter

Apt:

sudo apt update
sudo apt install -y prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter

DNF:

sudo dnf install -y prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter

Zypper (package name uses an underscore):

sudo zypper install -y prometheus-node_exporter
sudo systemctl enable --now prometheus-node_exporter

Node Exporter defaults to port 9100.

2b) GPU metrics with NVIDIA DCGM Exporter (container)

The DCGM exporter exposes GPU utilization, memory, power, and ECC stats on port 9400. Use Docker or Podman (adjust runtime as needed).

Docker:

sudo docker run -d --restart unless-stopped --name dcgm-exporter --gpus all -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.1.8-3.4.0-ubuntu22.04

Podman:

sudo podman run -d --restart=always --name dcgm-exporter --hooks-dir=/usr/share/containers/oci/hooks.d --device nvidia.com/gpu=all -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.1.8-3.4.0-ubuntu22.04

Tip: If you prefer packages, install NVIDIA drivers + DCGM via your distribution/NVIDIA repo and run dcgm-exporter as a service. Container is the fastest path.

2c) Service and model metrics from your AI app

Expose Prometheus metrics directly from your inference service. Example: FastAPI + prometheus_client.

# app.py
from fastapi import FastAPI, Request
import time, random
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()

INF_COUNT = Counter("inference_total", "Total inferences", ["model", "version"])
ERR_COUNT = Counter("inference_errors_total", "Total inference errors", ["model", "version"])
LATENCY = Histogram("inference_latency_seconds", "Inference latency", ["model", "version"])
GPU_WAIT = Histogram("inference_gpu_wait_seconds", "Time spent waiting for GPU", ["model", "version"])
MODEL_VER = Gauge("model_version_info", "Model version info", ["model", "version"])

MODEL = "resnet50"
VERSION = "2026-07-01"
MODEL_VER.labels(model=MODEL, version=VERSION).set(1)

@app.get("/predict")
def predict():
    start = time.time()
    try:
        # simulate inference
        gpu_wait = random.uniform(0, 0.01)
        time.sleep(gpu_wait)
        infer = random.uniform(0.01, 0.05)
        time.sleep(infer)

        LATENCY.labels(MODEL, VERSION).observe(time.time() - start)
        GPU_WAIT.labels(MODEL, VERSION).observe(gpu_wait)
        INF_COUNT.labels(MODEL, VERSION).inc()
        return {"ok": True}
    except Exception:
        ERR_COUNT.labels(MODEL, VERSION).inc()
        raise

@app.get("/metrics")
def metrics():
    data = generate_latest()
    return Response(content=data, media_type=CONTENT_TYPE_LATEST)

Run it:

pip install fastapi uvicorn prometheus-client
uvicorn app:app --host 0.0.0.0 --port 8000

This exposes metrics at http://localhost:8000/metrics.

Step 3: Point Prometheus at your targets

Edit /etc/prometheus/prometheus.yml to add scrape jobs for Node Exporter, DCGM, and your AI service.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'gpu'
    static_configs:
      - targets: ['localhost:9400']

  - job_name: 'ai-service'
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:8000']

Apply the change:

sudo systemctl reload prometheus

Sanity check:

  • Prometheus UI: http://localhost:9090/targets should show all “UP”.

  • Try a query: up

Step 4: Build your AI dashboard in Grafana

In Grafana: 1) Add data source: Prometheus → URL http://localhost:9090 → Save & Test. 2) Create a dashboard → Add panels with the PromQL below.

Useful queries:

  • GPU utilization (per GPU)
avg by (gpu) (DCGM_FI_DEV_GPU_UTIL)
  • GPU memory used (MiB)
avg by (gpu) (DCGM_FI_DEV_FB_USED)
  • Inference throughput (RPS) per model/version
sum by (model, version) (rate(inference_total[5m]))
  • p95 inference latency by model/version
histogram_quantile(0.95, sum by (le, model, version) (rate(inference_latency_seconds_bucket[5m])))
  • Error rate (%)
100 * sum(rate(inference_errors_total[5m])) / sum(rate(inference_total[5m]))
  • GPU wait share (how much time is spent queued vs compute)
histogram_quantile(0.95, sum by (le, model, version) (rate(inference_gpu_wait_seconds_bucket[5m])))
/
histogram_quantile(0.95, sum by (le, model, version) (rate(inference_latency_seconds_bucket[5m])))
  • Current model versions deployed (panel as “Stat” with value set to 1)
sum by (model, version) (model_version_info)

Panel tips:

  • Use model/version labels to filter and compare variants.

  • Add dashboard variables (model, version, instance) for quick pivots.

  • Group GPU panels by host if you run multiple servers.

Step 5: Add alerts that protect SLOs

Use Grafana Alerting or Prometheus alerting rules. Example Grafana alert expressions:

  • p95 latency breach (5m over 200ms) per model/version
histogram_quantile(0.95, sum by (le, model, version) (rate(inference_latency_seconds_bucket[5m]))) > 0.2
  • Error rate > 1% (5m)
( sum(rate(inference_errors_total[5m])) / sum(rate(inference_total[5m])) ) > 0.01
  • Underutilized GPU (avoid burning cash): utilization < 30% for 10m
avg_over_time( avg(DCGM_FI_DEV_GPU_UTIL) [10m] ) < 30
  • Memory pressure (watch for OOM on GPU)
avg by (gpu) (DCGM_FI_DEV_FB_USED) / avg by (gpu) (DCGM_FI_DEV_FB_TOTAL) > 0.9

Route notifications to Slack/Email/PagerDuty based on team ownership.

Real-world patterns that pay off quickly

  • Find CPU preprocessing bottlenecks: If p95 latency is high while GPUs stay <20% utilization, your pipeline may be CPU-bound. Offload preprocessing (e.g., DALI) or raise batch size.

  • Catch regression by version: Plot latency and error metrics with the version label. Version rollouts that spike metrics get rolled back fast.

  • Capacity planning: Track GPU utilization heatmaps and inference RPS. Use them to right-size instance counts and reservation windows.

  • Drift proxies: Add custom counters (e.g., bucketed input sizes, class distribution, embedding norms). Trend shifts over days often foreshadow accuracy drops.

Troubleshooting quick hits

  • Targets down in Prometheus UI:

    • Firewall/SELinux blocking ports? Open 9100, 9400, 8000 as needed.
    • Service not bound to 0.0.0.0? Adjust listen address or scrape localhost.

  • Empty latency histograms:

    • Ensure you’re using Histogram (not Summary) in your app and that buckets are configured for your latency range.

  • High-cardinality labels:

    • Avoid per-request IDs or user IDs as labels; use model/version/instance.

Your next step

  • Install Grafana and Prometheus on your Linux host with the commands above.

  • Expose metrics from your AI service and GPU stack.

  • Build a dashboard with the queries provided, then add alerts for your SLOs.

Once you see your models breathe—utilization, latency, errors—you can tune them confidently and stop wasting cycles and dollars. Start now: get Grafana running, wire up Prometheus, and ship your first AI dashboard today.

Related posts