Posted on
• Artificial Intelligence

Artificial Intelligence Prometheus Workflows

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Prometheus Workflows: From Invisible GPU Workloads to Actionable Signals

If you’ve ever kicked off a training job on Friday and returned Monday to find your GPUs idle or your model underperforming, you’ve met the silent failure of AI workflows. The fix isn’t “more logging.” It’s first-class, queryable metrics that turn your AI pipelines into observable systems. Prometheus gives you that signal, and with a few Linux-friendly steps and Bash chops, you can build AI Prometheus workflows that are repeatable, debuggable, and cost-aware.

This article shows you why this matters, then walks you through a practical, Bash-centric setup: install the right pieces, export metrics from AI code and GPUs, wire scrapes and alerts, and put it all into real-world use.

Why Prometheus for AI?

  • Reproducibility and regression hunting: Track throughput, loss curves, and data loader latency across runs.

  • Cost and capacity: Measure GPU-hours, utilization, and memory pressure to rightsize clusters and budgets.

  • SLOs for inference: Latency, error rates, and saturation turn into alerts before incidents hit users.

  • Cross-stack clarity: Unify app-level (Python) and infra-level (GPU, CPU, disk) metrics into a single queryable time series store.

1) Install the Building Blocks

We’ll use:

  • Prometheus (scrapes and stores metrics)

  • Node Exporter (system metrics and a textfile collector for custom Bash metrics)

  • Pushgateway (for short-lived/batch jobs that can’t be scraped)

  • Python prometheus-client (to instrument training/inference code)

Note: Package names can vary slightly by distro. If a package isn’t found, update your repos and consult your distro’s package list. On RHEL/CentOS, you may need EPEL: sudo dnf install -y epel-release.

Install Prometheus, Node Exporter, Pushgateway, and pip:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y prometheus prometheus-node-exporter prometheus-pushgateway python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y prometheus node_exporter pushgateway python3-pip
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y prometheus prometheus-node_exporter prometheus-pushgateway python3-pip

Enable and start services:

# Prometheus
sudo systemctl enable --now prometheus

# Node Exporter (service name differs by distro; try both)
sudo systemctl enable --now prometheus-node-exporter || sudo systemctl enable --now node_exporter

# Pushgateway
sudo systemctl enable --now pushgateway || sudo systemctl enable --now prometheus-pushgateway

Prometheus default config is usually at /etc/prometheus/prometheus.yml. Services typically listen on:

  • Prometheus: 9090

  • Node Exporter: 9100

  • Pushgateway: 9091

2) Export Metrics From Your AI Code (Python)

Expose custom metrics (throughput, loss, latency) straight from your training or inference process.

Install the client:

pip3 install prometheus-client

Add a tiny HTTP metrics endpoint to your code:

# ai_metrics.py
from prometheus_client import start_http_server, Summary, Gauge, Counter
import time, random

# Metric definitions
train_step_seconds = Summary("ai_train_step_seconds", "Time per training step")
train_loss = Gauge("ai_train_loss", "Latest training loss", ["model"])
gpu_mem_bytes = Gauge("ai_gpu_mem_bytes", "Approx GPU memory allocated", ["device"])
samples_processed_total = Counter("ai_samples_processed_total", "Total samples processed", ["model"])

def get_gpu_mem(device_idx=0):
    # Replace this stub with torch.cuda.memory_allocated(device) or your runtime’s API
    # import torch; return torch.cuda.memory_allocated(device_idx)
    return random.randint(10_000_000, 500_000_000)

def training_loop():
    model_name = "resnet50"
    for step in range(1000000):
        start = time.time()
        time.sleep(random.uniform(0.01, 0.05))  # simulate compute
        loss = random.uniform(0.1, 1.0)

        # Update metrics
        duration = time.time() - start
        train_step_seconds.observe(duration)
        train_loss.labels(model=model_name).set(loss)
        gpu_mem_bytes.labels(device="gpu0").set(get_gpu_mem(0))
        samples_processed_total.labels(model=model_name).inc(32)  # e.g., batch size

if __name__ == "__main__":
    # Expose at http://localhost:8000/metrics
    start_http_server(8000)
    training_loop()

Run your training script; Prometheus will scrape http://host:8000/metrics.

Tips:

  • Use labels such as model, dataset, run_id.

  • For inference, record latency buckets (Histogram) and error counts.

3) Get System and GPU Metrics With Node Exporter (Bash + textfile collector)

Node Exporter gives you CPU, memory, disk, and more. To surface custom one-off metrics (like GPU via nvidia-smi without containers), use its textfile collector.

1) Pick the correct service name (varies by distro) and enable the textfile collector:

# Figure out the service name
SVC=""
for name in prometheus-node-exporter node_exporter; do
  systemctl status "$name" >/dev/null 2>&1 && SVC="$name" && break
done
echo "Using service: $SVC"

# Create a textfile collector dir
sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown -R nobody:nogroup /var/lib/node_exporter/textfile_collector 2>/dev/null || true

# Add a systemd drop-in to pass the flag
sudo mkdir -p /etc/systemd/system/$SVC.service.d
cat <<'EOF' | sudo tee /etc/systemd/system/$SVC.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
EOF

sudo systemctl daemon-reload
sudo systemctl restart "$SVC"
sudo systemctl enable "$SVC"

Note: On some Debian/Ubuntu builds, the binary path is /usr/bin/prometheus-node-exporter. Adjust ExecStart if needed:

which node_exporter || which prometheus-node-exporter

2) Emit GPU metrics via nvidia-smi into a textfile:

#!/usr/bin/env bash
# /usr/local/bin/gpu_metrics.sh
set -euo pipefail

OUT="/var/lib/node_exporter/textfile_collector/gpu.prom.$$"
TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT

# Requires NVIDIA drivers (nvidia-smi in PATH)
if ! command -v nvidia-smi >/dev/null; then
  exit 0
fi

# Collect per-GPU utilization and memory (MiB)
# CSV: index, name, util%, mem_used, mem_total
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits > "$TMP"

{
  echo "# HELP ai_gpu_utilization_percent GPU utilization percent"
  echo "# TYPE ai_gpu_utilization_percent gauge"
  echo "# HELP ai_gpu_memory_used_bytes GPU memory used in bytes"
  echo "# TYPE ai_gpu_memory_used_bytes gauge"
  echo "# HELP ai_gpu_memory_total_bytes GPU total memory in bytes"
  echo "# TYPE ai_gpu_memory_total_bytes gauge"
  while IFS=, read -r idx name util mem_used mem_total; do
    idx="$(echo "$idx" | xargs)"
    name="$(echo "$name" | xargs | tr ' ' '_')"
    util="$(echo "$util" | xargs)"
    used_bytes=$(( $(echo "$mem_used" | xargs) * 1024 * 1024 ))
    total_bytes=$(( $(echo "$mem_total" | xargs) * 1024 * 1024 ))
    echo "ai_gpu_utilization_percent{gpu=\"$idx\",name=\"$name\"} $util"
    echo "ai_gpu_memory_used_bytes{gpu=\"$idx\",name=\"$name\"} $used_bytes"
    echo "ai_gpu_memory_total_bytes{gpu=\"$idx\",name=\"$name\"} $total_bytes"
  done < "$TMP"
} > "$OUT"

mv "$OUT" "${OUT%%.$$}"

Make it executable and schedule it:

sudo install -m 0755 /usr/local/bin/gpu_metrics.sh /usr/local/bin/gpu_metrics.sh

# Run every minute via systemd timer
cat <<'EOF' | sudo tee /etc/systemd/system/gpu-metrics.service
[Unit]
Description=Emit GPU metrics for node_exporter textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/gpu_metrics.sh
EOF

cat <<'EOF' | sudo tee /etc/systemd/system/gpu-metrics.timer
[Unit]
Description=Run GPU metrics script every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
Unit=gpu-metrics.service

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now gpu-metrics.timer

Now node_exporter will surface ai_gpu_* metrics at :9100/metrics.

4) Wire Prometheus Scrapes, Recording Rules, and Alerts

Prometheus config example (/etc/prometheus/prometheus.yml). Add scrape jobs for:

  • Node Exporter (system + GPU textfile metrics)

  • Your training/inference exporter (Python on 8000)

  • Pushgateway (for batch jobs)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'ai-training'
    static_configs:
      - targets: ['localhost:8000']  # Adjust to your host:port

  - job_name: 'pushgateway'
    honor_labels: true
    static_configs:
      - targets: ['localhost:9091']

Reload Prometheus:

sudo systemctl reload prometheus

Recording and alerting rules (/etc/prometheus/rules/ai.rules.yml):

groups:

- name: ai-workflows
  interval: 30s
  rules:
  - record: job:ai_train_throughput_samples_per_s
    expr: rate(ai_samples_processed_total[5m])

  - alert: AIGPUStarved
    expr: avg_over_time(ai_gpu_utilization_percent[10m]) < 20
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU utilization low for 10m"
      description: "GPUs appear underutilized; check data loader or I/O bottlenecks."

  - alert: AITrainingSlowdown
    expr: job:ai_train_throughput_samples_per_s < 50
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Training throughput dropped below 50 samples/s"
      description: "Investigate CPU/GPU contention or recent code/config changes."

Include it in Prometheus config:

rule_files:
  - /etc/prometheus/rules/*.yml

Reload:

sudo systemctl reload prometheus

Optional: Point Prometheus at Alertmanager if you want Slack/Email/PagerDuty notifications.

5) Real-World Prometheus Workflows for AI

A) Batch training with Pushgateway (Bash)

  • Use Pushgateway when your job is ephemeral and can’t be scraped.
#!/usr/bin/env bash
set -euo pipefail
JOB="nightly_train"
INST="$(hostname)"
PG="http://localhost:9091"

start_ts=$(date +%s)
status=0
trap 'status=$?; end_ts=$(date +%s); dur=$((end_ts-start_ts));
  cat <<METRICS | curl -sS --data-binary @- ${PG}/metrics/job/${JOB}/instance/${INST}
ai_job_duration_seconds ${dur}
ai_job_exit_code ${status}
METRICS
' EXIT

# Your job here:
python3 train.py --epochs 10 --dataset imagenet

Queries to learn from:

  • ai_job_exit_code{job="nightly_train"}

  • sum(increase(ai_job_duration_seconds[24h])) by (job, instance)

B) Inference SLOs Instrument inference latency and error rate:

from prometheus_client import start_http_server, Histogram, Counter

latency = Histogram("ai_inference_seconds", "Inference latency", buckets=[0.01,0.02,0.05,0.1,0.2,0.5,1,2])
errors = Counter("ai_inference_errors_total", "Inference errors", ["code"])

def handle_request(req):
    with latency.time():
        try:
            return infer(req)
        except Exception:
            errors.labels(code="500").inc()
            raise

if __name__ == "__main__":
    start_http_server(8001)
    serve_requests()

Alert examples:

  • p95 latency: histogram_quantile(0.95, sum(rate(ai_inference_seconds_bucket[5m])) by (le)) > 0.2

  • Error spike: rate(ai_inference_errors_total[5m]) > 0

C) GPU cost control Approximate GPU-hours by label (node, team):

  • Record rule:

- record: node:gpu_utilization_ratio
  expr: clamp_min(ai_gpu_utilization_percent / 100, 0)
  • Monthly GPU-hours:
sum_over_time(node:gpu_utilization_ratio[30d]) / 3600

Attach labels (e.g., via node metadata or exporter labels) to slice by team, project, or model.

Operational Tips

  • Retention: Adjust Prometheus retention for long training histories:
# /etc/default/prometheus or service unit ExecStart flags
--storage.tsdb.retention.time=30d

  • Namespacing: Prefix AI metrics (ai_*) to avoid collisions.

  • Label hygiene: Prefer low-cardinality labels (model, dataset) and avoid per-request IDs.

Conclusion and Next Steps

With a few packages and Bash-friendly glue, you can turn opaque AI jobs into measurable, alertable workflows. Start small:

  • Install Prometheus + Node Exporter + Pushgateway.

  • Expose a couple of core metrics from your training loop (throughput, loss).

  • Add GPU visibility via the textfile collector.

  • Write one alert that would have saved you last time.

Then iterate. Pull in inference latency, batch job statuses, and cost-aware GPU utilization. Your future self — and your GPUs — will thank you.

If you found this useful, pick one workload today and add three metrics: throughput, latency, and errors. Wire a 95th-percentile latency alert. You’ll feel the difference the next time things go quiet.

Related posts