Posted on
• Artificial Intelligence

Artificial Intelligence Observability Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Observability Best Practices (for Linux/Bash folks)

AI systems don’t fail loudly—they drift, degrade, and quietly burn money. If you’ve ever chased a “the model feels slower” ticket or discovered a postmortem week after an undetected regression, you already know: observability for AI isn’t optional. It’s your early‑warning system and your safety net.

This guide shows you how to stand up practical, Bash‑friendly observability for AI workloads on Linux. You’ll get concrete steps, install commands for apt/dnf/zypper, and real snippets you can paste into production today.

Why AI observability is different (and valid)

Traditional app monitoring focuses on CPU, memory, HTTP status codes, and latency. AI workloads add new failure modes:

  • Data drift and schema skew quietly corrupt predictions.

  • Model versions behave differently under the same traffic.

  • Latency, cost, and token usage (for LLMs) vary per request.

  • Safety signals (toxicity, PII leakage) must be tracked over time.

Observability therefore has to cover: 1) Service health (availability, latency, errors) 2) Model quality (accuracy, calibration, drift) 3) Data integrity (freshness, null rates, distribution shift) 4) Cost/safety (token usage, rate limits, policy violations)

The good news: you can capture a lot of this with standard Linux tools, Prometheus, and Grafana—plus a handful of disciplined practices.

Quick install: the toolbox

The examples below use jq, curl, Prometheus, Pushgateway, and Grafana. Pick the commands for your package manager.

  • jq and curl

    • apt:
    sudo apt-get update
sudo apt-get install -y jq curl
    • dnf:
    sudo dnf install -y jq curl
    • zypper:
    sudo zypper refresh
sudo zypper install -y jq curl

  • Prometheus

    • apt:
    sudo apt-get update
sudo apt-get install -y prometheus
sudo systemctl enable --now prometheus
    • dnf:
    sudo dnf install -y prometheus
sudo systemctl enable --now prometheus
    • zypper:
    sudo zypper refresh
sudo zypper install -y prometheus
sudo systemctl enable --now prometheus

  • Prometheus Pushgateway (for pushing metrics from Bash/batch jobs)

    • apt:
    sudo apt-get update
sudo apt-get install -y prometheus-pushgateway
sudo systemctl enable --now prometheus-pushgateway
    • dnf:
    sudo dnf install -y prometheus-pushgateway || sudo dnf install -y pushgateway
sudo systemctl enable --now pushgateway || sudo systemctl enable --now prometheus-pushgateway
    • zypper:
    sudo zypper refresh
sudo zypper install -y prometheus-pushgateway
sudo systemctl enable --now prometheus-pushgateway
    • If your repo doesn’t have it, use a tarball:
    cd /tmp
curl -LO https://github.com/prometheus/pushgateway/releases/latest/download/pushgateway-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64.tar.gz
tar xzf pushgateway-*-amd64.tar.gz
sudo mv pushgateway-*-amd64/pushgateway /usr/local/bin/
/usr/local/bin/pushgateway --web.listen-address=":9091" &

  • Grafana (official repository recommended)

    • apt:
    sudo apt-get install -y apt-transport-https software-properties-common
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl enable --now grafana-server
    • dnf:
    sudo rpm --import https://packages.grafana.com/gpg.key
sudo dnf config-manager --add-repo https://packages.grafana.com/oss/rpm
sudo dnf install -y grafana
sudo systemctl enable --now grafana-server
    • zypper:
    sudo rpm --import https://packages.grafana.com/gpg.key
sudo zypper addrepo https://packages.grafana.com/oss/rpm grafana
sudo zypper refresh
sudo zypper install -y grafana
sudo systemctl enable --now grafana-server

Configure Prometheus to scrape Pushgateway by editing /etc/prometheus/prometheus.yml and adding:

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['localhost:9091']

Then:

sudo systemctl restart prometheus

1) Define AI‑specific SLIs and SLOs (what to watch, and how much you care)

Don’t boil the ocean. Pick a small set of signals that reflect user experience and model quality.

Example SLIs:

  • Service: p95_latency_ms, error_rate, throughput_rps

  • Model: acceptance_rate, click_through_rate, exact_match, toxicity_rate

  • Data: null_feature_rate, schema_violation_rate, input_drift_ks_stat

  • Cost/LLM: tokens_per_request, cost_per_minute, rate_limit_hits

Example SLOs:

  • “p95_latency_ms < 300 for 99% of minutes”

  • “toxicity_rate < 0.5% rolling 1h”

  • “input_drift_ks_stat < 0.1 over 24h”

Express these as Prometheus alerts (see section 5). If you can’t alert on it, you don’t really own it.

2) Emit structured logs from Bash (JSON logs you can query)

Unstructured logs hide your model signals. Log in JSON so you can slice by model, version, and user segment. Here’s a tiny Bash logger:

#!/usr/bin/env bash
log_json() {
  local level="$1" latency_ms="$2" status="$3" input_hash="$4"
  local ts model="${MODEL_NAME:-resnet50}" ver="${MODEL_VERSION:-2024-07-01}"
  ts=$(date -Is)
  printf '{"ts":"%s","level":"%s","model":"%s","model_version":"%s","latency_ms":%d,"status":"%s","input_hash":"%s"}\n' \
    "$ts" "$level" "$model" "$ver" "$latency_ms" "$status" "$input_hash"
}

# Example usage in an inference wrapper
payload='{"input":"hello world"}'
start_ns=$(date +%s%N)
sleep $((RANDOM % 150))e-3  # simulate inference latency
latency_ms=$(( ( $(date +%s%N) - start_ns ) / 1000000 ))
hash=$(printf "%s" "$payload" | sha256sum | awk '{print $1}')
log_json info "$latency_ms" ok "$hash" | tee -a /var/log/ai/infer.log

Quick queries with jq:

  • p95 latency (naive, all in memory):
jq -s '[.[] | .latency_ms] | sort | .[(length*0.95|floor)]' /var/log/ai/infer.log
  • Error rate last 10 minutes:
awk -v since="$(date -u -d '10 minutes ago' +%s)" '
  {
    cmd = "jq -r \".ts, .status\" <<<$0"
    cmd | getline ts; close(cmd)
    cmd = "date -d \"" ts "\" +%s"; cmd | getline t; close(cmd)
    if (t >= since) { total++; if (index($0, "\"status\":\"ok\"") == 0) err++ }
  } END { if (total>0) print err/total; else print 0 }' /var/log/ai/infer.log

Tip: ship these logs to your central log store later, but start now with structured files.

3) Push AI metrics from Bash to Prometheus (with Pushgateway)

Batch jobs and scripts can’t always expose an HTTP endpoint. Use Pushgateway to get metrics into Prometheus quickly.

Push a few key SLIs per job/model/version:

#!/usr/bin/env bash
PGW="${PGW:-http://localhost:9091}"

job="ai_infer"
instance="$(hostname)"
model="${MODEL_NAME:-resnet50}"
ver="${MODEL_VERSION:-2024-07-01}"

observe_latency() {
  local ms="$1"
  cat <<EOF | curl -s --data-binary @- "$PGW/metrics/job/$job/instance/$instance/model/$model/version/$ver"
# TYPE ai_infer_latency_ms histogram
ai_infer_latency_ms{le="50"}  0
ai_infer_latency_ms{le="100"} 0
ai_infer_latency_ms{le="200"} 1
ai_infer_latency_ms{le="500"} 1
ai_infer_latency_ms{le="+Inf"} 1
ai_infer_latency_ms_sum $ms
ai_infer_latency_ms_count 1
EOF
}

set_gauges() {
  local tokens="$1" cost="$2"
  cat <<EOF | curl -s --data-binary @- "$PGW/metrics/job/$job/instance/$instance/model/$model/version/$ver"
# TYPE ai_infer_tokens gauge
ai_infer_tokens $tokens
# TYPE ai_infer_cost_usd gauge
ai_infer_cost_usd $cost
EOF
}

# Simulate one inference
start=$(date +%s%3N); sleep $((RANDOM % 150))e-3; end=$(date +%s%3N)
lat=$(( end - start ))
tokens=$(( 50 + RANDOM % 200 ))
cost=$(awk -v t="$tokens" 'BEGIN{ printf "%.6f", t*0.000002 }')  # fake LLM pricing

observe_latency "$lat"
set_gauges "$tokens" "$cost"
echo "Pushed metrics: latency=${lat}ms tokens=${tokens} cost=$cost"

Then add a Grafana dashboard for:

  • ai_infer_latency_ms (histogram_quantile for p95)

  • ai_infer_tokens (gauge)

  • ai_infer_cost_usd (gauge sum over time)

PromQL examples:

  • p95 latency by model/version (5m window):
histogram_quantile(0.95, sum by (le, model, version) (rate(ai_infer_latency_ms_bucket[5m])))
  • Cost per minute:
sum(rate(ai_infer_cost_usd[1m]))

4) Guard against regressions: canary, shadow, and drift checks

Ship models gently and watch them like a hawk.

  • Canary route 1–5% of traffic to the new version, compare side‑by‑side.

  • Shadow run the new model on mirrored traffic without impacting users.

  • Bucket metrics by model/version and segment.

Simple canary router in Bash:

#!/usr/bin/env bash
# 95% to stable, 5% to canary
rand=$((RANDOM % 100))
if [ "$rand" -lt 5 ]; then
  export MODEL_VERSION="2024-07-15"
  backend_url="http://canary:8000/infer"
else
  export MODEL_VERSION="2024-07-01"
  backend_url="http://stable:8000/infer"
fi

resp=$(curl -s -o /tmp/resp.json -w "%{time_total}" -H "Content-Type: application/json" -d @/tmp/payload.json "$backend_url")
lat_ms=$(awk -v s="$resp" 'BEGIN{printf "%d", s*1000}')
status="ok"; [ -s /tmp/resp.json ] || status="error"

hash=$(sha256sum /tmp/payload.json | awk '{print $1}')
printf -v log '{"ts":"%s","model":"%s","model_version":"%s","latency_ms":%d,"status":"%s"}\n' \
  "$(date -Is)" "${MODEL_NAME:-resnet50}" "$MODEL_VERSION" "$lat_ms" "$status"
echo "$log" | tee -a /var/log/ai/infer.log

# Push key metrics (reuse functions from section 3, or inline)

Detect drift simply by tracking feature stats over time. Example: monitor input length distribution and alert if it shifts.

Emit a drift metric in Bash (toy example):

len=$(jq '.input | length' /tmp/payload.json)
cat <<EOF | curl -s --data-binary @- http://localhost:9091/metrics/job/ai_infer_drift/model/${MODEL_NAME:-resnet50}/version/${MODEL_VERSION:-2024-07-01}
# TYPE ai_input_len gauge
ai_input_len $len
EOF

Then watch for distribution changes (e.g., p95 length jumping). For real drift, compute KS statistic offline and push as ai_input_drift_ks.

5) Automate alerts (latency, errors, drift, and cost)

Add alert rules in /etc/prometheus/rules/ai_alerts.yml and include them from prometheus.yml (rule_files). Example rules:

groups:

- name: ai-slos
  rules:
  - alert: AIP95LatencyTooHigh
    expr: histogram_quantile(0.95, sum by (le, model, version) (rate(ai_infer_latency_ms_bucket[5m]))) > 0.3
    for: 10m
    labels: {severity: page}
    annotations:
      summary: "High p95 latency for {{ $labels.model }}:{{ $labels.version }}"
      description: "p95 > 300ms for 10m"

  - alert: AITokenCostSpike
    expr: sum(rate(ai_infer_cost_usd[5m])) > 5
    for: 5m
    labels: {severity: ticket}
    annotations:
      summary: "Cost spike"
      description: "Estimated cost/min > $5 for 5m"

  - alert: AIDataDrift
    expr: avg_over_time(ai_input_len[30m]) / avg_over_time(ai_input_len[24h]) > 1.5
    for: 30m
    labels: {severity: warn}
    annotations:
      summary: "Potential input distribution shift"
      description: "Average input length up 50% vs 24h baseline"

Reload Prometheus or restart the service:

sudo systemctl restart prometheus

Wire Alertmanager to Slack/PagerDuty to close the loop.

Real‑world, minimal end‑to‑end test

1) Install jq, curl, Prometheus, Pushgateway, Grafana (see install section). 2) Add Pushgateway scrape to Prometheus and restart it. 3) Run the Bash push script from section 3 a few times to emit metrics. 4) In Grafana: - Add a Prometheus data source pointing at http://localhost:9090 - Create a dashboard with the PromQL from section 3 5) Add alerts (section 5), and verify they trigger by forcing latency spikes:

for i in {1..60}; do MODEL_NAME=demo MODEL_VERSION=2024-07-15 sleep 0.6; <paste your push script>; done

You now have observability from logging to metrics, across model versions, in a few minutes.

Conclusion and next steps

AI observability isn’t just charts—it’s your contract with users:

  • Define the few SLIs/SLOs that matter.

  • Emit structured logs and metrics by model and version.

  • Visualize and alert on latency, quality, drift, and cost.

  • Roll out models safely with canaries and shadowing.

Your next step:

  • Stand up Prometheus + Pushgateway + Grafana.

  • Wrap your inference entrypoint with the JSON logger and metric pusher.

  • Set two SLOs and two alerts; iterate from there.

If you want a ready‑to‑use starter, turn the snippets above into:

  • scripts/log_json.sh

  • scripts/push_metrics.sh

  • ops/prometheus.yml

  • ops/ai_alerts.yml

Start small, measure what matters, and let the signals guide your model ops.

Related posts