Posted on
• Artificial Intelligence

Artificial Intelligence Error Budget Reporting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Bash-first guide to Artificial Intelligence Error Budget Reporting

If your AI service “works on my machine” but feels flaky in production, you don’t have a performance problem—you have a reliability problem. Error budgets turn reliability into a measurable, automatable contract: how much failure your users can tolerate before you must slow down change. In this post, we’ll build an error budget reporter in Bash, wire it to Prometheus or logs, and automate daily reporting on any Linux distro.

What you’ll get:

  • A practical definition of AI error budgets and why they matter

  • A drop-in Bash script to compute SLO attainment, budget remaining, and burn rates

  • Install instructions for apt, dnf, and zypper

  • A log-parsing fallback if you don’t have metrics yet

  • A lightweight automation pattern with systemd timers

Why AI needs error budgets (even more than “traditional” services)

  • AI is probabilistic. You can pass all unit tests and still produce intermittent 5xxs, timeouts, or low-quality responses under load or drift.

  • User perception matters more than averages. A few minutes of elevated error rates can consume an entire month’s error budget.

  • Trade-off guardrails. Error budgets align product velocity (deploys, experiments) with reliability (user trust). If you burn too fast, you slow changes; if you have budget, you can move quickly with confidence.

Terminology, in 30 seconds:

  • SLI (Service Level Indicator): how you measure “good” (e.g., successful inferences and p95 latency under 300 ms).

  • SLO (Service Level Objective): target for SLIs (e.g., 99.5% successful requests over 30 days).

  • Error budget: 1 − SLO (e.g., 0.5% of requests can fail in the period).

  • Burn rate: how fast you’re spending the error budget relative to allowance.

Prerequisites (install once)

We’ll use curl, jq, bc, and gawk in the examples.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq bc gawk

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y curl jq bc gawk

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq bc gawk

Step 1: Define useful SLIs for AI services

Pick at least one “golden SLI”; two is better:

  • Availability SLI: fraction of requests that return non-5xx (and optionally exclude 429 or include timeouts if they hurt UX).

  • Latency SLI: p95 under your target (e.g., 300 ms).

  • Optional quality SLI: fraction of responses meeting a threshold (e.g., acceptance rate, toxicity score below X). This usually needs application-specific metrics, but you can still track it with simple counters.

Example metric families (Prometheus labels shown as hints):

  • Total requests: http_requests_total{job="inference"}

  • Failures: http_requests_total{job="inference",status=~"5..|429"}

  • Latency histogram: http_request_duration_seconds_bucket{job="inference"}

Note: adjust label names to your environment.

Step 2: Compute the error budget with Bash

  • SLO target (T): 0.995 means 99.5% success.

  • Error budget: 1 − T (here, 0.5% of requests can be “bad”).

  • Within a window (e.g., last 30 days), we compare observed bad requests to allowed bad requests.

Step 3: Prometheus-backed error budget reporter (Bash)

Save this as ai-error-budget.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

# Configuration (override via environment variables)
PROM_URL="${PROM_URL:-http://localhost:9090}"
JOB_MATCHER="${JOB_MATCHER:-job=\"inference\"}"   # e.g., 'job="inference",env="prod"'
SLO_TARGET="${SLO_TARGET:-0.995}"                 # 99.5% good
WINDOW="${WINDOW:-30d}"                           # PromQL range selector, e.g., 7d, 30d
SHORT_WINDOW="${SHORT_WINDOW:-1h}"                # for fast-burn alerting
LONG_WINDOW="${LONG_WINDOW:-6h}"                  # for slow-burn alerting

calc() { echo "scale=10; $*" | bc -l; }

prom() {
  local q="$1"
  curl -sG "$PROM_URL/api/v1/query" --data-urlencode "query=$q" \
    | jq -r '.data.result[0].value[1] // "0"'
}

# Queries (adjust metric names/labels to match your setup)
total_q="sum(increase(http_requests_total{$JOB_MATCHER}[$WINDOW]))"
fail_q="sum(increase(http_requests_total{$JOB_MATCHER,status=~\"5..|429\"}[$WINDOW]))"

short_total_q="sum(increase(http_requests_total{$JOB_MATCHER}[$SHORT_WINDOW]))"
short_fail_q="sum(increase(http_requests_total{$JOB_MATCHER,status=~\"5..|429\"}[$SHORT_WINDOW]))"

long_total_q="sum(increase(http_requests_total{$JOB_MATCHER}[$LONG_WINDOW]))"
long_fail_q="sum(increase(http_requests_total{$JOB_MATCHER,status=~\"5..|429\"}[$LONG_WINDOW]))"

total=$(prom "$total_q")
fail=$(prom "$fail_q")
allowed_error=$(calc "1 - $SLO_TARGET")

if [ "$(echo "$total > 0" | bc -l)" -eq 1 ]; then
  error_ratio=$(calc "$fail / $total")
else
  error_ratio="0"
fi

budget_consumed=$(calc "$error_ratio / $allowed_error")   # ratio of budget spent
budget_remaining=$(calc "1 - $budget_consumed")
slo_attained=$(calc "1 - $error_ratio")

# Burn rates
short_total=$(prom "$short_total_q")
short_fail=$(prom "$short_fail_q")
long_total=$(prom "$long_total_q")
long_fail=$(prom "$long_fail_q")

short_err=$( [ "$(echo "$short_total > 0" | bc -l)" -eq 1 ] && calc "$short_fail / $short_total" || echo "0" )
long_err=$(  [ "$(echo "$long_total  > 0" | bc -l)" -eq 1 ] && calc "$long_fail  / $long_total"  || echo "0" )

short_burn=$( [ "$(echo "$allowed_error > 0" | bc -l)" -eq 1 ] && calc "$short_err / $allowed_error" || echo "0" )
long_burn=$(  [ "$(echo "$allowed_error > 0" | bc -l)" -eq 1 ] && calc "$long_err  / $allowed_error"  || echo "0" )

# Markdown report
printf "\n# AI Error Budget Report\n\n"
printf "- Window: %s\n" "$WINDOW"
printf "- SLO target: %.3f\n" "$SLO_TARGET"
printf "- Observed total: %.0f, failures: %.0f\n\n" "$total" "$fail"

printf "Metric | Value\n"
printf "---|---\n"
printf "SLO attained | %.4f\n" "$slo_attained"
printf "Error budget consumed | %.2f%%\n" "$(calc "$budget_consumed * 100")"
printf "Error budget remaining | %.2f%%\n\n" "$(calc "$budget_remaining * 100")"

printf "Burn rate (short: %s) | %.3f\n" "$SHORT_WINDOW" "$short_burn"
printf "Burn rate (long: %s) | %.3f\n\n" "$LONG_WINDOW" "$long_burn"

# Exit status for CI/automation (non-zero if in danger)
# Example policy: page if short_burn > 2 AND long_burn > 1
page_short=$(echo "$short_burn > 2" | bc -l)
page_long=$(echo "$long_burn  > 1" | bc -l)

if [ "$page_short" -eq 1 ] && [ "$page_long" -eq 1 ]; then
  printf "Alert: Error budget burn is too high (short>2 and long>1). Consider halting risky changes.\n"
  exit 2
fi

exit 0

Usage:

chmod +x ai-error-budget.sh

# Minimum (default localhost Prometheus, job="inference", 30 days, 99.5% SLO)
./ai-error-budget.sh

# With environment overrides
PROM_URL="http://prometheus:9090" \
JOB_MATCHER='job="inference",env="prod"' \
SLO_TARGET=0.999 \
WINDOW=7d \
SHORT_WINDOW=1h \
LONG_WINDOW=24h \
./ai-error-budget.sh

What you’ll see (example):

# AI Error Budget Report

- Window: 30d

- SLO target: 0.995

- Observed total: 12899402, failures: 5223

Metric | Value
---|---
SLO attained | 0.9996
Error budget consumed | 7.88%
Error budget remaining | 92.12%

Burn rate (short: 1h) | 0.35
Burn rate (long: 6h) | 0.41

Tip: Use the script’s exit code in CI/CD to gate risky rollouts if burn rates spike.

Step 4: No Prometheus? Parse your logs with awk

If you don’t yet expose metrics, you can estimate availability from Nginx/Apache access logs. This example scans the last 24 hours of Nginx combined logs and treats 5xx as failures.

# Adjust paths and hours
LOG_DIR="/var/log/nginx"
HOURS=24
SINCE_EPOCH=$(date -d "$HOURS hours ago" +%s)

# Concatenate current + rotated logs (gz ok)
log_stream() {
  find "$LOG_DIR" -maxdepth 1 -type f \( -name "access.log" -o -name "access.log.*" \) \
    -print0 \
  | xargs -0 -I{} sh -c '
      case "{}" in
        *.gz) zcat "{}" ;;
        *)    cat "{}"  ;;
      esac
    '
}

log_stream | gawk -v since="$SINCE_EPOCH" '
function mon(m){ split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a," "); for(i=1;i<=12;i++) if(a[i]==m) return i; return 0 }
{
  # Combined log: ... [10/Mar/2026:13:42:55 +0000] "GET /..." status bytes ...
  if (match($4, /\[([0-9]{2})\/([A-Za-z]{3})\/([0-9]{4}):([0-9]{2}):([0-9]{2}):([0-9]{2})/, a)) {
    ts = mktime(a[3]" "mon(a[2])" "a[1]" "a[4]" "a[5]" "a[6])
    if (ts >= since) {
      total++
      status = $9
      if (status ~ /^5/) fail++
    }
  }
}
END{
  if (total>0){
    err = fail/total
    slo = 0.995
    allowed = 1 - slo
    consumed = err/allowed
    rem = 1 - consumed
    printf("# AI Error Budget Report (logs, last %d h)\n\n", (systime()-since)/3600)
    printf("Observed total: %d, failures: %d\n\n", total, fail)
    printf("SLO attained: %.4f\n", 1-err)
    printf("Error budget consumed: %.2f%%\n", consumed*100)
    printf("Error budget remaining: %.2f%%\n", rem*100)
  } else {
    print "No log entries in range."
  }
}'

This won’t capture latency or model-quality failures, but it’s a fast starting point.

Step 5: Automate with systemd timers (lightweight, portable)

Create a service unit to run the reporter daily and drop a Markdown file to a shared location.

/etc/systemd/system/ai-error-budget.service

[Unit]
Description=AI Error Budget Reporter

[Service]
Type=oneshot
Environment=PROM_URL=http://prometheus:9090
Environment=JOB_MATCHER=job="inference",env="prod"
Environment=SLO_TARGET=0.995
Environment=WINDOW=30d
Environment=SHORT_WINDOW=1h
Environment=LONG_WINDOW=6h
ExecStart=/usr/local/bin/ai-error-budget.sh > /var/www/html/ai-error-budget.md

/etc/systemd/system/ai-error-budget.timer

[Unit]
Description=Run AI Error Budget Reporter daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo cp ai-error-budget.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/ai-error-budget.sh
sudo systemctl daemon-reload
sudo systemctl enable --now ai-error-budget.timer

Optional: Wire the exit code to alerting:

  • Send to Slack or email if the service exits with 2.

  • Or connect to your incident system via a wrapper script.

Real-world patterns to watch

  • Spiky burn (deployment breakage): short-window burn rate > 2 while long-window is low. Roll back or pause deploys.

  • Slow burn (data drift, dependency brownout): both short and long > 1 for hours. Prioritize reliability work, add circuit breakers.

  • Chronic latency misses: availability looks fine but p95/p99 latency breaks user trust. Add a latency SLI and alert on its burn rate too.

Conclusion and next steps

Error budgets bring discipline to AI operations: they quantify reliability, throttle risky changes when needed, and build user trust. You now have:

  • A Bash script to compute SLO attainment, budget consumption, and burn rates via Prometheus (or logs).

  • A portable automation pattern to publish daily Markdown reports, with clear thresholds for alerting.

Your next step: 1) Set SLO_TARGET and JOB_MATCHER for your production inference. 2) Run the script manually and verify numbers. 3) Enable the systemd timer and share the Markdown report with your team. 4) Add a latency SLI and a simple “page if burn too high” policy.

When in doubt, protect the budget first—velocity follows reliability.

Related posts