Posted on
• Artificial Intelligence

Artificial Intelligence Service Level Objectives Explained

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Service Level Objectives (SLOs) Explained — A Bash-Friendly Guide

Shipping AI without SLOs is like flying blind: everything’s fine… until a 2 a.m. page because the model got “creative,” latency spiked, or costs quietly doubled. If you run Linux, you already have everything you need to start measuring and enforcing AI SLOs with a few Bash scripts and standard OSS tools.

This post explains what AI SLOs are, why they’re different from classic web SLOs, and how to stand up a pragmatic, Bash-based SLO practice in under an hour using curl, jq, Prometheus, and Grafana. You’ll get copy-paste snippets and install steps for apt, dnf, and zypper.

The problem and why SLOs matter for AI

Classic SLOs (availability, latency, error rate) still apply to AI inference services, but AI introduces new, high-impact dimensions:

  • Quality is probabilistic: acceptance rate, helpfulness, or safety aren’t simple 2xx/5xx.

  • Costs are elastic: token usage or GPU time can blow budgets invisibly.

  • Data drift hurts: RAG indexes stale out, models update, prompts evolve.

  • Safety/compliance: toxicity or PII leakage rates matter as much as uptime.

Without explicit, measurable objectives, your “works on my machine” success can become “why is prod down/costly/unusable” overnight.

Key terms, mapped to AI

  • SLI (Service Level Indicator): a measurable signal.

    • Examples: p95 latency, success ratio, average tokens/request, “human-accepted” rate, index freshness seconds.

  • SLO (Objective): target for an SLI.

    • Examples: “p95 latency ≤ 800ms,” “≥ 99% success,” “avg tokens ≤ 1,500,” “acceptance ≥ 95%,” “RAG index freshness ≤ 6h.”

  • SLA (Agreement): a contract (business/customer-facing). You’ll derive SLAs from SLOs, but you’ll operate day-to-day on SLOs.

Quick install: tools we’ll use

We’ll use curl (HTTP), jq (JSON parsing), bc (math), Prometheus + Pushgateway (metrics), and Grafana (dashboards).

  • Debian/Ubuntu (apt)

    sudo apt update
sudo apt install -y curl jq bc prometheus prometheus-pushgateway grafana
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-pushgateway
sudo systemctl enable --now grafana-server

  • Fedora/RHEL/CentOS (dnf)

    # On RHEL/CentOS, you may need: sudo dnf install -y epel-release
sudo dnf install -y curl jq bc prometheus pushgateway grafana
sudo systemctl enable --now prometheus
sudo systemctl enable --now pushgateway
sudo systemctl enable --now grafana-server

  • openSUSE/SLE (zypper)

    sudo zypper refresh
sudo zypper install -y curl jq bc prometheus prometheus-pushgateway grafana
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-pushgateway
sudo systemctl enable --now grafana-server

Tell Prometheus to scrape the Pushgateway:

sudo sed -i '/^scrape_configs:/a \
  - job_name: "pushgateway"\n    static_configs:\n      - targets: ["localhost:9091"]' /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus

4 actionable steps to operationalize AI SLOs with Bash

1) Pick 3–5 SLOs that actually move the needle

Start small and practical. Examples you can defend in a postmortem:

  • Availability: “≥ 99.5% of synthetic requests succeed (2xx) over 7 days.”

  • Latency: “p95 ≤ 800ms for prompts ≤ 2k tokens during business hours.”

  • Cost: “avg cost/request ≤ $0.02 over 1 day.”

  • Quality proxy: “human-accepted rate ≥ 95% over 7 days” (feedback loop).

  • Freshness (RAG): “index freshness ≤ 6h (max age of latest document).”

You can evolve these as you learn where your system fails in reality.

2) Probe your AI endpoint with a tiny Bash script

This synthetic probe hits your inference endpoint, times it, parses token usage, estimates cost, and pushes metrics to the Pushgateway.

Create ai_slo_probe.sh:

#!/usr/bin/env bash
set -euo pipefail

: "${ENDPOINT_URL:?Set ENDPOINT_URL to your inference endpoint}"
JOB_NAME="${JOB_NAME:-ai_slo}"
PUSHGATEWAY_URL="${PUSHGATEWAY_URL:-http://localhost:9091}"
TOKEN_PRICE_PER_1K="${TOKEN_PRICE_PER_1K:-0.00}"   # e.g., 0.005 for $0.005/1k tokens
PROMPT="${PROMPT:-What is the capital of France?}"
TMP="$(mktemp)"
cleanup(){ rm -f "$TMP"; }
trap cleanup EXIT

# Customize headers/body to match your service.
# Example expects JSON with optional .usage.total_tokens or .usage.prompt_tokens+.completion_tokens
HTTP_AND_TIME="$(curl -sS -o "$TMP" \
  -w '%{http_code} %{time_total}\n' \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg p "$PROMPT" '{input:$p}')" \
  "$ENDPOINT_URL")"

read -r HTTP_CODE TIME_TOTAL <<<"$HTTP_AND_TIME"
SUCCESS=0
[[ "$HTTP_CODE" =~ ^2 ]] && SUCCESS=1

TOKENS="$(jq -r '.usage.total_tokens // (.usage.prompt_tokens + .usage.completion_tokens) // 0' "$TMP" 2>/dev/null || echo 0)"
# Cost estimation (simple): tokens/1000 * price
COST="$(echo "scale=6; ($TOKENS/1000)*$TOKEN_PRICE_PER_1K" | bc)"

# Optional: naive safety/quality flags if your service returns them.
# FLAGGED: 1 if unsafe/toxic/blocked; 0 otherwise.
FLAGGED="$(jq -r '.safety.flagged // 0' "$TMP" 2>/dev/null || echo 0)"

# Push gauges to Pushgateway. These are synthetic SLIs sampled per probe.
# Grouping labels identify this probe series by job, instance, and endpoint hash.
EPHASH="$(printf '%s' "$ENDPOINT_URL" | md5sum | awk '{print $1}')"
cat <<EOF | curl --silent --show-error --data-binary @- \
  "$PUSHGATEWAY_URL/metrics/job/$JOB_NAME/instance/$(hostname)/endpoint/$EPHASH"
# TYPE ai_slo_success gauge
ai_slo_success{endpoint="$EPHASH"} $SUCCESS
# TYPE ai_slo_latency_seconds gauge
ai_slo_latency_seconds{endpoint="$EPHASH"} $TIME_TOTAL
# TYPE ai_slo_tokens gauge
ai_slo_tokens{endpoint="$EPHASH"} $TOKENS
# TYPE ai_slo_cost_dollars gauge
ai_slo_cost_dollars{endpoint="$EPHASH"} $COST
# TYPE ai_slo_output_flagged gauge
ai_slo_output_flagged{endpoint="$EPHASH"} $FLAGGED
EOF

echo "Probed $ENDPOINT_URL -> status=$HTTP_CODE success=$SUCCESS latency=${TIME_TOTAL}s tokens=$TOKENS cost=\$${COST} flagged=$FLAGGED"

Make it executable and run:

chmod +x ./ai_slo_probe.sh
ENDPOINT_URL="http://localhost:8080/infer" TOKEN_PRICE_PER_1K=0.005 ./ai_slo_probe.sh

Automate with cron (every minute):

( crontab -l 2>/dev/null; echo '* * * * * ENDPOINT_URL="http://localhost:8080/infer" TOKEN_PRICE_PER_1K=0.005 /path/to/ai_slo_probe.sh >> /var/log/ai_slo_probe.log 2>&1' ) | crontab -

Tip: This is a synthetic SLI. For production-grade coverage, also instrument your application to emit request counters and histograms. But synthetic checks catch outages, spikes, and regressions fast.

3) Turn SLIs into PromQL queries and alerts

With the gauges above and a 1-minute probe, you can express SLOs in Prometheus terms:

  • Availability (7-day target ≥ 99.5%):
# 1 - avg_over_time(success) is the observed error fraction.
(1 - avg_over_time(ai_slo_success[7d])) > (1 - 0.995)
  • Latency (p95 ≤ 800ms over 6h):
quantile_over_time(0.95, ai_slo_latency_seconds[6h]) > 0.8
  • Cost budget (avg ≤ $0.02 over 1d):
avg_over_time(ai_slo_cost_dollars[1d]) > 0.02
  • Safety (flag rate ≤ 0.5% over 7d):
avg_over_time(ai_slo_output_flagged[7d]) > 0.005

Create an alert rule file, e.g., /etc/prometheus/rules/ai_slo.yml:

groups:

- name: ai_slo
  rules:
  - alert: AIAvailabilityBudgetBurn
    expr: (1 - avg_over_time(ai_slo_success[2h])) > (1 - 0.995)
    for: 15m
    labels:
      severity: page
    annotations:
      summary: "AI availability SLO burn"
      description: "Observed error fraction > 0.5% over 2h"

  - alert: AILatencyP95High
    expr: quantile_over_time(0.95, ai_slo_latency_seconds[1h]) > 0.8
    for: 10m
    labels:
      severity: ticket
    annotations:
      summary: "p95 latency > 800ms"
      description: "Sustained high latency in the last hour"

  - alert: AICostPerRequestHigh
    expr: avg_over_time(ai_slo_cost_dollars[24h]) > 0.02
    for: 30m
    labels:
      severity: ticket
    annotations:
      summary: "Average cost per request too high"
      description: "Reduce context size, caching, or route to cheaper model"

Validate and load:

promtool check rules /etc/prometheus/rules/ai_slo.yml
sudo sed -i '/^rule_files:/a \  - "/etc/prometheus/rules/ai_slo.yml"' /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus

4) Add real-world signals: freshness and acceptance rate

  • RAG index freshness (max age of last ingest). Suppose a file timestamp marks the newest doc:
#!/usr/bin/env bash
# ai_slo_freshness.sh
set -euo pipefail
INDEX_TS_FILE="/var/lib/myrag/last_ingest_timestamp" # epoch seconds
PUSHGATEWAY_URL="${PUSHGATEWAY_URL:-http://localhost:9091}"
JOB_NAME="${JOB_NAME:-ai_slo}"
EPHASH="rag_index"

NOW="$(date +%s)"
LAST="$(cat "$INDEX_TS_FILE")"
AGE="$(( NOW - LAST ))"

cat <<EOF | curl --silent --show-error --data-binary @- \
  "$PUSHGATEWAY_URL/metrics/job/$JOB_NAME/instance/$(hostname)/endpoint/$EPHASH"
# TYPE ai_slo_index_freshness_seconds gauge
ai_slo_index_freshness_seconds{endpoint="$EPHASH"} $AGE
EOF

Cron it hourly and alert if max_over_time(ai_slo_index_freshness_seconds[6h]) > 21600 (6h).

  • Human acceptance rate. If you capture user feedback to a JSONL file like:
{"accepted":true}
{"accepted":false}
...

You can push a rolling acceptance ratio:

#!/usr/bin/env bash
# ai_slo_acceptance.sh
set -euo pipefail
LOG="/var/log/ai_feedback.jsonl"
PUSHGATEWAY_URL="${PUSHGATEWAY_URL:-http://localhost:9091}"
JOB_NAME="${JOB_NAME:-ai_slo}"
EPHASH="acceptance"

# Last 1000 feedbacks as a sliding window (tune as needed)
ACCEPTS="$(tail -n 1000 "$LOG" | jq -r 'select(.accepted==true) | 1' | wc -l || echo 0)"
TOTAL="$(tail -n 1000 "$LOG" | wc -l || echo 0)"
RATIO=0
if [ "$TOTAL" -gt 0 ]; then
  RATIO="$(echo "scale=6; $ACCEPTS/$TOTAL" | bc)"
fi

cat <<EOF | curl --silent --show-error --data-binary @- \
  "$PUSHGATEWAY_URL/metrics/job/$JOB_NAME/instance/$(hostname)/endpoint/$EPHASH"
# TYPE ai_slo_acceptance_ratio gauge
ai_slo_acceptance_ratio{endpoint="$EPHASH"} $RATIO
EOF

Alert if avg_over_time(ai_slo_acceptance_ratio[7d]) < 0.95.

Real-world SLO templates you can use today

  • Chat/RAG assistant

    • Success: ≥ 99.5% 2xx on synthetic checks over 7d
    • Latency: p95 ≤ 1.2s for ≤ 2k prompt tokens
    • Cost: avg ≤ $0.015/request over 1d
    • Freshness: index age ≤ 6h
    • Safety: flagged rate ≤ 0.5%

  • Batch vision pipeline

    • Throughput: ≥ 95% jobs complete within SLA window
    • Latency: p95 per image ≤ 300ms
    • Quality proxy: human acceptance ≥ 98%
    • Cost: avg GPU-minutes/job ≤ target

  • Search with reranker

    • Availability: ≥ 99.9% success
    • Latency: p95 pipeline ≤ 250ms
    • Cost: avg tokens/query ≤ 500
    • Safety: zero PII in logs (periodic scan results reported as gauge)

These are measurable, actionable, and easy to display in Grafana.

Grafana in one minute

  • Access Grafana (default is http://localhost:3000, admin/admin on first login).

  • Add Prometheus data source: URL http://localhost:9090.

  • Create a dashboard with panels:

    • “Availability (7d)” → 1 - avg_over_time(ai_slo_success[7d])
    • “p95 latency (1h)” → quantile_over_time(0.95, ai_slo_latency_seconds[1h])
    • “Avg cost (1d)” → avg_over_time(ai_slo_cost_dollars[1d])
    • “Acceptance (7d)” → avg_over_time(ai_slo_acceptance_ratio[7d])
    • “Index freshness (6h)” → max_over_time(ai_slo_index_freshness_seconds[6h])

Color thresholds to match your SLOs.

Common pitfalls (and how to avoid them)

  • Measuring only uptime: AI can be “up” but low-quality, unsafe, or too expensive.

  • Ignoring input size: latency/cost SLOs should be scoped to prompt/input lengths.

  • No synthetic checks: if you only rely on app counters, you’ll miss blackouts from the outside.

  • No feedback loop: without human acceptance data, quality SLOs drift into wishful thinking.

  • All-or-nothing alerts: use windows and “for” durations; page for availability, ticket for cost/latency.

Conclusion and next steps (CTA)

You don’t need a new platform to get control of AI reliability, quality, and cost. Start with:

1) Pick 3–5 SLOs that reflect user value and budget.
2) Drop in the Bash probes above and wire Prometheus → Grafana.
3) Add one alert that would have prevented your last incident.
4) Iterate weekly: tune thresholds, add a quality signal, and tie alerts to runbooks.

If you found this useful, try rolling it to your staging AI service today. In under an hour, you’ll have real visibility—and a safer path to production.

Happy shipping.

Related posts