Posted on
• Artificial Intelligence

Artificial Intelligence API Gateway Monitoring

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence API Gateway Monitoring with Bash: A Practical Guide

When your AI features start 502’ing in the middle of a product demo, you don’t want to discover it from your users. AI workloads are bursty, vendor rate limits shift, and latency can swing wildly. Your API gateway is the choke point where all of this becomes observable—if you’re watching it.

This guide shows how to monitor an AI API gateway from Linux using plain Bash and standard tools. You’ll instrument logs, run synthetic probes, export lightweight metrics, and wire up fast alerts. No heavy dependencies required—and where we do use packages, we include apt, dnf, and zypper instructions.

Why monitor an AI API gateway?

  • Variability is the norm: upstream models and vector stores experience variable latency, cold starts, and capacity caps.

  • Cost is tied to tokens: a bug can silently multiply your token spend. Monitoring usage helps you catch runaways early.

  • Rate limits bite: 429s often surface only at peak traffic. Measure them continuously, not just after incidents.

  • You own the SLO: regardless of your vendors, your users judge your response times and error rates.

Core signals to watch:

  • Tail latency (p95/p99) through the gateway

  • Error and throttle rate (5xx and 429)

  • Timeouts and upstream failures

  • Token usage and cost per request

  • Cache hit ratio (if applicable)

Quick install: tools we’ll use

We’ll use curl, jq, bc, and (optionally) nginx for JSON access logs.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq bc nginx
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl jq bc nginx
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq bc nginx

Note: You can replace nginx with your gateway (Kong, Envoy, Tyk, etc.). We’ll show nginx examples because it’s widely available.

1) Define SLOs and thresholds that matter

Pick objective, automatable targets:

  • Availability: 99.9% successful (2xx) calls

  • Latency: p95 under 2.0s, hard timeout at 15s

  • Throttle: 429 under 0.5% of requests

  • Upstream errors: 5xx under 0.1%

  • Cost guardrail: tokens per minute or per day under thresholds

Put your numbers in environment variables so your scripts and alerts use the same truth:

# /etc/ai-gw/env
GW_URL="https://api.example.com/v1/chat/completions"
API_KEY="REDACTED"
MODEL="gpt-4o-mini"
HARD_TIMEOUT="15"
P95_LATENCY_BUDGET="2.0"
MAX_429_RATE="0.5"

2) Instrument the gateway: JSON logs with upstream timings

If you use nginx as an API gateway or edge, enable structured logs and upstream timings. Add this to nginx.conf or a conf.d file:

log_format ai_json escape=json
  '{ "time":"$time_iso8601",'
  '  "remote":"$remote_addr",'
  '  "method":"$request_method",'
  '  "path":"$request_uri",'
  '  "status":$status,'
  '  "bytes":$body_bytes_sent,'
  '  "req_time":$request_time,'
  '  "upstream_time":"$upstream_response_time",'
  '  "upstream":"$upstream_addr",'
  '  "corr":"$http_x_request_id",'
  '  "ai_model":"$http_x_ai_model" }';

access_log /var/log/nginx/ai_gateway.json ai_json;

# For proxying to AI vendor or internal model endpoint
proxy_read_timeout 30s;
proxy_connect_timeout 5s;

Reload nginx:

sudo nginx -t && sudo systemctl reload nginx

On busy systems, ensure logrotate handles /var/log/nginx/ai_gateway.json.

Example: compute fast, on-demand stats from the last 5 minutes:

jq -c 'select(.path|test("/v1/chat/completions"))' /var/log/nginx/ai_gateway.json \
| tail -n 5000 \
| jq -r '[.status, (.req_time|tonumber), (.upstream_time|split(",")[-1]|tonumber? // 0)] | @tsv' \
| awk '
  {count++; s=$1; rt=$2; if (s ~ /^5/ || s ~ /^4(?!29)/) e++; if (s==429) r++; lat[NR]=rt}
  END {
    asort(lat);
    p95=lat[int(0.95*count)]; if (p95=="") p95=0;
    er= (count>0 ? (100*(e/count)) : 0);
    rr= (count>0 ? (100*(r/count)) : 0);
    printf("requests=%d errors=%.2f%% 429s=%.2f%% p95=%.3fs\n", count, er, rr, p95);
  }'

This gives a quick health pulse even without a metrics stack.

3) Synthetic probes with Bash (latency, status, tokens)

Run a minimal request through the gateway every minute to detect regressions, even when traffic is low.

Create a probe script:

#!/usr/bin/env bash
# /usr/local/bin/ai-gw-probe.sh
set -euo pipefail

: "${GW_URL:?set GW_URL}"
: "${API_KEY:?set API_KEY}"
MODEL="${MODEL:-gpt-4o-mini}"
HARD_TIMEOUT="${HARD_TIMEOUT:-15}"
PROM_FILE="${PROM_FILE:-/var/tmp/ai_gw_probe.prom}"
TMP_OUT="$(mktemp)"

req_body=$(cat <<JSON
{"model":"$MODEL","messages":[{"role":"user","content":"ping"}],"max_tokens":16}
JSON
)

# Perform request and capture status + total time
read -r http_code time_total <<<"$(
  curl -sS -o "$TMP_OUT" \
       -w "%{http_code} %{time_total}\n" \
       -H "Authorization: Bearer $API_KEY" \
       -H "Content-Type: application/json" \
       --connect-timeout 5 --max-time "$HARD_TIMEOUT" \
       -d "$req_body" "$GW_URL" || echo "000 $HARD_TIMEOUT"
)"

# Parse tokens if the API returns usage in the body
tokens=$(jq -r '.usage.total_tokens // 0' "$TMP_OUT" 2>/dev/null || echo 0)

# Normalize float output
latency=$(awk -v t="$time_total" 'BEGIN{printf "%.3f", (t+0)}')

# Prometheus-style metrics (can be tailed or scraped via textfile collector)
ts=$(date +%s)
tmp_prom="$(mktemp)"
{
  echo "# HELP ai_gw_probe_up 1 if probe succeeded (HTTP 2xx), else 0"
  echo "# TYPE ai_gw_probe_up gauge"
  if [[ "$http_code" =~ ^2 ]]; then echo "ai_gw_probe_up 1 $ts"; else echo "ai_gw_probe_up 0 $ts"; fi

  echo "# HELP ai_gw_probe_latency_seconds End-to-end latency"
  echo "# TYPE ai_gw_probe_latency_seconds gauge"
  echo "ai_gw_probe_latency_seconds $latency $ts"

  echo "# HELP ai_gw_probe_http_status_count Count by last observed status"
  echo "# TYPE ai_gw_probe_http_status_count counter"
  echo "ai_gw_probe_http_status_count{code=\"$http_code\"} 1"

  echo "# HELP ai_gw_probe_tokens_total Last response token usage"
  echo "# TYPE ai_gw_probe_tokens_total gauge"
  echo "ai_gw_probe_tokens_total $tokens $ts"
} > "$tmp_prom"
mv "$tmp_prom" "$PROM_FILE"

# Optional lightweight alerting to a webhook if budgets are violated
P95_BUDGET="${P95_LATENCY_BUDGET:-2.0}"
MAX_LAT_EXCEEDED=$(awk -v a="$latency" -v b="$P95_BUDGET" 'BEGIN{print (a>b)?"1":"0"}')
if [[ "$MAX_LAT_EXCEEDED" == "1" || ! "$http_code" =~ ^2 ]]; then
  if [[ -n "${ALERT_WEBHOOK_URL:-}" ]]; then
    curl -sS -X POST -H 'Content-Type: application/json' \
      -d "{\"text\":\"AI GW probe alert: status=$http_code latency=${latency}s tokens=$tokens url=$GW_URL\"}" \
      "$ALERT_WEBHOOK_URL" >/dev/null || true
  fi
fi

rm -f "$TMP_OUT"

Permissions and environment:

sudo install -m 0755 /usr/local/bin/ai-gw-probe.sh /usr/local/bin/ai-gw-probe.sh
sudo mkdir -p /etc/ai-gw
sudo tee /etc/ai-gw/env >/dev/null <<'EOF'
GW_URL="https://api.example.com/v1/chat/completions"
API_KEY="REDACTED"
MODEL="gpt-4o-mini"
HARD_TIMEOUT="15"
P95_LATENCY_BUDGET="2.0"
# Optional: send minimal alerts to a chat/webhook
# ALERT_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"
EOF

Automate via systemd timer (no cron dependency):

# /etc/systemd/system/ai-gw-probe.service
[Unit]
Description=AI Gateway Synthetic Probe

[Service]
Type=oneshot
EnvironmentFile=/etc/ai-gw/env
ExecStart=/usr/local/bin/ai-gw-probe.sh

# /etc/systemd/system/ai-gw-probe.timer
[Unit]
Description=Run AI Gateway Synthetic Probe every minute

[Timer]
OnBootSec=15s
OnUnitActiveSec=60s
AccuracySec=5s
Unit=ai-gw-probe.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-gw-probe.timer
systemctl status ai-gw-probe.timer

You’ll now get a refreshed metrics file at /var/tmp/ai_gw_probe.prom with up/latency/status/tokens every minute.

Tip: If you run Prometheus node_exporter with the textfile collector, point it at /var/tmp to scrape these metrics. If you don’t, you can still tail this file or ship it with your log forwarder.

4) Turn logs into live counters (jq + tail)

Transform gateway logs into rolling error and throttle rates. This can run as a background service or be integrated with your existing log pipeline.

Example: monitor 429 and 5xx share over a moving window:

tail -n0 -F /var/log/nginx/ai_gateway.json \
| jq -cr 'select(.path|test("^/v1/(chat/completions|responses)")) | {status: .status, rt: (.req_time|tonumber)}' \
| awk -v window=300 '
  function now(){ return systime() }
  BEGIN { FS="\t" }
  {
    t=now(); s=$0;
    split(s, pair, "\"status\":"); gsub(/[^0-9]/, "", pair[2]); code=pair[2]+0;
    # push into arrays keyed by timestamp second
    bucket=t%window; total[bucket]++; if (code>=500) s5xx[bucket]++; if (code==429) s429[bucket]++; lat[bucket]=lat[bucket]" "gensub(/.*"rt":([0-9.]+).*/,"\\1","g",s)
    # aggregate across window
    sum=err=thr=0; delete samples
    for (i=0;i<window;i++){ idx=(t-i)%window; sum+=total[idx]; err+=s5xx[idx]; thr+=s429[idx]; n=split(lat[idx],arr," "); for (j in arr) if (arr[j]!="") samples[++k]=arr[j]+0 }
    # compute p95
    if (k>0) { n=k; asort(samples); p95=samples[int(0.95*n)]; } else { p95=0 }
    if (sum>0) printf("win=%ds count=%d 5xx=%.2f%% 429=%.2f%% p95=%.3fs       \r", window, sum, 100*err/sum, 100*thr/sum, p95);
    fflush();
  }'

This gives a low-overhead terminal dashboard with a sliding 5-minute view.

5) Real-world playbook: catch issues fast

  • Throttle spikes (429): Often correlate with deployments or sudden traffic. The tail script above will show 429% climbing; your probe will start failing too. Short-term mitigation: exponential backoff and request smoothing. Long-term: rate-limit awareness and retries with jitter in clients.

  • Latency regressions: If p95 jumps above budget but 5xx/429 are normal, suspect upstream cold starts or model switches. Consider gateway-side caching for embeddings and static prompts.

  • Cost runaways: If your responses expose usage fields (prompt_tokens, completion_tokens), sum them per minute and daily. Trigger an alert when above budget so you can disable features or adjust prompts promptly.

Budget guardrail example (append to your probe script or run separately):

TOK_FILE="/var/tmp/ai_tokens_day.$(date +%F)"
echo "$(date +%s) $tokens" >> "$TOK_FILE"
daily_sum=$(awk '{s+=$2} END{print s+0}' "$TOK_FILE")
DAILY_TOKEN_BUDGET="${DAILY_TOKEN_BUDGET:-2000000}" # 2M tokens/day
if [ "$daily_sum" -gt "$DAILY_TOKEN_BUDGET" ] && [ -n "${ALERT_WEBHOOK_URL:-}" ]; then
  curl -sS -X POST -H 'Content-Type: application/json' \
    -d "{\"text\":\"AI token budget exceeded: $daily_sum / $DAILY_TOKEN_BUDGET\"}" \
    "$ALERT_WEBHOOK_URL" >/dev/null || true
fi

Rotate the token file at midnight via a simple systemd timer or a daily job.

Optional: scrape your probe metrics with Prometheus/Grafana

If you already run Prometheus, enable node_exporter’s textfile collector and point it at /var/tmp to pick up ai_gw_probe.prom. Build Grafana panels for:

  • Probe up/latency over time

  • 429 and 5xx shares from logs (ship counts via your log pipeline)

  • Tokens per minute and daily budget

If you don’t have a metrics stack yet, start with the Bash workflows above. They’re easy to containerize or migrate later.

Conclusion and next steps

Your AI gateway is where reliability, latency, and cost become visible. With a few Bash scripts and ubiquitous Linux tools, you can:

  • Define and enforce SLOs

  • Instrument gateway logs for structured, queryable data

  • Run synthetic checks to detect issues before users do

  • Alert on latency, errors, and token budgets

Next steps: 1) Install curl/jq/bc and enable JSON logs on your gateway. 2) Drop in the ai-gw-probe.sh script and systemd timer. 3) Set thresholds in /etc/ai-gw/env and wire a webhook for quick alerts. 4) Iterate: add per-route, per-model labels; include cache hit ratios; integrate with your metrics stack when ready.

Questions or want a ready-made dashboard? Start with the scripts here, then layer in Prometheus and Grafana for long-term trends. Your future incident retrospectives will thank you.

Related posts