Posted on
• Artificial Intelligence

Artificial Intelligence Uptime Reporting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Uptime Reporting with Bash: From Pings to Production-Grade Proof

If your AI service is down, it doesn’t matter how accurate your model is—your users will churn. The catch: AI services are uniquely fragile. They sit on GPUs, containers, and networked model backends that can be flaky under load, and “port is open” is not the same as “inference works.” In this guide, you’ll build a simple, Linux-native uptime reporting pipeline in Bash that measures availability, latency, and GPU context—then turns that into daily, auditable reports you can ship to your team.

You’ll get:

  • A minimal, dependency-light checker that exercises your AI endpoint the way your users do

  • A one-minute systemd timer or cron job to collect data

  • A reporting script to compute 24h/7d uptime and latency percentiles

  • Optional Prometheus textfile metrics output (works with node_exporter if you already run it)

Why AI uptime deserves special treatment

  • Health != Inference: A 200 OK on “/health” might not mean the model can generate a token. You need an application-level probe.

  • GPUs are finite: Memory pressure and utilization spikes can cause timeouts and OOMs without crashing the process.

  • Cold starts and scaling: Autoscaling, model swaps, and container restarts introduce transient errors that pings miss.

  • SLAs matter: If you can’t calculate “what percent of requests succeeded” across a defined window, you don’t have an SLO—you have a hope.

What you’ll build

  • A Bash script that hits your AI service endpoint, records success/failure, HTTP status, latency, and (if available) GPU utilization and memory.

  • Log storage in CSV under /var/log/ai-uptime/.

  • Optional Prometheus textfile metrics output for easy dashboards.

  • A systemd timer (or cron) to run it every minute.

  • A report generator to compute uptime and latency stats over a window.

Install prerequisites

We’ll use curl, jq, and bc. Install with your package manager:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y curl jq bc cron

  • Fedora/RHEL/CentOS Stream (dnf):

    sudo dnf install -y curl jq bc cronie

  • openSUSE (zypper):

    sudo zypper refresh
sudo zypper install -y curl jq bc cron

Notes:

  • cron/cronie is only needed if you prefer cron over systemd timers.

  • If you run node_exporter with the textfile collector, ensure its directory (often /var/lib/node_exporter/textfile_collector) exists and is writable by your uptime user.

Create a dedicated user and directories

sudo useradd --system --no-create-home --shell /usr/sbin/nologin ai-uptime
sudo install -d -o ai-uptime -g ai-uptime -m 750 /opt/ai-uptime
sudo install -d -o ai-uptime -g ai-uptime -m 750 /var/log/ai-uptime

If you plan to output Prometheus textfiles and already run node_exporter with the textfile collector, also:

# Example path; adjust to your environment
sudo install -d -o ai-uptime -g ai-uptime -m 750 /var/lib/node_exporter/textfile_collector

Step 1: The AI health check script

Save this as /opt/ai-uptime/check_ai_health.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

# Config via env or defaults
SERVICE_NAME="${SERVICE_NAME:-model-a}"
URL="${URL:-http://127.0.0.1:8000/health}"
METHOD="${METHOD:-GET}"                       # GET or POST
TIMEOUT="${TIMEOUT:-8}"                       # seconds
HEADERS="${HEADERS:-Content-Type: application/json}"
PAYLOAD="${PAYLOAD:-}"                        # e.g. {"model":"local","prompt":"ping","max_tokens":1}

LOG_DIR="${LOG_DIR:-/var/log/ai-uptime}"
CSV="${CSV:-$LOG_DIR/ai-uptime.csv}"

# Optional Prometheus textfile output (if this dir exists)
PROM_DIR="${PROM_DIR:-/var/lib/node_exporter/textfile_collector}"

# Optional: also check a systemd unit or docker container
SYSTEMD_SERVICE="${SYSTEMD_SERVICE:-}"        # e.g., your-ai.service
DOCKER_CONTAINER="${DOCKER_CONTAINER:-}"      # e.g., ai-server-1

TS="$(date +%s)"
TMP="$(mktemp)"
cleanup() { rm -f "$TMP"; }
trap cleanup EXIT

# Run request and capture status code + latency
HTTP_CODE=""
LATENCY=""

CURL_ARGS=( -sS -m "$TIMEOUT" -o "$TMP" -w '%{http_code} %{time_total}' -X "$METHOD" )
if [[ -n "$HEADERS" ]]; then
  IFS=$'\n' read -r -d '' -a HDR_ARR < <(printf '%s\0' $HEADERS)
  for h in "${HDR_ARR[@]}"; do CURL_ARGS+=( -H "$h" ); done
fi
if [[ -n "$PAYLOAD" ]]; then
  CURL_ARGS+=( --data "$PAYLOAD" )
fi
CURL_ARGS+=( "$URL" )

if OUT="$("curl" "${CURL_ARGS[@]}" 2>/dev/null || true)"; then
  HTTP_CODE="$(awk '{print $1}' <<<"$OUT")"
  LATENCY="$(awk '{print $2}' <<<"$OUT")"
else
  HTTP_CODE="000"
  LATENCY=""
fi

# Decide "UP" based on HTTP 2xx and optional application-level signal
UP=0
if [[ "$HTTP_CODE" =~ ^2[0-9][0-9]$ ]]; then
  UP=1
  # If JSON body includes "error" or explicit failure, mark down
  if command -v jq >/dev/null 2>&1; then
    if jq -e '(.error? // .status? // .ok?) as $s | (type=="object")' "$TMP" >/dev/null 2>&1; then
      # Prefer explicit fields if present
      if jq -e 'has("error") and (.error != null and .error != false)' "$TMP" >/dev/null 2>&1; then
        UP=0
      elif jq -e 'has("status") and (.status|tostring|ascii_downcase) != "ok"' "$TMP" >/dev/null 2>&1; then
        UP=0
      elif jq -e 'has("ok") and (.ok|tostring|ascii_downcase)=="false"' "$TMP" >/dev/null 2>&1; then
        UP=0
      fi
    fi
  fi
fi

# Optional: capture service/container state
SVC_ACTIVE=""
if [[ -n "$SYSTEMD_SERVICE" ]]; then
  if systemctl is-active --quiet "$SYSTEMD_SERVICE"; then SVC_ACTIVE="active"; else SVC_ACTIVE="inactive"; fi
fi

CTR_HEALTH=""
if [[ -n "$DOCKER_CONTAINER" ]] && command -v docker >/dev/null 2>&1; then
  CTR_HEALTH="$(docker inspect --format='{{.State.Health.Status}}' "$DOCKER_CONTAINER" 2>/dev/null || true)"
fi

# Optional: capture GPU stats if nvidia-smi exists
GPU_UTIL=""
GPU_MEM=""
if command -v nvidia-smi >/dev/null 2>&1; then
  IFS=',' read -r GPU_UTIL GPU_MEM < <(nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader,nounits 2>/dev/null | head -n1 | tr -d ' %' | tr ' ' ',')
fi

# Prepare CSV
mkdir -p "$LOG_DIR"
if [[ ! -f "$CSV" ]]; then
  echo "timestamp,service,up,http_code,latency_seconds,gpu_util_percent,gpu_mem_mb,systemd_active,docker_health" > "$CSV"
fi

LATENCY="${LATENCY:-NaN}"
HTTP_CODE="${HTTP_CODE:-000}"
echo "$TS,$SERVICE_NAME,$UP,$HTTP_CODE,$LATENCY,${GPU_UTIL:-},${GPU_MEM:-},${SVC_ACTIVE:-},${CTR_HEALTH:-}" >> "$CSV"

# Optional: Prometheus textfile
if [[ -d "$PROM_DIR" ]]; then
  SAFE_SERVICE="$(echo "$SERVICE_NAME" | tr -c 'a-zA-Z0-9:_-' '_')"
  OUTFILE="$PROM_DIR/ai_uptime_${SAFE_SERVICE}.prom"
  {
    echo "ai_service_up{service=\"$SERVICE_NAME\"} $UP"
    echo "ai_service_latency_seconds{service=\"$SERVICE_NAME\"} ${LATENCY/NaN/0}"
    echo "ai_service_http_code{service=\"$SERVICE_NAME\",code=\"$HTTP_CODE\"} 1"
    if [[ -n "$GPU_UTIL" ]]; then
      echo "ai_service_gpu_utilization_percent{service=\"$SERVICE_NAME\"} $GPU_UTIL"
    fi
    if [[ -n "$GPU_MEM" ]]; then
      echo "ai_service_gpu_memory_used_megabytes{service=\"$SERVICE_NAME\"} $GPU_MEM"
    fi
  } > "$OUTFILE.$$"
  mv "$OUTFILE.$$" "$OUTFILE"
fi

# Log a human message
logger -t ai-uptime "service=$SERVICE_NAME up=$UP code=$HTTP_CODE latency=$LATENCY gpu_util=${GPU_UTIL:-} gpu_mem=${GPU_MEM:-} url=$URL"

Make it executable and owned by our user:

sudo chown ai-uptime:ai-uptime /opt/ai-uptime/check_ai_health.sh
sudo chmod 750 /opt/ai-uptime/check_ai_health.sh

Quick test:

sudo -u ai-uptime SERVICE_NAME=model-a URL=http://127.0.0.1:8000/health /opt/ai-uptime/check_ai_health.sh
tail -n1 /var/log/ai-uptime/ai-uptime.csv

Tip: To probe real inference, set:

METHOD=POST
HEADERS=$'Content-Type: application/json\nAuthorization: Bearer YOUR_TOKEN'   # if needed
PAYLOAD='{"model":"local","prompt":"ping","max_tokens":1}'

Step 2: Schedule the checker

Systemd timer (recommended for minute-precision and logging):

Create /etc/systemd/system/ai-uptime.service

[Unit]
Description=AI Uptime Check

[Service]
Type=oneshot
User=ai-uptime
Group=ai-uptime
Environment=SERVICE_NAME=model-a
Environment=URL=http://127.0.0.1:8000/health
# Example for inference POST:
# Environment=METHOD=POST
# Environment=HEADERS=Content-Type: application/json
# Environment=PAYLOAD={"model":"local","prompt":"ping","max_tokens":1}
ExecStart=/opt/ai-uptime/check_ai_health.sh

Create /etc/systemd/system/ai-uptime.timer

[Unit]
Description=Run AI Uptime Check every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=1s
Unit=ai-uptime.service

[Install]
WantedBy=timers.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-uptime.timer
sudo systemctl list-timers | grep ai-uptime

Prefer cron? Add this line (runs every minute):

# Debian/Ubuntu: ensure "cron" service is running; Fedora/RHEL: "crond" from cronie
* * * * * SERVICE_NAME=model-a URL=http://127.0.0.1:8000/health /opt/ai-uptime/check_ai_health.sh >/dev/null 2>&1

Service per model: Duplicate the .service file with different Environment= blocks or create multiple unit/timer pairs (e.g., ai-uptime@model-a.service, ai-uptime@model-b.service) with different URLs.

Step 3: Reporting script (24h/7d uptime and latency)

Save as /opt/ai-uptime/report_ai_uptime.sh.

#!/usr/bin/env bash
set -euo pipefail

CSV="${CSV:-/var/log/ai-uptime/ai-uptime.csv}"
SERVICE="${SERVICE:-}"          # filter by service name; empty = all
WINDOW="${WINDOW:-86400}"       # seconds; 86400=24h, 604800=7d

now="$(date +%s)"
start="$(( now - WINDOW ))"

if [[ ! -f "$CSV" ]]; then
  echo "No data at $CSV" >&2
  exit 1
fi

# Filter rows in window and (optionally) service
awk -F, -v s="$SERVICE" -v start="$start" 'NR>1 && $1>=start && (s=="" || $2==s)' "$CSV" > /tmp/ai_report.$$ || true
rows=$(wc -l < /tmp/ai_report.$$ | tr -d ' ')
if (( rows == 0 )); then
  echo "No samples in window."
  rm -f /tmp/ai_report.$$
  exit 0
fi

# Compute uptime
up=$(awk -F, '{sum+=$3} END{print sum+0}' /tmp/ai_report.$$)
pct=$(awk -v u="$up" -v r="$rows" 'BEGIN{ if (r==0) {print 0} else {printf "%.4f", (u/r*100)} }')

# Latency stats
lat_file="/tmp/ai_report_lat.$$"
awk -F, '$5 ~ /^[0-9.]+$/ {print $5}' /tmp/ai_report.$$ | sort -n > "$lat_file"
count=$(wc -l < "$lat_file" | tr -d ' ')
p50_idx=$(( (count*50 + 99)/100 ))
p95_idx=$(( (count*95 + 99)/100 ))
avg=$(awk '{sum+=$1} END{ if (NR==0) print 0; else printf "%.6f", sum/NR }' "$lat_file")
p50=$(awk -v i="$p50_idx" 'NR==i{print $1}' "$lat_file")
p95=$(awk -v i="$p95_idx" 'NR==i{print $1}' "$lat_file")

# Longest outage (consecutive down samples)
longest=0
streak=0
while IFS=, read -r ts svc upflag _; do
  if [[ -n "$SERVICE" && "$svc" != "$SERVICE" ]]; then continue; fi
  if [[ "$upflag" == "0" ]]; then
    ((streak++))
    ((streak > longest)) && longest=$streak
  else
    streak=0
  fi
done < <(sort -t, -k1,1n /tmp/ai_report.$$)

# Print report
echo "AI Uptime Report"
echo "Window: last $WINDOW seconds (~$((WINDOW/3600))h)"
if [[ -n "$SERVICE" ]]; then echo "Service: $SERVICE"; fi
echo "Samples: $rows"
echo "Uptime: $pct %  (Up=$up, Down=$((rows-up)))"
echo "Latency: avg=${avg}s, p50=${p50:-N/A}s, p95=${p95:-N/A}s"
if (( longest > 0 )); then
  echo "Longest consecutive outage: $longest samples (~$((longest)) min if 1-min schedule)"
else
  echo "No outages detected."
fi

rm -f /tmp/ai_report.$$ "$lat_file"

Make it executable:

sudo chown ai-uptime:ai-uptime /opt/ai-uptime/report_ai_uptime.sh
sudo chmod 750 /opt/ai-uptime/report_ai_uptime.sh

Example usage:

# 24h report for model-a
sudo -u ai-uptime SERVICE=model-a /opt/ai-uptime/report_ai_uptime.sh

# 7d all-services rollup
sudo -u ai-uptime WINDOW=604800 /opt/ai-uptime/report_ai_uptime.sh

Optional daily report via systemd:

  • Create /etc/systemd/system/ai-uptime-report.service

    [Unit]
Description=AI Uptime Daily Report

[Service]
Type=oneshot
User=ai-uptime
Group=ai-uptime
Environment=WINDOW=86400
ExecStart=/opt/ai-uptime/report_ai_uptime.sh

  • Create /etc/systemd/system/ai-uptime-report.timer

    [Unit]
Description=Run AI Uptime Daily Report at 08:00

[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true

[Install]
WantedBy=timers.target

  • Enable it:

    sudo systemctl daemon-reload
sudo systemctl enable --now ai-uptime-report.timer

Step 4: Real-world target examples

  • Simple health endpoint:

    URL=http://127.0.0.1:8000/health METHOD=GET

    If the body returns {"status":"ok"}, the script will treat it as up.

  • OpenAI-compatible inference POST (local or remote):

    URL=http://127.0.0.1:8000/v1/completions
METHOD=POST
HEADERS=$'Content-Type: application/json\nAuthorization: Bearer YOUR_TOKEN'
PAYLOAD='{"model":"local","prompt":"ping","max_tokens":1}'

  • Containerized deployment checks (optional):

    SYSTEMD_SERVICE=your-ai.service
DOCKER_CONTAINER=ai-server-1

    These add context but don’t affect UP unless you act on them downstream.

  • GPU-enabled servers: The script automatically captures GPU utilization and memory if nvidia-smi exists. No config needed.

Tip: Monitor multiple services by creating multiple systemd service/timer pairs with different Environment= blocks, or call the script in a loop from a wrapper.

Step 5: Alerts and integrations (minimal but effective)

Start simple:

  • Page on consecutive failures:

    # 5 consecutive downs within a minute cadence
tail -n5 /var/log/ai-uptime/ai-uptime.csv | awk -F, '$3==0' | wc -l

    If result is 5, send a webhook:

    curl -X POST -H 'Content-Type: application/json' \
-d '{"text":"ALERT: model-a down 5x in a row"}' \
https://hooks.slack.com/services/XXX/YYY/ZZZ

  • Grafana-friendly metrics: If you already run node_exporter with a textfile collector, the script writes ai_service_up and latency metrics into .prom files. Point Prometheus at node_exporter and build dashboards or alerts.

3–5 actionable practices to keep you honest

1) Probe what users do, not just ports
Always run a tiny inference (e.g., prompt “ping”, max_tokens=1). A 200 on /health can hide broken model backends.

2) Track SLI inputs you can explain
Record up/down, HTTP code, and latency at 1-minute resolution. Add GPU memory/utilization to correlate slowdowns with load.

3) Separate network and app errors
000 status (curl failure) vs 5xx (server error) vs 2xx with an “error” field tell very different stories when you negotiate SLAs.

4) Automate a daily SLA report
A rolling 24h and 7d report keeps your team honest. Post it in a Slack channel every morning before stand-up.

5) Keep dependencies boring
Bash + curl + jq + systemd is universally available on Linux servers and easy to audit.

Troubleshooting tips

  • Empty CSV: Your timer/cron may not be running. Check:

    systemctl status ai-uptime.timer
journalctl -u ai-uptime.service --since "10 min ago"

  • Non-JSON endpoints: That’s fine. The script only uses jq if present and JSON-like; otherwise it falls back to HTTP status.

  • SSL/TLS issues: Add curl options in HEADERS or extend CURL_ARGS (e.g., --cacert, --resolve).

  • High-latency spikes only: Check GPU mem/util and container health columns in the CSV to correlate.

Conclusion and next steps

You now have a production-ready, Bash-first AI uptime reporter that does more than ping—it proves that inference works, quantifies reliability, and gives you metrics to improve. Next steps:

  • Put this behind a Grafana dashboard if you run Prometheus/node_exporter.

  • Add a webhook alert on streaks of failures or p95 latency breaches.

  • Extend the checker to hit multiple endpoints per model (health, inference, embeddings) and tag them separately.

If you found this useful, instrument one service today, share the 24h report with your team, and iterate. Reliability is a feature—ship it.

Related posts