Posted on
• Artificial Intelligence

Artificial Intelligence Incident Postmortems

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Incident Postmortems for Linux Bash Users: A Practical, Reproducible Playbook

When an AI system goes sideways at 2 a.m.—flooding GPUs with runaway jobs, leaking sensitive data via a crafty prompt, or silently drifting into bad decisions—the first minutes decide how long you’ll be in the war room. This guide gives you a Bash-first, reproducible playbook for AI incident postmortems: how to freeze the scene, reconstruct the timeline, reproduce deterministically, and prevent it from happening again.

You’ll get actionable scripts, commands that work on any modern Linux distro, and install instructions for apt, dnf, and zypper wherever we use new tools.

Why AI Postmortems Are Different (and Necessary)

AI incidents don’t always look like classic outages:

  • Non-determinism: Two identical runs may produce slightly different outputs.

  • Data dependencies: “The model changed” may actually mean the features, embeddings, or prompts changed.

  • Version sprawl: Container image tags, model checkpoints, tokenizer versions, pipelines, and drivers are all moving parts.

  • Latent failures: Silent degradation (model drift, bias amplification) can be missed by traditional uptime checks.

  • Security: Prompt injection and tool-enabled LLMs can exfiltrate or act in unsafe ways if guardrails fail.

A good postmortem isolates facts quickly, reduces speculation, and yields precise, testable fixes. Bash is still your Swiss army knife for getting there fast.

Tools You’ll Use (Install Once, Use Forever)

We’ll rely on a handful of CLI tools that are widely available:

  • jq: JSON parsing and normalization

  • sysstat: historical performance (sar)

  • auditd: security/audit logs

  • sqlite3: lightweight queryable storage (handy for quick drift checks)

  • ripgrep (rg): fast source and log search

  • git: version and diff code/config

Install on your distro:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y jq sysstat auditd sqlite3 ripgrep git
sudo systemctl enable --now sysstat
sudo systemctl enable --now auditd

  • Fedora/RHEL/CentOS (dnf):

    sudo dnf install -y jq sysstat audit sqlite ripgrep git
sudo systemctl enable --now sysstat
sudo systemctl enable --now auditd

  • openSUSE/SLE (zypper):

    sudo zypper refresh
sudo zypper install -y jq sysstat audit sqlite3 ripgrep git
sudo systemctl enable --now sysstat
sudo systemctl enable --now auditd

Tip: If GPUs are involved, make sure nvidia-smi is available (from your NVIDIA driver install) so you can capture driver/runtime info in the freeze step.

Actionable Playbook: 5 Steps You Can Run Today

1) Freeze the Scene: Snapshot Artifacts, Environment, and Logs

Create a forensics bundle the moment you detect trouble. This prevents “it moved while I was looking at it.”

Save this as ai-incident-freeze.sh and run it on the affected host(s). It’s designed to be safe to run repeatedly and won’t restart services.

#!/usr/bin/env bash
set -euo pipefail

# Usage:
#   INCIDENT_TAG="chatbot-prod" SERVICES="myapp.service nginx.service" MODEL_DIR="/srv/models/current" ./ai-incident-freeze.sh
# Optional ENV:
#   LOG_SINCE="-2h"  LOG_UNTIL="now"  EXTRA_PATHS="/etc/myapp/config.yml /srv/app/.env"
INCIDENT_TAG="${INCIDENT_TAG:-ai-incident}"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
OUT="incident-$INCIDENT_TAG-$TS"
mkdir -p "$OUT"/{env,logs,artifacts,sys,containers,code}

# System info
uname -a > "$OUT/sys/uname.txt" || true
cat /etc/os-release > "$OUT/sys/os-release.txt" || true
ldd --version > "$OUT/sys/libc.txt" 2>&1 || true

# GPU info (if present)
if command -v nvidia-smi >/dev/null 2>&1; then
  nvidia-smi -q > "$OUT/sys/nvidia-smi.txt" || true
fi

# Python/Conda envs (if present)
if command -v python3 >/dev/null 2>&1; then
  python3 --version > "$OUT/env/python.txt" 2>&1 || true
  python3 -m pip freeze > "$OUT/env/pip-freeze.txt" 2>&1 || true
fi
if command -v conda >/dev/null 2>&1; then
  conda info -a > "$OUT/env/conda-info.txt" 2>&1 || true
  conda env export > "$OUT/env/conda-env.yml" 2>&1 || true
fi

# Environment variables (redact obvious secrets)
export -p | sed -E 's/(SECRET|TOKEN|KEY|PWD|PASS)=[^ ]+/\1=REDACTED/g' > "$OUT/env/export.txt" || true

# Running containers and image digests (if Docker present)
if command -v docker >/dev/null 2>&1; then
  docker ps --no-trunc > "$OUT/containers/ps.txt" || true
  docker ps --format '{{.ID}} {{.Image}}' | while read -r id img; do
    docker inspect "$id" > "$OUT/containers/${id}.inspect.json" || true
  done
fi

# Git state (if run inside a repo)
if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  git status --porcelain=v1 > "$OUT/code/git-status.txt" || true
  git rev-parse HEAD > "$OUT/code/git-commit.txt" || true
  git remote -v > "$OUT/code/git-remotes.txt" || true
  git diff --stat > "$OUT/code/git-diff-stat.txt" || true
fi

# Model/artifacts (optional dir)
if [[ -n "${MODEL_DIR:-}" && -d "$MODEL_DIR" ]]; then
  find "$MODEL_DIR" -type f -maxdepth 1 -print0 | xargs -0 sha256sum > "$OUT/artifacts/model-sha256.txt" || true
  tar -czf "$OUT/artifacts/model-snapshot.tgz" -C "$MODEL_DIR" . || true
fi

# Configs and extras (optional)
if [[ -n "${EXTRA_PATHS:-}" ]]; then
  while read -r p; do
    [[ -e "$p" ]] && cp -a --parents "$p" "$OUT/artifacts/" || true
  done <<< "$EXTRA_PATHS"
fi

# Logs
LOG_SINCE="${LOG_SINCE:--1h}"
LOG_UNTIL="${LOG_UNTIL:-now}"
if command -v journalctl >/dev/null 2>&1; then
  journalctl -S "$LOG_SINCE" -U "$LOG_UNTIL" -o short-iso > "$OUT/logs/journal-all.log" || true
  # Per-service (optional)
  for s in ${SERVICES:-}; do
    journalctl -u "$s" -S "$LOG_SINCE" -U "$LOG_UNTIL" -o json > "$OUT/logs/${s//.service/}.jsonlog" || true
  done
fi

# System stats (requires sysstat)
if command -v sar >/dev/null 2>&1; then
  sar -A > "$OUT/sys/sar.txt" 2>&1 || true
fi

# Audit logs (requires auditd)
if [[ -f /var/log/audit/audit.log ]]; then
  cp -a /var/log/audit/audit.log "$OUT/logs/audit.log" || true
fi

# Hash the bundle index for integrity
find "$OUT" -type f -print0 | xargs -0 sha256sum > "$OUT/sha256sums.txt"

echo "Frozen snapshot in: $OUT"

What this gives you:

  • System, GPU, Python/Conda, container, and git metadata

  • Model checksum and config captures

  • Journald, sysstat, and audit logs within a time window

Real-world win: A retail recommender suffered a 25% CTR drop. The freeze bundle proved the model hadn’t changed—but a feature join version had. Rolling back the feature pipeline (not the model) restored metrics.

2) Reconstruct the Timeline: Merge Logs and Surface Signals

Once you have artifacts, build a timeline from logs. Normalize JSON logs, filter errors, and correlate with resource pressure.

  • Merge and filter key events:
# Show high-severity app logs in the incident window
jq -r 'select(.PRIORITY|tonumber <= 3) | [.SYSLOG_IDENTIFIER, .MESSAGE] | @tsv' \
  incident-*/logs/myapp.jsonlog | sort | uniq -c | sort -nr | head
  • Extract spikes around the failure:
# Extract timestamps and messages that mention OOM or CUDA errors
jq -r '[.SYSLOG_TIMESTAMP, .MESSAGE] | @tsv' incident-*/logs/myapp.jsonlog \
  | rg -i 'out of memory|cuda|oom|killed' \
  | sed -E 's/\t/  |  /' | head -n 50
  • Cross-check with sar for CPU, memory, disk I/O:
# 5-min window around 2026-07-10T12:00Z
sar -r -s 11:55:00 -e 12:05:00    # memory
sar -u -s 11:55:00 -e 12:05:00    # CPU
sar -b -s 11:55:00 -e 12:05:00    # I/O

Real-world win: A vision service degraded intermittently. Timeline correlation showed a CPU steal-time spike aligned with a neighbor workload. Migrating nodes fixed it—no model changes required.

3) Reproduce Deterministically: Pin, Seed, and Compare

Non-determinism can hide root causes. Make repros boring on purpose.

  • Pin software and drivers (use digests, not tags)

  • Fix random seeds and choose deterministic ops

  • Disable async GPU launches for clearer stack traces

Environment flags you can export before running your repro container or script:

export PYTHONHASHSEED=0
export CUBLAS_WORKSPACE_CONFIG=:4096:8
export CUDA_LAUNCH_BLOCKING=1
export TF_DETERMINISTIC_OPS=1
export TORCH_USE_CUDA_DSA=1

In Python (PyTorch example), enforce determinism:

import torch, random, numpy as np
seed = 42
random.seed(seed); np.random.seed(seed); torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
torch.use_deterministic_algorithms(True)

Re-run the exact image by digest and model by checksum:

# Example: run exact container by sha256 digest
docker run --rm --gpus all \
  --pull=never \
  docker.io/yourorg/model-server@sha256:abc123... \
  --model-path /models/model.pt --seed 42

Real-world win: After a driver upgrade, a model produced rare NaNs. Deterministic repro showed it only occurred with a specific cuBLAS path. Pinning CUDA and cuDNN images fixed it and prevented future regressions.

4) Diff What Matters: Configs, Artifacts, and Data Slices

People often chase ghosts. Instead, normalize, hash, and diff.

  • Normalize and hash JSON configs:
jq -S . config-prod.json | sha256sum
jq -S . config-prev.json | sha256sum
diff -u <(jq -S . config-prev.json) <(jq -S . config-prod.json) || true
  • Prove the model is the same:
sha256sum incident-*/artifacts/model-snapshot.tgz
  • Verify container image digests didn’t drift:
jq -r '.[0].RepoDigests[]?' incident-*/containers/*.inspect.json | sort -u
  • Search code and prompts for unsafe changes (prompt injection, external tools):
rg -n --hidden -S "(prompt|system|tool|api|key|token|curl)" -g '!*.pyc' .
  • Quick “data drift” check using sqlite3 (if you log features/inputs):
# Example: Count frequency shift of a categorical feature before/after a timestamp
sqlite3 traffic.db "
SELECT val, 
  SUM(CASE WHEN ts < strftime('%s','2026-07-10T12:00:00Z') THEN 1 ELSE 0 END) AS before,
  SUM(CASE WHEN ts >= strftime('%s','2026-07-10T12:00:00Z') THEN 1 ELSE 0 END) AS after
FROM requests
GROUP BY val
ORDER BY (after - before) DESC
LIMIT 20;"

Real-world win: An LLM-based support bot suddenly leaked internal notes. Diffing the prompt template revealed a last-minute “debug tools” section had shipped without the output filter. Restoring the filter and adding a canary prompt blocked recurrence.

5) Prevent Recurrence: Automate Checks and Make It Culture

Codify what worked into your CI/CD and runbooks:

  • Pre-deploy smoke tests: deterministic inference on a golden set; assert on metrics and ban-phrases for LLMs.

  • Model and config signing: require sha256 matches in deployment.

  • Canary and rollback: route 1–5% traffic, watch error and drift signals, auto-rollback on thresholds.

  • Resource guards: max batch size, per-request GPU memory caps, and timeouts.

  • Security rails: prompt sanitization, output allow/deny lists, egress controls for tool-using LLMs.

Drop a postmortem template into your repo so teams don’t start from scratch:

cat > postmortems/TEMPLATE.md <<'EOF'
# AI Incident Postmortem: <title>
Date/Time (UTC):
Owner(s):
Services/Models Affected:
Severity/Impact:

## Summary
<2–3 sentences anyone can understand>

## Timeline (UTC)

- 12:03: First alert fired (what/where)

- 12:07: Traffic shifted to canary

- 12:15: Rollback initiated
...

## What Happened

- Triggering event

- Immediate symptoms

- Detection signals

## Root Cause

- Primary cause

- Contributing factors (infra, data, code, config, human)

## Evidence

- Freeze bundle path: <link or artifact ID>

- Logs, metrics, diffs (attach or link)

## Corrective Actions (now)

- [ ] Fix X by Y (owner, due date)

- [ ] Add alert Z (owner, due date)

## Preventive Actions (future)

- [ ] Pin driver/container digests in environment A

- [ ] Add canary check for ban-phrases

- [ ] Sign and verify model artifacts in CI

## What Went Well / What To Improve

- Good:

- Improve:

## Appendix

- Commands used:

- Notes:
EOF

Real-World Scenarios Mapped to This Playbook

  • Cost explosion from a runaway embedding job:

    • Freeze showed a cron overlap after daylight savings shift; timeline and sar confirmed CPU/GPU saturation window; fix: cron schedule + concurrency guard.

  • Model drift after a new data source:

    • Hashes showed the model unchanged; sqlite3 query surfaced distribution shift in user locales; fix: feature store validation + canary.

  • Prompt injection breach via uploaded files:

    • rg search found a commit enabling a new “file_tool” without domain allowlisting; fix: output filters + egress deny by default + regression tests.

Conclusion and Call to Action

AI incidents don’t need to be chaotic. With a Bash-first toolkit, you can capture facts quickly, reproduce deterministically, and converge on fixes that stick.

Your next steps: 1) Install the tooling (jq, sysstat, auditd, sqlite3, ripgrep, git) using apt/dnf/zypper above. 2) Drop ai-incident-freeze.sh onto every AI host and test it during a game day. 3) Add deterministic repro flags to your run scripts and CI. 4) Commit the postmortem template and make it the default incident artifact. 5) Wire canary tests and rollback into deployment.

If you found this useful, turn it into a runbook in your repo and share it with your on-call rotation. The best time to prepare for an AI incident is before the pager goes off.

Related posts