Posted on
• Artificial Intelligence

Artificial Intelligence Incident Investigation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Incident Investigation on Linux: A Bash‑First Playbook

When an AI system misbehaves, it’s rarely a single stack trace away from a fix. Models drift, prompts mutate, tokenizers change, seeds leak, and APIs silently upgrade. The longer you wait, the colder the trail gets.

This post gives you a Bash-first incident investigation playbook for AI systems running on Linux. You’ll learn how to freeze evidence, fingerprint models and data, reproduce issues in a clean container, minimize to a root cause, and package your findings. Everything is reproducible and automatable, so the next time “the model got weird,” you’ll know exactly what to do.

Why this matters

  • AI is probabilistic: Non-determinism and floating-point variance can mask or mimic real regressions.

  • Dependencies are deep: Tokenizers, BLAS backends, GPU drivers, and even locale settings can alter outputs.

  • Supply moves fast: Providers update APIs, models, and datasets—sometimes without a breaking change notice.

  • Effective response demands rigor: Immediate evidence capture and reproducible repro steps cut MTTR and avoid blame games.

The value: Less guesswork, faster root cause analysis, and a reusable workflow you can hand to teammates or vendors.

Prerequisites: Install investigation tools

We’ll use standard CLI tools only. Install them once on your investigation host or jump box.

Tools: jq, git, podman, python3 (with venv + pip), lsof, psmisc (for pstree), curl, auditd (optional, for OS auditing).

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jq git podman python3 python3-venv python3-pip lsof psmisc curl auditd

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y jq git podman python3 python3-pip python3-virtualenv lsof psmisc curl audit

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y jq git podman python3 python3-pip python3-virtualenv lsof psmisc curl audit

Notes:

  • On Fedora/openSUSE, the stdlib venv module is typically available with python3; python3-virtualenv adds the virtualenv tool if you prefer it.

  • auditd/audit is optional and may require configuration; use it only where policy allows.

The playbook (5 actionable steps)

1) First 15 minutes: Freeze and quarantine evidence

Create a case directory and snapshot the state before anything restarts or “hotfixes” happen.

set -euo pipefail

CASE="ai-incident-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$CASE"/{logs,env,artifacts,code,data}

# Host and kernel
uname -a | tee "$CASE/env/uname.txt"
cat /etc/os-release | tee "$CASE/env/os-release.txt"

# CPU/mem basics
lscpu | tee "$CASE/env/lscpu.txt" || true
free -h | tee "$CASE/env/memory.txt" || true

# GPU info if present
if command -v nvidia-smi >/dev/null 2>&1; then
  nvidia-smi -q | tee "$CASE/env/nvidia-smi.txt"
else
  echo "nvidia-smi not found" | tee "$CASE/env/nvidia-smi.txt"
fi

# Process snapshot and open sockets
ps auxf | tee "$CASE/logs/ps_auxf.txt"
pstree -ap | tee "$CASE/logs/pstree.txt" || true
ss -tulpn | tee "$CASE/logs/ports.txt" || true

# Environment variables (mask common secrets)
env | sort > "$CASE/env/env.raw.txt"
sed -E 's/([A-Z0-9_]*(API|TOKEN|SECRET|KEY|PASSWORD)[A-Z0-9_]*)=.*/\1=[REDACTED]/I' \
  "$CASE/env/env.raw.txt" > "$CASE/env/env.txt"

# Python environment if applicable
python3 -V 2>&1 | tee "$CASE/env/python_version.txt" || true
python3 -m pip freeze 2>/dev/null | tee "$CASE/env/pip_freeze.txt" || true

# Kernel warnings/errors (requires permissions)
dmesg --ctime --level=err,warn 2>/dev/null | tee "$CASE/logs/dmesg_warn.txt" || true

# Optional: if you know the AI service PID, capture open files
# PID=12345
# lsof -p "$PID" -Pn | tee "$CASE/logs/lsof_$PID.txt"

Tips:

  • Redact before you share. The sed filter above masks obvious secrets; expand it for your environment.

  • If you rely on API calls, enable request/response logging early (headers + bodies, with PII redaction).

2) Fingerprint models, data, and code

Incidents often come down to “it’s not the same bits.” Prove what you ran.

# Code provenance (run inside your repo)
git rev-parse HEAD | tee "$CASE/code/git_commit.txt" || true
git status --porcelain=v1 | tee "$CASE/code/git_status.txt" || true
git remote -v | tee "$CASE/code/git_remote.txt" || true
git submodule status | tee "$CASE/code/git_submodules.txt" || true
git diff > "$CASE/code/git_diff.patch" || true

# Model and tokenizer fingerprints (adjust patterns as needed)
find . -maxdepth 3 -type f \
  \( -name "*.pt" -o -name "*.bin" -o -name "*.safetensors" -o -name "tokenizer.*" -o -name "*.json" \) \
  -print0 | xargs -0 sha256sum | sort -k2 \
  | tee "$CASE/artifacts/model_tokenizer_sha256.txt"

# Dataset/version cues
# If sourcing from HTTP/S3, capture object metadata and ETag
# Example:
# URL="https://example.com/path/to/dataset.parquet"
# curl -sI "$URL" | tee "$CASE/data/dataset_headers.txt"

# Canonicalize and diff JSON configs for clarity
# Example: diff two configs you suspect changed
# jq -S . config.good.json > "$CASE/code/config.good.canon.json"
# jq -S . config.bad.json  > "$CASE/code/config.bad.canon.json"
# diff -u "$CASE/code/config.good.canon.json" "$CASE/code/config.bad.canon.json" | tee "$CASE/code/config.diff.txt"

Also capture runtime inputs that matter:

  • The exact prompt(s) or system messages used

  • Seed values, sampling parameters (temperature, top_p, top_k, max_tokens)

  • Pre/post-processing code versions

  • External API versions or model identifiers

Store them under $CASE/data/ or $CASE/code/ with timestamps.

3) Reproduce in a clean, pinned container

Reproduction beats speculation. Run your workload in a minimal container with the same inputs and pinned dependencies.

# Minimal Python repro using Podman and the official Python image
# Assumptions:
# - Your repo has requirements.txt (pinned if possible)
# - Your entrypoint is run.py (adjust as needed)

podman run --rm -it \
  -v "$PWD":/src:ro \
  -v "$PWD/$CASE":/case:Z \
  --workdir /src \
  docker.io/python:3.11-slim bash -lc '
    set -euo pipefail
    python -m venv /venv
    . /venv/bin/activate
    pip install --no-input --upgrade pip
    # Prefer hashed installs; fall back if you don’t have hashes yet
    (pip install --no-input --require-hashes -r requirements.txt) || pip install --no-input -r requirements.txt

    # Determinism knobs (use what’s relevant to your stack)
    export PYTHONHASHSEED=0
    export CUBLAS_WORKSPACE_CONFIG=:16:8
    export TF_DETERMINISTIC_OPS=1
    export TOKENIZERS_PARALLELISM=false
    export LC_ALL=C.UTF-8 TZ=UTC

    # Example: run with fixed seeds and capture I/O
    SEED=42
    python run.py --seed "$SEED" --config config.json \
      2>&1 | tee /case/logs/repro.log
  '

Notes:

  • For GPU repro, you’ll need a GPU-enabled container runtime configuration. That’s environment-specific—capture your driver/container details in the evidence and use a host with matching versions.

  • If your app fetches models at runtime, first vendor them:

    • Use pip download -r requirements.txt -d vendor/ and point pip to --no-index --find-links vendor/.
    • Download model files to /case/artifacts/ and mount read-only to the container.

4) Minimize and bisect to isolate the root cause

Turn the problem into the smallest reproducible example.

  • Git bisect the code:
git bisect start
git bisect bad HEAD
git bisect good <last-known-good-commit>

# test_case.sh must exit 0 on success and 1 on failure
# Example skeleton:
# #!/usr/bin/env bash
# set -euo pipefail
# python run.py --seed 42 --config config.json | tee /tmp/out.txt
# grep -q "EXPECTED_SUBSTRING" /tmp/out.txt

git bisect run ./test_case.sh
git bisect reset

  • Bisect configs/prompts/data:

    • Canonicalize JSON before diffing to avoid ordering noise:
    jq -S . configA.json > /tmp/A.json
jq -S . configB.json > /tmp/B.json
diff -u /tmp/A.json /tmp/B.json | tee "$CASE/code/config_diff.txt"
    • For prompts, strip timestamps/IDs and reduce:
    sed -E 's/[0-9a-f]{8,}/<ID>/g' prompt.txt | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}.*/<DATE>/g' \
  > "$CASE/data/prompt.redacted.txt"
    • Split long pipelines so you can test each stage’s output deterministically.

  • Check for non-determinism quickly:

export PYTHONHASHSEED=0 LC_ALL=C.UTF-8 TZ=UTC
for i in $(seq 1 3); do
  python run.py --seed 42 --config config.json | sha256sum
done
# If hashes differ, something is still non-deterministic.

5) Quick checks for common AI incident causes

  • Silent provider/model upgrades:

    • Log and pin model IDs and API versions.
    • Save response headers to confirm what you hit:
    curl -sD - https://api.example.com/v1/complete -o /dev/null \
  -H "Authorization: Bearer $API_TOKEN" \
  | tee "$CASE/logs/api_headers.txt"

  • Tokenizer or preprocessing drift:

    • Keep tokenizer files together with the model and hash them (see step 2).
    • Ensure the same normalization/lowercasing/langpack is used across runs.

  • Floating-point/backends and BLAS:

    • Document/lock MKL/OPENBLAS settings and CUDA/cuDNN versions.
    • Use deterministic flags where feasible; understand the perf trade-offs.

  • Locale/time/FS ordering:

    • Export LC_ALL=C.UTF-8 and TZ=UTC.
    • Avoid relying on readdir() order; sort inputs before processing.

  • Configuration shadowing:

    • Log the absolute path of the active config file and dump its effective values at startup.
    • Grep for duplicate keys or environment overrides.

Package it up for review

Bundle the case directory for teammates or vendors.

# Create a short, structured incident note
cat > "$CASE/incident.md" <<'EOF'
# Incident: <title>

- Case: <filled automatically by folder name>

- Detected at (UTC): <YYYY-MM-DDTHH:MM:SSZ>

- Reporter: <name>

- Impact: <user/system impact>

- Suspected scope: <service/model/version>

- Last known good: <commit/tag/date>

- Repro steps:
  1) <exact commands, or reference /logs/repro.log>

- Evidence:
  - env/: host, kernel, env, python
  - code/: git commit, diff, config diffs
  - artifacts/: model/tokenizer checksums
  - logs/: ps, pstree, ports, dmesg, repro.log

- Hypotheses:
  - <#1>
  - <#2>

- Next actions:
  - <#1>
  - <#2>
EOF

tar -czf "$CASE.tar.gz" "$CASE"
echo "Created bundle: $CASE.tar.gz"

Security and privacy:

  • Redact secrets and PII before sharing bundles externally.

  • Follow your org’s retention and incident handling policies.

Real-world example snapshots

  • “Same code, different answers”: Root cause was a tokenizer minor version bump. The fix was to hash and pin the tokenizer files alongside the model; checksums immediately revealed drift.

  • “Prod-only hallucinations”: Locale differed (C vs C.UTF-8) and a preprocessing regex behaved differently. Standardizing LC_ALL=C.UTF-8 made outputs reproducible and exposed a logic bug.

  • “Occasional nonsense”: Non-deterministic GPU ops. Exporting CUBLAS_WORKSPACE_CONFIG and setting seeds in code reduced variance; a tiny beam-search change amplified the residual jitter—fixed by capping temperature and enabling deterministic kernels.

Conclusion and next steps (CTA)

The best time to design your AI incident process is before you need it. Turn this playbook into a couple of shell scripts and add them to your repos:

  • Create scripts: ai-freeze.sh, ai-repro.sh, ai-bundle.sh

  • Pin and hash: requirements, models, tokenizers, configs

  • Log the right facts on every run: seeds, model IDs, API versions, locales

  • Practice a dry run on staging so the first real incident feels routine

If you want, reply with your stack details (framework, container runtime, GPU/CPU) and I’ll tailor these scripts to your environment.

Related posts