AI Incident Investigations on Linux: a Bash-First Playbook

It’s 02:07. Your AI service just pushed a toxic response, leaked a customer phone number, or started timing out after a model rollout. Do you restart and hope for the best—or do you preserve evidence, reconstruct the chain of events, and fix the real root cause?

On Linux, Bash is your incident response Swiss army knife. This guide shows you how to investigate AI incidents quickly and reproducibly with standard CLI tools, reproducible workflows, and a drop-in triage script.

Why AI incident investigations are different (and valid to treat as their own class)

The “state” is bigger than code. Incidents often depend on the exact model weights, tokenizer, dataset slice, retrieval index, and even the prompt template. You need all of them to reproduce.
Non-determinism and drift are real. Seeds, parallelism, kernel/GPU drivers, or a remote model gateway version can change outputs “mysteriously.”
Boundaries matter. Most AI stacks straddle multiple systems: networked tools, RAG stores, containers, GPUs, systemd units. Logs in one place are not enough.
Safety and privacy. You must capture enough context to debug without accidentally exfiltrating secrets or PII in artifacts.

Treating AI incidents with a disciplined, Bash-first approach helps you preserve context, reduce MTTR, and make fixes stick.

Quick install: investigation toolchain

The following packages cover log collection, protocol capture, tracing, and basic system profiling. Install with your distro’s package manager.

Debian/Ubuntu (apt):

sudo apt-get update && sudo apt-get install -y \
  git jq yq python3 python3-venv python3-pip \
  tcpdump tshark strace ltrace sysstat ripgrep \
  auditd bpftrace

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y \
  git jq yq python3 python3-virtualenv python3-pip \
  tcpdump wireshark-cli strace ltrace sysstat ripgrep \
  audit bpftrace

openSUSE/SLES (zypper):

sudo zypper refresh && sudo zypper install -y \
  git jq yq python3 python3-virtualenv python3-pip \
  tcpdump wireshark strace ltrace sysstat ripgrep \
  audit bpftrace

Optional setup:

Enable persistent journal logs:

sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

Start auditd (Linux Audit):

# Ubuntu/Debian:
sudo systemctl enable --now auditd
# Fedora/RHEL/openSUSE:
sudo systemctl enable --now auditd.service

Allow non-root packet capture (optional; prefer root in incidents):

sudo setcap 'cap_net_raw,cap_net_admin=eip' /usr/sbin/tcpdump || true

Core playbook: 5 actionable steps

1) Make incidents reproducible before you change anything

Freeze model/code/config seeds and environment so you can reproduce the issue offline.

Capture the exact code revision:

git -C /srv/ai-app rev-parse --short HEAD 2>/dev/null || echo "Not a git repo"

Record Python and packages:

python3 -V
python3 -m pip freeze --all | sort > /tmp/pip-freeze.txt

Save config files and prompts:

tar -C /srv/ai-app/config -czf /tmp/config.tgz .

Snapshot model/artifact versions (examples):

# Hugging Face cache (if used)
ls -alh ~/.cache/huggingface/hub > /tmp/hf-cache-list.txt

# Container image digests (if containerized)
docker ps --no-trunc > /tmp/docker-ps.txt
docker inspect "$(docker ps -q)" > /tmp/docker-inspect.json 2>/dev/null || true

Capture seeds and relevant env without secrets:

printenv | grep -E 'SEED|CUDA|PYTORCH|TOKENIZER|HF_|TRANSFORMERS_' \
  | grep -Evi 'SECRET|KEY|TOKEN|PASS' > /tmp/ai-env.txt

Tip: Don’t restart services until you’ve captured logs and state. A restart can destroy volatile evidence.

2) Collect before you correct: use a triage script

Drop this script on any affected node to grab a consistent artifact bundle. It avoids obvious secret keys while keeping enough context to debug.

#!/usr/bin/env bash
# ai-incident-collect.sh — Minimal, reproducible artifact collector
# Usage:
#   sudo bash ai-incident-collect.sh --unit ai.service --logdir /var/log/ai \
#        --modeldir /srv/models --repodir /srv/ai-app --seconds 0
set -euo pipefail

UNIT=""
LOGDIR=""
MODELDIR=""
REPODIR=""
SECONDS_CAPTURE=0

while [[ $# -gt 0 ]]; do
  case "$1" in
    --unit) UNIT="$2"; shift 2;;
    --logdir) LOGDIR="$2"; shift 2;;
    --modeldir) MODELDIR="$2"; shift 2;;
    --repodir) REPODIR="$2"; shift 2;;
    --seconds) SECONDS_CAPTURE="$2"; shift 2;;
    *) echo "Unknown arg: $1" >&2; exit 1;;
  esac
done

TS="$(date -u +%Y%m%d-%H%M%S)"
OUT="ai-incident-${TS}"
mkdir -p "$OUT"

# System context
{
  echo "# uname"; uname -a
  echo "# os-release"; cat /etc/os-release || true
  echo "# uptime"; uptime
  echo "# disk"; df -h
  echo "# mem"; free -h
  echo "# cpu"; nproc; lscpu || true
  echo "# kernel cmdline"; cat /proc/cmdline
} > "${OUT}/system.txt"

# Network snapshot
{
  echo "# ip addr"; ip -br addr
  echo "# route"; ip route
  echo "# sockets"; ss -tupan
  echo "# resolv"; cat /etc/resolv.conf
} > "${OUT}/network.txt"

# Journal logs (if systemd unit provided)
if [[ -n "${UNIT}" ]]; then
  journalctl -u "${UNIT}" --since "24 hours ago" --no-pager > "${OUT}/journal-${UNIT}.log" || true
  systemctl status "${UNIT}" > "${OUT}/systemctl-${UNIT}.txt" || true
fi

# Process tree for common AI runtimes
ps auxww > "${OUT}/ps.txt"
pstree -a -p > "${OUT}/pstree.txt" || true

# Python env and packages
{
  which python3 || true
  python3 -V || true
  python3 -m pip freeze --all 2>/dev/null | sort || true
} > "${OUT}/python.txt"

# Git repo state (if provided)
if [[ -n "${REPODIR}" && -d "${REPODIR}/.git" ]]; then
  (
    cd "${REPODIR}"
    git rev-parse HEAD
    git status -sb
    git diff --stat
  ) > "${OUT}/git.txt" 2>&1 || true
fi

# Container info (if docker/podman present)
if command -v docker >/dev/null 2>&1; then
  docker ps --no-trunc > "${OUT}/docker-ps.txt" || true
  docker inspect $(docker ps -q) > "${OUT}/docker-inspect.json" 2>/dev/null || true
fi
if command -v podman >/dev/null 2>&1; then
  podman ps --no-trunc > "${OUT}/podman-ps.txt" || true
  podman inspect $(podman ps -q) > "${OUT}/podman-inspect.json" 2>/dev/null || true
fi

# GPU/accelerator
if command -v nvidia-smi >/dev/null 2>&1; then
  nvidia-smi -q -x > "${OUT}/nvidia-smi.xml" || true
fi
if command -v rocm-smi >/dev/null 2>&1; then
  rocm-smi --showallinfo > "${OUT}/rocm-smi.txt" || true
fi

# HF/transformers cache listing (non-sensitive)
if [[ -d "${HOME}/.cache/huggingface/hub" ]]; then
  find "${HOME}/.cache/huggingface/hub" -maxdepth 2 -type d -printf '%P\n' \
    > "${OUT}/hf-cache.txt" || true
fi

# Selected env (avoid obvious secrets)
printenv | grep -E 'SEED|CUDA|PYTORCH|TRANSFORMERS|HF_|TOKENIZER|OMP_' \
  | grep -Evi 'SECRET|KEY|TOKEN|PASS' \
  > "${OUT}/env.txt" || true

# Copy logs and model dir manifests
if [[ -n "${LOGDIR}" && -d "${LOGDIR}" ]]; then
  tar -C "${LOGDIR}" -czf "${OUT}/logs.tgz" . || true
fi
if [[ -n "${MODELDIR}" && -d "${MODELDIR}" ]]; then
  (cd "${MODELDIR}" && find . -type f -maxdepth 2 -print0 | xargs -0 sha256sum) \
    > "${OUT}/model-sha256.txt" || true
fi

# Optional short network capture (requires root), defaults to 0 seconds (skip)
if [[ "${SECONDS_CAPTURE}" -gt 0 ]]; then
  IFACE="$(ip -br link | awk '!/LOOPBACK/ && /UP/ {print $1; exit}')"
  timeout "${SECONDS_CAPTURE}" tcpdump -i "${IFACE}" -w "${OUT}/capture.pcap" || true
fi

tar -czf "${OUT}.tgz" "${OUT}"
echo "Wrote ${OUT}.tgz (keep secure: may contain sensitive logs)"

Example usage:

sudo bash ai-incident-collect.sh \
  --unit ai.service \
  --logdir /var/log/ai \
  --modeldir /srv/models \
  --repodir /srv/ai-app \
  --seconds 0

Store the resulting tarball in restricted incident storage.

3) Trace behavior at the boundaries (network, filesystem, GPU)

When symptoms are unclear, observe I/O safely.

Network egress spike or strange calls:

# Rolling capture with ring buffer (rotate 10 files of 25MB each)
sudo tcpdump -i any -s 0 -w /tmp/ai-ring-%Y%m%d%H%M%S.pcap \
  -G 60 -W 10 'tcp port 80 or tcp port 443'

Who is the process talking to?

# Replace <PID> with your model server pid
sudo ss -tpna | grep <PID>

Files the process touches (short window):

sudo strace -ff -e trace=file -p <PID> -o /tmp/strace-file  -tt -T
sleep 20; sudo kill -INT $(pidof strace) 2>/dev/null || true

Detect unexpected model/embedding index writes with auditd:

# Watch model directory for writes/executions
sudo auditctl -w /srv/models -p wa -k model_io
# Later, query records:
sudo ausearch -k model_io | aureport -f -i

GPU saturation or ECC events during incident:

nvidia-smi dmon -s pucm -d 1 -o DT -f /tmp/nvidia-dmon.csv 2>/dev/null || true

Caution: packet captures and traces can contain sensitive data. Minimize scope, timebox, and secure the outputs.

4) Log prompts, outputs, and decisions—responsibly

For LLM apps, the “what” and “why” are the prompt, system template, retrieved context, and tool calls. Log them with redaction.

Keep journald persistent (see install section).
Create a lightweight redactor for logs:

# Example: stream-redact.sh — masks emails and phone numbers in stdin
sed -E 's/[[:alnum:]\._%+-]+@[[:alnum:]\.-]+\.[A-Za-z]{2,}/[REDACTED_EMAIL]/g; \
        s/\b[0-9]{3}[-. ][0-9]{3}[-. ][0-9]{4}\b/[REDACTED_PHONE]/g'

Pipe app logs through redaction before disk:

# Example systemd unit snippet
[Service]
ExecStart=/srv/ai-app/bin/server
StandardOutput=journal+console
StandardError=journal
# Or if you write to files:
# ExecStart=/bin/sh -c '/srv/ai-app/bin/server 2>&1 | /usr/local/bin/stream-redact.sh >> /var/log/ai/app.log'

Rotate and protect:

sudo chown root:adm /var/log/ai
sudo chmod 0750 /var/log/ai

5) Classify root cause and lock in the fix

Use a simple taxonomy to speed remediation:

Data/config: wrong retrieval index, outdated embeddings, bad prompt template, missing seed.
Model: incompatible tokenizer/weights, quantization bug, unexpected upgrade.
Infra: driver/library mismatch, container image drift, OOM/latency from network or GPU contention.
Adversarial input: prompt injection, malicious tool output.

Map your next step:

Data/config → rebuild index, fix template, backfill embeddings, add version pinning.
Model → pin model+tokenizer digest, add checksum validation at startup.
Infra → pin container digests, add health checks, set resource limits, codify drivers/toolkit versions.
Adversarial → add input/output filters, retrieval allowlists, sandbox tools, constrain network egress.

Real-world style example: the “PII leak” hallucination that wasn’t

Symptom: A customer support bot returned what looked like a real phone number.

Investigation path: 1) Freeze context with the triage script before restarting. Artifacts show the exact code commit, model SHA, and prompt template. 2) Network and strace show the app contacting a self-hosted vector DB and reading an embeddings file updated 10 minutes earlier. 3) Auditd logs confirm write activity to the embeddings directory from a batch job. 4) Reproduction offline with the captured index reproduces the leak: the vector DB contained unredacted tickets from a staging sync job. 5) Fix: restore previous index, add redaction to the sync pipeline, pin the embeddings snapshot in prod, and add an auditd rule plus a checksum gate at service startup.

Outcome: MTTD reduced next time because vector index writes now fire alerts, and incidents become fully reproducible within minutes.

Handy one-liners you’ll reuse

Find incidents in the last hour across systemd:

journalctl --since '1 hour ago' -p 4..7 -o short-monotonic

Grep for chain-of-thought or tool calls in logs (example patterns):

rg -n --no-heading -e 'TOOL_CALL|RETRIEVAL|PROMPT:' /var/log/ai

Verify container immutability in deployment:

docker inspect <svc> --format '{{.Image}}' | xargs docker image inspect --format '{{.Id}} {{.RepoTags}}'

Diff current config vs. last known-good:

diff -u /etc/ai/config.yaml /etc/ai/config.last || true

Wrap-up and next steps (CTA)

Incidents in AI systems aren’t “just bugs.” They’re entangled states spanning code, models, data, and infrastructure. With a Bash-first playbook, you can preserve evidence, reproduce reliably, and fix the real causes.

Do this today:

Install the toolchain with your package manager.
Drop ai-incident-collect.sh into your fleet and document it in your runbook.
Enable persistent journald and auditd on production hosts.
Rehearse: run a tabletop exercise and verify you can reproduce a test “incident” offline.

Need a starting point? Take the triage script here, adapt paths to your stack, and commit it to your SRE/MLops repo. Your next 02:07 will go very differently.