Posted on
• Artificial Intelligence

Artificial Intelligence Incident Response Playbooks

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Incident Response Playbooks for Linux Bash Operators

If your GPUs spike to 100%, your model starts calling home, or your chatbot suddenly “remembers” secrets it shouldn’t—what do you do in the next 15 minutes? Traditional IR (incident response) playbooks miss the quirks of AI systems: model weights, prompts, datasets, vector stores, and opaque pipelines. This post gives you a Bash-first, Linux-native playbook you can run at 3 a.m., with concrete commands and packages you can install via apt, dnf, or zypper.

Why this matters:

  • AI workloads expand your attack surface (poisoned models, prompt injection, data exfiltration, rogue egress).

  • GPU boxes are a juicy target for cryptomining.

  • Regulations increasingly require auditability and repeatable response steps.

  • Speed matters. The first 15 minutes decide how much evidence you keep and how far the blast radius grows.

Below, you’ll get a ready-to-adapt playbook, simple scripts, and real-world scenarios you can use in drills.

Quick setup: tools you’ll need

These are common, repo-native packages. Install all of them now and you’ll avoid “command not found” when it’s urgent.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y jq yq auditd podman lsof nftables rsync tar coreutils curl
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y jq yq audit podman lsof nftables rsync tar coreutils curl
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y jq yq audit podman lsof nftables rsync tar coreutils curl

Optional but handy:

  • ClamAV (scan for known malware)

    • apt: sudo apt install -y clamav
    • dnf: sudo dnf install -y clamav
    • zypper: sudo zypper install -y clamav

  • If you already use vendor GPU drivers, you likely have nvidia-smi (NVIDIA) or rocm-smi (AMD). If present, we’ll use them for GPU triage; if not, we’ll fall back to generic tools.

Set up persistent journald (so logs survive reboots):

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Enable auditd (file access auditing):

# apt-based distros
sudo systemctl enable --now auditd

# dnf/zypper-based distros (same command)
sudo systemctl enable --now auditd

What makes AI incidents different

  • New “assets” to protect: model weights/checkpoints, prompts/system instructions, datasets, training logs, vector indexes, and feature stores.

  • Model supply chain: downloads from public hubs, pip/conda dependencies, container images—more places to slip in malware or poisoning.

  • High-sensitivity data: embeddings and logs can leak PII or secrets.

  • GPU resources: easy to steal for cryptomining.

  • Non-determinism and dynamic prompts: post-incident analysis must capture context, not just binaries.

The AI IR Playbook (Bash-first)

1) Prepare and baseline (do this before an incident)

Create a manifest of models and checkpoints:

MODEL_DIR=/srv/models
sudo find "$MODEL_DIR" -type f -regextype posix-extended -regex '.*\.(pt|pth|onnx|tflite|bin|safetensors|gguf)$' -print0 \
  | sudo xargs -0 sha256sum | sudo tee /var/log/model_hashes.$(date -I).txt

Record runtime environments (helps you rebuild deterministically):

# Python packages
pip freeze | sudo tee /var/log/pip_freeze.$(date -I).txt 2>/dev/null || true

# Conda (if used)
conda env export | sudo tee /var/log/conda_env.$(date -I).yml 2>/dev/null || true

Audit rules to watch model directories and config:

sudo tee /etc/audit/rules.d/ai-models.rules >/dev/null <<'EOF'
-w /srv/models -p wa -k model_mods
-w /etc/ai/ -p wa -k ai_config
EOF
sudo augenrules --load || sudo service auditd restart

Keep a simple runbook file (path to model dir, service names, LB addresses, identities that deploy, on-call contacts). When every second counts, you don’t want to guess.

Tip: Pin and verify external model downloads by commit and checksum. Save known-good hashes with the baseline above.

2) Detect early (symptoms and commands)

Watch recent file modifications to models and configs:

sudo ausearch -k model_mods --start recent
sudo ausearch -k ai_config --start recent

Look for unusual outbound connections from your model server user/process:

# Show TCP connections with processes
ss -tpna

# What’s touching your model directory right now?
sudo lsof -Fn -- /srv/models

GPU anomalies (if CLI is available):

# NVIDIA
command -v nvidia-smi && nvidia-smi pmon -s um || true

# AMD ROCm
command -v rocm-smi && rocm-smi --showpidgpus --json || true

Service logs (journalctl):

# Last 15 minutes for your inference service
sudo journalctl -u inference.service --since "-15 min" -o short-iso

Heuristics to flag:

  • New/modified model files off-hours

  • Inference service opening outbound sockets it normally doesn’t need

  • GPU utilization at 100% with strange process names or users

  • Spikes in token counts or prompt length; repeated “system prompt” tampering patterns

  • Unexpected model downloads (new directories in /srv/models or cache paths)

3) Triage and contain (limit blast radius fast)

Contain outbound traffic safely (egress allowlist). Example: allow only RFC1918 (internal) and drop the rest:

sudo nft add table inet isolate
sudo nft add chain inet isolate egress { type filter hook output priority 0\; }
sudo nft add rule inet isolate egress ip daddr != {10.0.0.0/8,192.168.0.0/16,172.16.0.0/12} drop

Pause or stop services without deleting evidence:

# Systemd service
sudo systemctl stop inference.service

# Podman containers (identify, then pause and snapshot)
podman ps
podman pause <ctr_id>
podman commit <ctr_id> incident_snapshot:$(date +%s)
podman export <ctr_id> > /var/tmp/container.<ctr_id>.tar

Quarantine suspicious model files (preserve, don’t delete):

SUSP=/srv/models/suspect.safetensors
sudo mv "$SUSP" "${SUSP}.quarantine.$(date +%s)"
# Optional (ext filesystems): make immutable to avoid tampering
sudo chattr +i "${SUSP}.quarantine."*

If you suspect a poisoned index or embeddings, snapshot before you purge:

sudo tar --xattrs --acls -cpf /var/tmp/vector_index.$(date +%s).tar /srv/vector-store

4) Collect evidence (make forensics easy)

Use a single case folder to keep everything tidy:

CASE=/var/ir/case-$(date +%s)
sudo install -d -m 700 "$CASE"

# System + service logs
sudo journalctl -S -2h -o short-iso > "$CASE/journal.last2h.log"
sudo journalctl -u inference.service -S -24h -o short-iso > "$CASE/inference.last24h.log" 2>/dev/null || true

# Auditd traces
sudo ausearch -k model_mods -ts recent > "$CASE/audit_model_mods.log"
sudo ausearch -k ai_config -ts recent > "$CASE/audit_ai_config.log"

# Processes, env, open files
ps auxfw > "$CASE/ps.txt"
env -0 > "$CASE/env.nullsep.txt"
sudo lsof -nP > "$CASE/lsof.txt"
sudo lsof -Fn -- /srv/models > "$CASE/lsof_models.txt"

# Network and routing
ss -tpna > "$CASE/ss.txt"
ip -d addr show > "$CASE/ip_addr.txt"
ip route show table all > "$CASE/ip_route.txt"

# GPU state (if available)
command -v nvidia-smi && nvidia-smi -q > "$CASE/nvidia-smi.txt" || true
command -v rocm-smi && rocm-smi --showpidgpus --json > "$CASE/rocm-smi.json" || true

# Models and configs snapshot with checksums
sudo tar --xattrs --acls -cpf "$CASE/models.tar" /srv/models
sha256sum "$CASE/models.tar" > "$CASE/models.tar.sha256"
sudo tar --xattrs --acls -cpf "$CASE/etc-ai.tar" /etc/ai 2>/dev/null || true

# If containerized, save images and containers
podman ps --no-trunc > "$CASE/podman_ps.txt" 2>/dev/null || true
for ID in $(podman ps -q 2>/dev/null); do
  podman inspect "$ID" > "$CASE/ctr.$ID.inspect.json" || true
done

Optional malware sweep (doesn’t replace real forensics):

sudo freshclam 2>/dev/null || true
sudo clamscan -r --bell -i /srv/models > "$CASE/clamav_models.txt" 2>/dev/null || true

Hash the entire case for integrity:

( cd /var/ir && sudo tar -cpf - "$(basename "$CASE")" ) | sha256sum | sudo tee "$CASE.bundle.sha256"

5) Eradicate and recover (return to known-good)

  • Verify model artifacts against your baseline:
sudo find /srv/models -type f -regextype posix-extended -regex '.*\.(pt|pth|onnx|tflite|bin|safetensors|gguf)$' -print0 \
  | sudo xargs -0 sha256sum | tee "$CASE/post_fix_hashes.txt"

# Compare with last baseline (example)
diff -u /var/log/model_hashes.*.txt "$CASE/post_fix_hashes.txt" || true
  • If mismatched or unknown, restore from a pinned, checksummed source. Example (pin to a commit and verify):
cd /srv/models/clean
curl -fsSLO https://huggingface.co/owner/repo/resolve/<commit>/model.safetensors
echo '<expected_sha256>  model.safetensors' | sha256sum -c -

  • Rebuild containers from a minimal, pinned Containerfile; avoid mutable latest tags. Rotate API keys/secrets used by the affected service.

  • For RAG or fine-tuned models, rehydrate from clean checkpoints and rebuild indexes from verified data. If you suspect data poisoning, rebuild training datasets from trusted sources and re-run training jobs on isolated compute.

  • Re-enable egress gradually via allowlist (don’t flip the firewall off all at once).

  • Add a regression test for the incident’s trigger (e.g., a malicious prompt sample) to prevent recurrence.

A tiny helper: one-command triage script

Drop this in a private admin repo and adapt the paths/service names.

#!/usr/bin/env bash
# ai-ir.sh — quick isolate + collect for AI services
# Usage: sudo ./ai-ir.sh inference.service /srv/models /etc/ai
set -euo pipefail
SERVICE=${1:-inference.service}
MODEL_DIR=${2:-/srv/models}
CONF_DIR=${3:-/etc/ai}
TS=$(date +%s)
CASE=/var/ir/case-$TS
sudo install -d -m 700 "$CASE"

echo "[*] Pausing egress (allow RFC1918 only)"
sudo nft list table inet isolate >/dev/null 2>&1 || {
  sudo nft add table inet isolate
  sudo nft add chain inet isolate egress "{ type filter hook output priority 0; }"
  sudo nft add rule inet isolate egress ip daddr != "{10.0.0.0/8,192.168.0.0/16,172.16.0.0/12}" drop
}

echo "[*] Stopping $SERVICE"
sudo systemctl stop "$SERVICE" || true

echo "[*] Collecting evidence to $CASE"
sudo journalctl -S -2h -o short-iso > "$CASE/journal.last2h.log" || true
sudo journalctl -u "$SERVICE" -S -24h -o short-iso > "$CASE/service.last24h.log" || true
sudo ausearch -k model_mods -ts recent > "$CASE/audit_model_mods.log" || true
sudo ausearch -k ai_config -ts recent > "$CASE/audit_ai_config.log" || true
ps auxfw > "$CASE/ps.txt"
env -0 > "$CASE/env.nullsep.txt"
sudo lsof -nP > "$CASE/lsof.txt" || true
sudo lsof -Fn -- "$MODEL_DIR" > "$CASE/lsof_models.txt" || true
ss -tpna > "$CASE/ss.txt" || true
ip -d addr show > "$CASE/ip_addr.txt" || true
ip route show table all > "$CASE/ip_route.txt" || true
command -v nvidia-smi && nvidia-smi -q > "$CASE/nvidia-smi.txt" || true
command -v rocm-smi && rocm-smi --showpidgpus --json > "$CASE/rocm-smi.json" || true
sudo tar --xattrs --acls -cpf "$CASE/models.tar" "$MODEL_DIR" || true
sha256sum "$CASE/models.tar" > "$CASE/models.tar.sha256" || true
sudo tar --xattrs --acls -cpf "$CASE/etc-ai.tar" "$CONF_DIR" || true
echo "[*] Done. Case: $CASE"

Real-world mini-scenarios (and what works)

1) Prompt-injection exfil in a RAG service
Symptoms: Logs show the model quoting “system prompt,” sudden outbound egress to unknown IPs.
Works: Immediate egress block via nftables; snapshot vector index; grep logs for injection markers; rotate API keys; add domain/IP allowlist; strip/validate user-provided context; set max prompt sizes.

2) Poisoned model from public hub
Symptoms: New .safetensors appears; baseline hashes don’t match; suspicious post-init network calls.
Works: Quarantine the file; verify against pinned commit + sha256; rebuild container from minimal base; disable implicit auto-downloads; enforce curl + sha256sum for any model update.

3) GPU cryptominer disguised as training
Symptoms: nvidia-smi shows unexpected process at 100% GPU; ss -tpna reveals connections to mining pools; high CPU for stratum-like traffic.
Works: Pause container; block egress; export and inspect container; re-deploy from signed image; enforce resource quotas and per-user allowlists; add detections for stratum protocol patterns.

Conclusion and next steps

AI incidents blend classic IR with new moving parts: model artifacts, prompts, datasets, GPUs, and volatile context. The fix is preparation plus a repeatable, Bash-friendly playbook.

Your next steps:

  • Install the tools (apt/dnf/zypper commands above) and enable auditd now.

  • Baseline your models and environments today; schedule a weekly hash check.

  • Copy the triage script, tailor paths/service names, and run a 30-minute game day.

  • Add egress allowlists and pin model downloads by commit and checksum.

Want a deeper dive? Turn each section into an internal runbook page with owners, SLAs, and “break glass” tokens. Print it. When it’s 3 a.m., you’ll be glad you did.

Related posts