Posted on
• Artificial Intelligence

Artificial Intelligence Incident Response

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Incident Response: A Bash‑First Playbook for Linux

When your AI service starts answering oddly, your GPU fans spike at 2 a.m., or a “harmless” dataset update suddenly bloats your egress bill—seconds matter. AI systems blend traditional infrastructure with new attack surfaces: model files, datasets, inference gateways, vector stores, and GPU runtimes. This post gives you a Bash-first, Linux-native incident response (IR) playbook tailored to AI workloads, with practical commands, tooling, and steps you can run today.

Problem/value:

  • AI stacks are high-value targets (data leakage, key exfiltration, cryptomining).

  • They’re complex: models, prompts, plugins/tools, and data pipelines change fast.

  • Traditional IR applies, but you also need model- and pipeline-aware actions.

What you’ll get:

  • A minimal, distro-friendly toolbox (apt/dnf/zypper installs included).

  • Actionable steps to prepare, detect, contain, eradicate, and recover.

  • Real-world examples you can map to your environment.

Install the Linux IR toolbox

Install these foundational tools once per host. If a package isn’t available on your distro, use your vendor’s extra repos or upstream instructions.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y auditd sysstat strace lsof tcpdump tshark jq whois git gnupg clamav clamav-daemon yara nftables

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y audit sysstat strace lsof tcpdump wireshark-cli jq whois git gnupg2 clamav clamav-update yara nftables

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y audit sysstat strace lsof tcpdump wireshark-cli jq whois git gpg2 clamav yara nftables

Optional service enables:

# Start auditd
sudo systemctl enable --now auditd

# Start ClamAV freshclam (defs) and daemon if used
# (Package/service names vary by distro)
sudo systemctl enable --now clamav-freshclam || true
sudo systemctl enable --now clamav-daemon || true

Why AI incident response is different (and valid)

  • New assets to protect: model artifacts (.pt, .onnx, .gguf, .safetensors), embeddings, vector DBs, fine-tuning datasets, system prompts.

  • New threats: prompt injection tool abuse, training data poisoning, model tampering/exfiltration, covert GPU cryptomining, dependency/supply-chain attacks, misconfigured inference endpoints.

  • Old truths still apply: least privilege, change control, logging, network containment, and fast, repeatable response.

Core playbook: 5 actionable steps

1) Prepare: Baseline models, logs, and change visibility

  • Inventory and hash model artifacts. Re-run after legitimate updates.
sudo mkdir -p /var/lib/ai-ir
sudo find /opt/models -type f -regextype posix-extended -regex '.*\.(pt|onnx|gguf|bin|safetensors)$' -print0 \
 | sort -z | xargs -0 sha256sum | sudo tee /var/lib/ai-ir/models.sha256
  • Watch critical paths with auditd (models and AI service configs).
sudo bash -c 'cat >/etc/audit/rules.d/ai-ir.rules <<EOF
-w /opt/models -p wa -k model-files
-w /etc/my-llm -p wa -k llm-config
EOF'
sudo augenrules --load
sudo systemctl restart auditd
  • Verify audit hits when files change:
sudo ausearch -k model-files --input-logs | sudo aureport -f -i
  • Baseline service behavior: CPU/GPU, ports, network egress.
# CPU/mem
ps aux --sort=-%cpu | head
mpstat 1 5

# GPU, if available
if command -v nvidia-smi >/dev/null; then nvidia-smi; fi

# Listening ports and connections
ss -tupna
  • Establish a triage pack script you can run under pressure:
sudo bash -c '
S=/tmp/ai-ir; mkdir -p "$S"
journalctl -u my-llm.service --since "-2h" > "$S/journal.txt" 2>/dev/null || true
ss -tpna > "$S/net.txt"
ps auxf > "$S/ps.txt"
lsof -nP > "$S/lsof.txt"
[ -d /opt/models ] && sha256sum /opt/models/* > "$S/model-now.sha256" 2>/dev/null || true
tar czf "/tmp/ai-ir-$(hostname)-$(date +%Y%m%d%H%M).tgz" -C /tmp ai-ir
echo "Triage pack: /tmp/ai-ir-$(hostname)-$(date +%Y%m%d%H%M).tgz"
'

2) Detect: Spot anomalies in logs, GPUs, and egress

  • Triage AI gateway/inference logs (JSON example):
# Errors and long responses
jq -r 'select(.level=="error") | .ts+" "+.msg' /var/log/ai/gateway.json 2>/dev/null
jq -r 'select(.latency_ms>2000) | .ts+" "+.route+" "+(.latency_ms|tostring)' /var/log/ai/gateway.json 2>/dev/null
  • Look for telltale process names and open sockets:
sudo lsof -i -nP | grep -E 'python|uvicorn|gunicorn|torchrun|text-generation|inference'
sudo ss -tpna | awk 'NR==1 || /:80|:443|:8000|:8080/'
  • Flag suspicious GPU use (cryptomining often runs headless):
if command -v nvidia-smi >/dev/null; then
  nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv
fi
  • Capture a bounded packet sample for analysis (don’t run indefinitely):
sudo tcpdump -i any -w /tmp/suspect.pcap -c 5000 host 0.0.0.0/0 and tcp
tshark -r /tmp/suspect.pcap -q -z conv,tcp | head
  • Enrich IPs quickly:
whois 203.0.113.50 | sed -n '1,40p'

3) Contain: Isolate fast, keep evidence

  • Stop only the affected service first (retain state for forensics):
sudo systemctl stop my-llm.service
sudo systemctl status my-llm.service
  • Block known-bad egress using nftables (replace IPs as needed):
sudo nft add table inet aiir 2>/dev/null || true
sudo nft 'add chain inet aiir out { type filter hook output priority 0; policy accept; }' 2>/dev/null || true
sudo nft add set inet aiir badips '{ type ipv4_addr; flags interval; }' 2>/dev/null || true
sudo nft add element inet aiir badips { 203.0.113.50, 198.51.100.23 }
sudo nft add rule inet aiir out ip daddr @badips drop
  • If containerized, pause or disconnect the container network (example):
docker ps --filter "name=my-llm" --format '{{.ID}}' | xargs -r docker pause
# or
docker network disconnect -f bridge my-llm-container
  • Immediately revoke/rotate exposed API keys and tokens.

4) Eradicate and analyze: Verify models, scan, and diff

  • Verify model integrity against your baseline:
cd / && sudo sha256sum -c /var/lib/ai-ir/models.sha256 | grep -v ': OK' || true
  • YARA scan for embedded secrets or known-bad strings:
cat > /tmp/secrets.yar <<'EOF'
rule SecretsInArtifacts {
  strings:
    $openai = /sk-[A-Za-z0-9]{20,}/
    $aws    = /AKIA[0-9A-Z]{16}/
    $gh     = /ghp_[A-Za-z0-9]{36,}/
  condition:
    any of them
}
EOF
yara -r /tmp/secrets.yar /opt/models /srv/ai 2>/dev/null
  • Malware scan relevant dirs:
clamscan -r -i /opt/models /srv/ai
  • Inspect recent code/config changes:
cd /srv/ai && git status && git log --since "48 hours ago" --stat --oneline
sudo ausearch -k llm-config --input-logs | sudo aureport -f -i
  • Network diff and forensics highlight:
comm -13 <(sort /tmp/baseline-ports.txt) <(ss -tupna | sort) 2>/dev/null || true
tshark -r /tmp/suspect.pcap -Y "tcp.flags.reset==1" -T fields -e ip.dst | sort | uniq -c | sort -nr | head
  • If you distribute signed model artifacts, verify signatures/hashes with GnuPG or your chosen tool:
gpg --verify model.onnx.asc model.onnx  # example if you keep detached signatures

5) Recover and harden: Rebuild cleanly and prevent repeats

  • Rebuild from known-good images, IaC, and pinned dependencies.

  • Rotate every credential possibly exposed; audit all .env files:

grep -R --include='.env*' -nE '(OPENAI|HUGGINGFACE|AWS|AZURE|GCP|TOKEN|SECRET)=' /etc /srv 2>/dev/null
  • Tighten permissions around models and configs:
sudo chown -R root:ai /opt/models /etc/my-llm
sudo chmod -R go-rwx /opt/models /etc/my-llm

  • Run inference as a dedicated non-root user; enforce SELinux/AppArmor; restrict egress with nftables sets; rate-limit endpoints at your reverse proxy.

  • Add specific detections for AI abuse (oversized prompts, tool-calls to disallowed domains, sudden token spikes) to your logs and alerts.

Three quick real-world‑style scenarios

  • Unauthorized GPU miner named “torchrun”

    • Signal: nvidia-smi shows high utilization from an odd process; unexpected outbound to crypto pools.
    • Action: stop service, nftables drop to pool IPs, capture pcap, kill rogue PID, verify model hashes, rebuild, rotate keys.

  • Prompt‑injection abuse of a tool plugin

    • Signal: logs show tool calls to disallowed domains and unusually long responses.
    • Action: contain by disabling tool integration, review logs with jq, restrict allowlist domains, add prompt/tool policy checks, rotate creds exposed in transcripts.

  • Model artifact tampering

    • Signal: sha256sum mismatch vs baseline; auditd shows write events during a deployment window.
    • Action: pull known-good model, verify signature, investigate deployment channel, enforce signed artifacts and immutable storage.

Conclusion and next steps

AI incident response favors teams that prepare baselines, know where their models and secrets live, and can isolate fast without destroying evidence. Turn the snippets above into your runbook:

  • Create a per-host baseline (hashes, ports, logs).

  • Add audit rules for model and config directories.

  • Practice containment with nftables and systemd.

  • Rehearse a tabletop exercise for prompt injection, model tampering, and GPU abuse.

Make it repeatable: save your triage pack script, store clean baselines under version control, and audit every change to your AI stack. Your future 2 a.m. self will thank you.

Related posts