Posted on
• Artificial Intelligence

Artificial Intelligence Linux Incident Response

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence + Bash = Faster, Smarter Linux Incident Response

It’s 03:17 and a pager goes off. A critical Linux host is behaving oddly—CPU spikes, a noisy auth.log, or a suspicious outbound connection. In those minutes that follow, speed and clarity decide whether you contain an incident or lose the thread.

This post shows how to combine battle-tested Bash triage with a tiny bit of local AI/ML to highlight what’s actually weird on a Linux box—right now. You’ll get runnable commands, a minimal ML script, and a repeatable flow you can operationalize in systemd. The result: less toil, fewer false trails, and faster incident decisions.

Why AI for Linux IR makes sense

  • Linux telemetry is noisy. Process trees, sockets, auth attempts, cron, systemd units, and kernel messages can drown responders.

  • Unsupervised ML is good at “what looks unusual right now?”—no labeled data required.

  • AI doesn’t replace analysts or controls (auditd, integrity monitoring, YARA). It prioritizes anomalies so humans look in the right places first.

  • You can run it locally, offline, and cheaply with standard Python packages—no cloud dependency required.

Prerequisites: Install the essentials

We’ll use standard Linux tools to collect data, plus Python for a small, local anomaly detector.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y auditd audispd-plugins jq lsof sysstat yara python3 python3-venv python3-pip
  • RHEL/CentOS/Fedora (dnf):
sudo dnf install -y audit jq lsof sysstat yara python3 python3-pip
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y audit auditd jq lsof sysstat yara python3 python3-pip

Then enable auditd:

sudo systemctl enable --now auditd

Add a minimal exec logging rule (test in staging first; logging can be chatty):

echo '-a always,exit -F arch=b64 -S execve -k exec' | sudo tee /etc/audit/rules.d/ai-ir.rules
# Optional for 32-bit userspace on 64-bit kernels:
echo '-a always,exit -F arch=b32 -S execve -k exec' | sudo tee -a /etc/audit/rules.d/ai-ir.rules
sudo augenrules --load
sudo systemctl restart auditd

Core workflow

1) Rapid triage: a one-command evidence bundle

Create a lightweight collector that runs anywhere. It captures system state, network, auth, cron, services, new SUID files, and a process feature CSV we’ll feed to an ML model.

Save as ai-ir-grab.sh:

#!/usr/bin/env bash
set -euo pipefail

TS="$(date -u +%Y%m%dT%H%M%SZ)"
HOST="$(hostname -f 2>/dev/null || hostname)"
DIR="/tmp/ir-${HOST}-${TS}"
mkdir -p "$DIR"

echo "[*] Collecting baseline to $DIR"

# System basics
{
  echo "# uname -a"; uname -a
  echo "# date -u"; date -u
  echo "# uptime"; uptime
  echo "# who"; who
} > "$DIR/system.txt" 2>&1

# Users and auth (recent)
last -n 50 > "$DIR/last.txt" 2>&1 || true
lastlog > "$DIR/lastlog.txt" 2>&1 || true
journalctl -S -24h -u ssh -u sshd -o short-iso > "$DIR/journal_ssh_24h.log" 2>&1 || true
journalctl -S -24h -o short-iso > "$DIR/journal_24h.log" 2>&1 || true

# Services and timers
systemctl list-units --type=service --state=running > "$DIR/services_running.txt" 2>&1 || true
systemctl list-timers --all > "$DIR/timers.txt" 2>&1 || true

# Processes
ps auxww > "$DIR/ps_aux.txt" 2>&1 || true
ps -eo pid,ppid,user,pcpu,pmem,etimes,comm --no-headers \
 | awk 'BEGIN{OFS=","; print "pid,ppid,user,pcpu,pmem,etimes,comm"} {print $1,$2,$3,$4,$5,$6,$7}' > "$DIR/procs.csv" 2>/dev/null || true

# Networking
ss -tulpen > "$DIR/ss_listen.txt" 2>&1 || true
ss -tanp > "$DIR/ss_tcp_all.txt" 2>&1 || true
ip -br a > "$DIR/ip_addr.txt" 2>&1 || true
ip r > "$DIR/ip_route.txt" 2>&1 || true
arp -an > "$DIR/arp.txt" 2>&1 || true

# Open files (can be large)
lsof -nP > "$DIR/lsof.txt" 2>&1 || true

# Cron and persistence clues
cp -a /etc/crontab "$DIR/" 2>/dev/null || true
cp -a /etc/cron.* "$DIR/" 2>/dev/null || true
for u in $(cut -d: -f1 /etc/passwd); do
  crontab -l -u "$u" >> "$DIR/user_crontabs.txt" 2>/dev/null || true
done

# SUID/SGID files (might be slow)
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -printf "%M %u %g %s %TY-%Tm-%Td %TH:%TM %p\n" 2>/dev/null | sort > "$DIR/suid_sgid.txt" || true

# Recently changed in sensitive dirs (last 48h)
find /etc /usr /bin /sbin /lib /lib64 /opt -xdev -type f -mtime -2 -printf "%TY-%Tm-%Td %TH:%TM %p\n" 2>/dev/null | sort > "$DIR/recent_changes_48h.txt" || true

# Executable hashes of running processes
> "$DIR/running_binaries.txt"
for pid in $(ps -e -o pid=); do
  exe="$(readlink -f /proc/$pid/exe 2>/dev/null || true)"
  if [ -n "${exe:-}" ] && [ -e "$exe" ]; then
    echo "$exe" >> "$DIR/running_binaries.txt"
  fi
done
sort -u "$DIR/running_binaries.txt" -o "$DIR/running_binaries.txt"
xargs -r sha256sum < "$DIR/running_binaries.txt" > "$DIR/running_binaries.sha256" 2>/dev/null || true

# Audit (recent execs)
ausearch -k exec --start recent > "$DIR/audit_exec_recent.txt" 2>/dev/null || true

# Optional: quick YARA sweep of /tmp and user homes
if command -v yara >/dev/null 2>&1; then
  mkdir -p "$DIR/yara"
  # You need your own rules; placeholder for demonstration
  echo 'rule Dummy_Example { condition: false }' > "$DIR/yara/dummy.yar"
  yara -r "$DIR/yara/dummy.yar" /tmp ~/. 2>/dev/null > "$DIR/yara/scan.txt" || true
fi

# Package verification (may be slow; skip if not present)
if command -v dpkg >/dev/null 2>&1; then dpkg -V > "$DIR/pkg_verify.txt" 2>&1 || true; fi
if command -v rpm  >/dev/null 2>&1; then rpm -Va > "$DIR/pkg_verify.txt" 2>&1 || true; fi

# Compress bundle
tar -C /tmp -czf "${DIR}.tar.gz" "$(basename "$DIR")"
echo "[*] Bundle ready: ${DIR}.tar.gz"

Run it:

chmod +x ai-ir-grab.sh
sudo ./ai-ir-grab.sh

You now have an incident bundle plus a clean procs.csv with features for modeling.

2) Tiny local ML: flag unusual processes with Isolation Forest

We’ll use a small, unsupervised model to score processes and surface the odd ones first. No internet needed.

Create and activate a virtual environment:

python3 -m venv ~/ai-ir-venv
source ~/ai-ir-venv/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn numpy

Save as anomaly_iforest.py:

#!/usr/bin/env python3
import argparse, pandas as pd, numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler

ap = argparse.ArgumentParser()
ap.add_argument("--csv", required=True, help="Path to procs.csv")
ap.add_argument("--top", type=int, default=15, help="Number of top anomalies to show")
args = ap.parse_args()

df = pd.read_csv(args.csv)
# Basic cleaning
df = df.dropna()
# Encode categorical columns
for col in ["user","comm"]:
    df[col] = df[col].astype("category")
    df[col + "_code"] = df[col].cat.codes

features = ["pcpu","pmem","etimes","ppid","user_code","comm_code"]
X = df[features].astype(float).values

# Scale numeric features (helps tree stability across ranges)
sc = StandardScaler()
X_scaled = sc.fit_transform(X)

# Fit Isolation Forest
model = IsolationForest(
    n_estimators=300,
    contamination="auto",
    random_state=42,
    n_jobs=-1,
    max_samples="auto"
)
model.fit(X_scaled)
scores = -model.score_samples(X_scaled)  # higher = more anomalous
df["_anomaly_score"] = scores

out = df.sort_values("_anomaly_score", ascending=False).head(args.top)
cols = ["_anomaly_score","pid","ppid","user","pcpu","pmem","etimes","comm"]
print(out[cols].to_string(index=False))

print("\nNext steps per PID:")
for pid in out["pid"].astype(int).tolist():
    print(f"- PID {pid}: sudo ls -l /proc/{pid}/exe; sudo lsof -p {pid}; sudo ss -tanp | grep ' {pid} '; sudo strings -a $(readlink -f /proc/{pid}/exe) | head")

Run it against the triage output:

python3 anomaly_iforest.py --csv /tmp/ir-*/procs.csv --top 15

You’ll get a ranked list of processes by “weirdness,” along with immediate follow-up commands.

Notes:

  • The features here are deliberately simple. Add others (e.g., count of open sockets, binary path entropy, parent process name code) to improve signal.

  • Expect false positives on first run. Build a baseline: export today’s scores, and compare deltas across days.

3) Hunt auth anomalies with a quick Bash-to-ML pipeline

Auth noise often hides real attacks. Summarize access patterns and score the outliers.

Extract SSH auth summaries for the last 24h:

journalctl -S -24h -u ssh -u sshd -o short-iso \
 | awk '/sshd/ && /Failed|Accepted/ {print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10}' > /tmp/ssh_events_24h.txt

Produce a simple CSV of counts per source IP and outcome:

awk '
  /Failed/  {split($0,a,"rhost="); split(a[2],b," "); ip=b[1]; fail[ip]++}
  /Accepted/{split($0,a,"rhost="); split(a[2],b," "); ip=b[1]; ok[ip]++}
  END{
    print "ip,failed,accepted"
    for (i in fail) {printf "%s,%d,%d\n", i, fail[i], (i in ok)?ok[i]:0}
    for (i in ok) if (!(i in fail)) {printf "%s,%d,%d\n", i, 0, ok[i]}
  }
' /tmp/ssh_events_24h.txt > /tmp/ssh_ip_counts.csv

Score IPs with high fail/accept imbalance using the same environment:

python3 - << 'PY'
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler

df = pd.read_csv("/tmp/ssh_ip_counts.csv")
df = df.fillna(0)
df["ratio"] = (df["failed"]+1) / (df["accepted"]+1)
X = df[["failed","accepted","ratio"]].values
X = StandardScaler().fit_transform(X)
m = IsolationForest(n_estimators=200, random_state=42).fit(X)
scores = -m.score_samples(X)
df["_anomaly_score"] = scores
print(df.sort_values("_anomaly_score", ascending=False).head(10).to_string(index=False))
PY

Use the results to prioritize blocks, investigations, or MFA checks.

4) Real-world style example

  • Symptom: A production host shows periodic CPU spikes. SSH logins appear normal at first glance.

  • Triage:

    • Run sudo ./ai-ir-grab.sh to collect evidence.
    • Run python3 anomaly_iforest.py --csv /tmp/ir-*/procs.csv.

  • Highlighted anomaly (sample output):

 _anomaly_score   pid  ppid   user  pcpu  pmem  etimes       comm
        0.9876  7421     1   www-data  24.3   1.2    180    kworker

  • Rapid checks:

    • sudo ls -l /proc/7421/exe shows the binary symlink points to /tmp/.cache/.k instead of a system path.
    • sudo lsof -p 7421 reveals an outbound connection to an unfamiliar IP on high port.
    • sha256sum of that binary doesn’t match any internal allowlist.

  • Containment: Block egress at the host firewall, isolate the node, and begin deeper forensic acquisition. The AI step didn’t “solve” the incident—it put the real lead in front of you fast.

5) Operationalize it with systemd timers

Run the triage and scoring periodically (e.g., hourly) and drop reports in a known directory.

Service unit /etc/systemd/system/ai-ir-collect.service:

[Unit]
Description=AI-assisted IR triage and scoring

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/ai-ir-collect-and-score.sh

Wrapper script /usr/local/sbin/ai-ir-collect-and-score.sh:

#!/usr/bin/env bash
set -euo pipefail

BUNDLE_OUT="$(sudo /usr/local/sbin/ai-ir-grab.sh | awk '/Bundle ready:/ {print $3}')"
CSV="$(tar -tzf "$BUNDLE_OUT" | grep procs.csv || true)"
OUTDIR="/var/log/ai-ir"
mkdir -p "$OUTDIR"

if [ -n "$CSV" ]; then
  TMPDIR="$(mktemp -d)"
  tar -C "$TMPDIR" -xzf "$BUNDLE_OUT" "$CSV"
  source ~/ai-ir-venv/bin/activate
  python3 /usr/local/sbin/anomaly_iforest.py --csv "$TMPDIR/$CSV" --top 20 > "$OUTDIR/last_procs_anomalies.txt" || true
  rm -rf "$TMPDIR"
fi

Timer unit /etc/systemd/system/ai-ir-collect.timer:

[Unit]
Description=Run AI IR triage hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-ir-collect.timer

Tip: If you want notifications, integrate with your preferred notifier (Syslog, SIEM forwarder, or a local mailer). On Debian-like systems, sudo apt install -y mailutils provides a simple mail command; on RHEL/SUSE, sudo dnf install -y mailx or sudo zypper install -y mailx.

Good practices and guardrails

  • Baseline first: Expect many “weird” but benign processes on day one. Save scores and compare over time; build a small allowlist for known batch jobs and agents.

  • Least privilege: These collectors often need root. Protect scripts, outputs, and hashes; rotate and restrict permissions.

  • Test audit rules: Exec logging can be verbose. Start narrow, measure impact, and expand rules as needed.

  • Keep models simple: Isolation Forest, one-class SVM, or PyOD are enough for host-level anomaly surfacing. Complexity ≠ value during an incident.

  • Respect privacy: Scrub sensitive data before exporting bundles outside the host or environment.

Conclusion and next steps

AI won’t replace your Linux incident response playbooks—but it can slash the time from “too much noise” to “this is probably it.” With a Bash-first evidence bundle and a tiny local model, you’ll consistently spotlight what’s off-pattern: odd parents, unexpected longevity, surprising resource use, and strange login patterns.

Your next step:

  • Roll this into a staging host today. Tune features and thresholds for your environment.

  • Add one or two features that matter for you (e.g., parent process name codes, socket counts per PID).

  • Wire the timer to your alerting path and track mean-time-to-triage improvements.

If you want a deeper dive, extend the same pattern to:

  • Integrity change feeds (AIDE, package verification deltas)

  • Kernel event streams (eBPF) summarized and scored

  • Cross-host comparisons (what’s unique on this node vs. its peers)

Small, local AI plus solid Bash hygiene is a force multiplier. Start simple. Iterate fast. Catch more.

Related posts