Artificial Intelligence for Linux Security Auditing (with Bash-first workflows)

If you’re responsible for Linux systems, you already know: the logs are endless, the alerts are noisy, and attackers don’t wait. What if you could make your logs work for you—surfacing the unusual, the rare, and the truly suspicious—without buying a heavy commercial SIEM? That’s where a little practical AI can help.

This article shows how to add lightweight, auditable AI to your Linux security auditing using tools you already trust: Bash, systemd, auditd, and a small Python ML stack. You’ll:

• Capture high-fidelity security signals (auditd, journald, process accounting)

• Build and run a compact anomaly detector against SSH auth logs

• Automate it with systemd timers so you get regular insights, not one-off experiments

All commands include apt, dnf, and zypper options where packages are installed.

---

Why AI for Linux auditing is worth your time

• Pattern discovery beats static rules: Rule-based tools are great, but they can miss weird-but-real attacks. Unsupervised ML can find rare events you didn’t think to write a rule for.

• Prioritization: AI helps you bubble up the top 10 suspicious events from millions of lines, so you can investigate faster.

• Works with what you have: Journald, auditd, and your existing logs are already fantastic training data. You don’t need to ship your data to the cloud or install a giant stack.

---

1) Capture the right signals (auditd, journald, process accounting)

First, make sure you’re collecting rich, reliable telemetry. That means auditd for system calls and file watches, persistent journald, and optional process accounting.

• Install auditd

  • apt:

  sudo apt update
  sudo apt install -y auditd audispd-plugins
  

  • dnf:

  sudo dnf install -y audit audispd-plugins
  

  • zypper:

  sudo zypper install -y audit
  

  • Enable:

  sudo systemctl enable --now auditd
  sudo auditctl -s
  

• Make journald persistent

  sudo mkdir -p /var/log/journal
  sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
  sudo systemctl restart systemd-journald
  

• (Optional but useful) Enable process accounting for process-level forensics

  • apt (Debian/Ubuntu: package “acct”, service “acct”):

  sudo apt install -y acct
  sudo systemctl enable --now acct
  

  • dnf (RHEL/Fedora: package “psacct”, service “psacct”):

  sudo dnf install -y psacct
  sudo systemctl enable --now psacct
  

  • zypper (openSUSE: package “acct”, service “acct”):

  sudo zypper install -y acct
  sudo systemctl enable --now acct
  

• Add a few high-value audit rules (file integrity and user/group DB changes)

  sudo tee /etc/audit/rules.d/hardening.rules >/dev/null <<'RULES'
  -w /etc/passwd -p wa -k passwd_changes
  -w /etc/shadow -p wa -k shadow_changes
  -w /etc/group  -p wa -k group_changes
  -w /usr/bin/sudo -p x -k sudo_exec
  -a always,exit -F arch=b64 -S execve -k exec_monitor
  -a always,exit -F arch=b32 -S execve -k exec_monitor
  RULES
  
  sudo augenrules --load || sudo service auditd restart
  

These rules ensure you’ll have the raw material AI needs: concrete events tied to identities, processes, and files.

---

2) Install a tiny, local ML toolchain

We’ll keep it simple: Python 3 plus scikit-learn, pandas, and numpy. No cloud. No data leaves your box.

• Install Python and pip

  • apt:

  sudo apt update
  sudo apt install -y python3 python3-pip python3-venv
  

  • dnf:

  sudo dnf install -y python3 python3-pip
  

  • zypper:

  sudo zypper install -y python3 python3-pip
  

• Install Python libraries (user scope)

  python3 -m pip install --user --upgrade pip
  python3 -m pip install --user scikit-learn pandas numpy
  

Tip: If you prefer isolation, you can use a venv:

python3 -m venv ~/.venvs/ai-audit
~/.venvs/ai-audit/bin/pip install scikit-learn pandas numpy

---

3) Real-world example: SSH anomaly detection in under 50 lines

This script ingests your SSH authentication logs, extracts simple features (hour, success/failure, username length, IP octets, method), and uses an Isolation Forest to rank unusual events.

It works with both Debian/Ubuntu (/var/log/auth.log) and RHEL/Fedora/openSUSE (/var/log/secure).

• Install the script

  sudo mkdir -p /opt/ai-audit
  sudo tee /opt/ai-audit/detect_auth_anomalies.py >/dev/null <<'PY'
  #!/usr/bin/env python3
  import os, re, json, socket
  from datetime import datetime
  import numpy as np
  from sklearn.ensemble import IsolationForest
  
  # Pick the auth log path
  CANDIDATES = ["/var/log/auth.log", "/var/log/secure"]
  LOG = next((p for p in CANDIDATES if os.path.exists(p)), None)
  if not LOG:
  print("No auth log found (checked /var/log/auth.log and /var/log/secure).")
  raise SystemExit(1)
  
  # Basic parser for sshd lines
  line_re = re.compile(r'^(?P<ts>[A-Z][a-z]{2}\s+\d+\s+\d{2}:\d{2}:\d{2})\s+(?P<host>\S+)\s+sshd\[\d+\]:\s+(?P<msg>.+)$')
  ip_re = re.compile(r'(\d{1,3}\.){3}\d{1,3}')
  user_re = re.compile(r'\bfor\s+([A-Za-z0-9._-]+)\b')
  
  def parse_dt(ts_str):
  # Add current year; journald-like syslog timestamps omit the year
  now = datetime.now()
  try:
    dt = datetime.strptime(f"{ts_str} {now.year}", "%b %d %H:%M:%S %Y")
    # handle year rollover (Jan logs in early Jan referring to Dec of previous year)
    if (now - dt).days < -300:
      dt = datetime.strptime(f"{ts_str} {now.year - 1}", "%b %d %H:%M:%S %Y")
    return dt
  except Exception:
    return None
  
  rows = []
  raw = []
  
  with open(LOG, "r", errors="ignore") as fh:
  for line in fh:
    m = line_re.match(line)
    if not m:
      continue
    ts = parse_dt(m.group("ts"))
    if not ts:
      continue
    msg = m.group("msg")
  
    # Focus on key sshd outcomes
    interesting = any(s in msg for s in ["Failed password", "Accepted password", "Invalid user", "Accepted publickey"])
    if not interesting:
      continue
  
    ip = ip_re.search(msg)
    ip = ip.group(0) if ip else "0.0.0.0"
  
    user_m = user_re.search(msg)
    user = user_m.group(1) if user_m else "unknown"
  
    method = 0
    if "publickey" in msg:
      method = 1
    elif "password" in msg:
      method = 2
  
    failed = 1 if ("Failed password" in msg or "Invalid user" in msg) else 0
    invalid_user = 1 if "Invalid user" in msg else 0
  
    # Featureize IP (IPv4 octets, crude but often useful)
    try:
      octs = [int(x) for x in ip.split(".")]
      if len(octs) != 4 or any(o < 0 or o > 255 for o in octs):
        octs = [-1, -1, -1, -1]
    except Exception:
      octs = [-1, -1, -1, -1]
  
    feat = [
      ts.hour,
      failed,
      invalid_user,
      len(user),
      method,
      *octs
    ]
    rows.append(feat)
    raw.append({
      "timestamp": ts.isoformat(),
      "user": user,
      "ip": ip,
      "message": msg.strip()
    })
  
  if not rows:
  print("No relevant sshd events found.")
  raise SystemExit(0)
  
  X = np.array(rows, dtype=float)
  
  # Train Isolation Forest on the observed data (unsupervised)
  clf = IsolationForest(
  n_estimators=200,
  contamination="auto",
  random_state=42
  )
  clf.fit(X)
  
  # Higher score => more abnormal if we invert sign
  scores = -clf.score_samples(X)
  
  # Take top N anomalies
  N = min(30, len(scores))
  idxs = np.argsort(scores)[-N:][::-1]
  
  anomalies = []
  for i in idxs:
  r = raw[i]
  anomalies.append({
    "timestamp": r["timestamp"],
    "user": r["user"],
    "ip": r["ip"],
    "anomaly_score": float(scores[i]),
    "message": r["message"]
  })
  
  out_path = "/var/log/ai-ssh-anomalies.json"
  try:
  with open(out_path, "w") as f:
    json.dump(anomalies, f, indent=2)
  except PermissionError:
  # Fallback to user-writable location if not root
  home_out = os.path.expanduser("~/ai-ssh-anomalies.json")
  with open(home_out, "w") as f:
    json.dump(anomalies, f, indent=2)
  out_path = home_out
  
  print(f"[ai-audit] Wrote {len(anomalies)} anomalies to {out_path}")
  for a in anomalies[:10]:  # also print the top few
  print(f"{a['timestamp']} user={a['user']} ip={a['ip']} score={a['anomaly_score']:.4f} :: {a['message']}")
  PY
  
  sudo chmod +x /opt/ai-audit/detect_auth_anomalies.py
  

• Run it manually

  python3 /opt/ai-audit/detect_auth_anomalies.py
  

• Automate with systemd (runs every 30 minutes and logs to journald)

  sudo tee /etc/systemd/system/ai-ssh-audit.service >/dev/null <<'UNIT'
  [Unit]
  Description=AI SSH anomaly detector
  
  [Service]
  Type=oneshot
  ExecStart=/usr/bin/env bash -lc 'python3 /opt/ai-audit/detect_auth_anomalies.py | tee -a /var/log/ai-ssh-anomalies.log | logger -t ai-ssh-audit'
  Nice=10
  ProtectSystem=full
  ProtectHome=true
  PrivateTmp=true
  CapabilityBoundingSet=
  NoNewPrivileges=true
  UNIT
  
  sudo tee /etc/systemd/system/ai-ssh-audit.timer >/dev/null <<'UNIT'
  [Unit]
  Description=Run AI SSH anomaly detector periodically
  
  [Timer]
  OnBootSec=5m
  OnUnitActiveSec=30m
  AccuracySec=1m
  Persistent=true
  
  [Install]
  WantedBy=timers.target
  UNIT
  
  sudo systemctl daemon-reload
  sudo systemctl enable --now ai-ssh-audit.timer
  

Check status and logs:

systemctl list-timers | grep ai-ssh-audit
journalctl -u ai-ssh-audit.service --no-pager -n 50

What you’ll get: a short, ranked list of “most unusual” SSH events with timestamps, users, and IPs, saved to JSON and echoed to your logs.

---

4) Turn insights into action (alerting and hardening)

• Pivot quickly: For a flagged IP, check its recent activity:

  sudo journalctl -S -2h -u ssh | grep '1\.2\.3\.4'
  sudo ausearch -k exec_monitor | grep '1\.2\.3\.4' || true
  

• Add targeted bans (pair with fail2ban or a short-term firewall rule):

  sudo ipset create tempban hash:ip timeout 7200
  sudo ipset add tempban 1.2.3.4
  sudo iptables -I INPUT -m set --match-set tempban src -j DROP
  

• Expand coverage: Duplicate the approach for:

  • Sudden new binaries in /usr/local/bin
  • Abnormal sudo invocations (key: sudo_exec)
  • Rare process names from acct/psacct history

• Keep it local and private: Avoid shipping sensitive logs off-host unless policy requires it. The example pipeline runs entirely on your system.

---

A quick, real-world-flavored win

Admins often see waves of “Failed password” attempts—but which ones matter? In practice, the top anomalies tend to be:

• New usernames never seen before (e.g., ops-admin vs. normal svc_ users)

• Logins at rare hours for your environment

• Source IP blocks that don’t match your usual geography

• Method drift (e.g., sudden “Accepted publickey” from a host that has only ever used passwords)

The example detector floats these to the top so you can spend time investigating, not scrolling.

---

Conclusion and next steps (CTA)

You don’t need a massive budget to get value from AI in Linux security auditing. With auditd, persistent journald, a bit of Python, and systemd timers, you can:

• Capture the right data

• Surface the truly unusual

• Act faster on what matters

Next steps:

• Roll this to a staging box or non-critical host first and tune thresholds (features, score cutoffs, timer interval).

• Add features unique to your environment (e.g., known-good subnets, service accounts).

• Extend the same pattern to auditd rule hits and process accounting for deeper coverage.

Have an improvement or a field story? Share your tweaks and findings—let’s make Linux auditing smarter, together.