Posted on
• Artificial Intelligence

Artificial Intelligence PAM Audits

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence PAM Audits: Turn Noisy Auth Logs into Actionable Signals

If you’ve ever chased a mysterious lockout, stared at endless auth logs, or wondered if someone quietly tweaked your PAM stack, you know the pain: authentication is both critical and noisy. Linux’s Pluggable Authentication Modules (PAM) gives us flexibility and power—but also more surface area to misconfigure or abuse.

Here’s the good news: you can use a small amount of automation and a dash of AI to make PAM audits smarter, faster, and more reliable—using tools you already trust on Linux plus a lightweight anomaly detector. This guide shows you how to capture the right signals, baseline your PAM stack, and apply AI to flag suspicious behavior before it becomes an incident.

Why “AI for PAM Audits” makes sense

  • Logs are dense and inconsistent. PAM and auth logs vary across distros, services (sshd, sudo, su, display managers), and modules (pam_unix, pam_sss). Humans miss patterns; machines thrive on them.

  • Attacks blend into the noise. Password spraying, lateral movement, and misconfig changes are subtle. Anomaly detection can surface “unusual for this host” without brittle rules.

  • PAM misconfig is risky. A single misplaced pam_permit, disabled pam_faillock, or altered order can break controls. Baselines + change detection + AI triage keep you safe.

What follows is a practical, Linux-first workflow focused on Bash, auditd, journalctl, jq, and a tiny Python script using IsolationForest to flag anomalies.

Prerequisites (install with your package manager)

  • auditd to watch PAM config changes

  • jq to normalize journalctl output

  • Python 3 + pip to run a small anomaly detector

  • pamtester to safely generate test auth events

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y auditd jq python3 python3-pip python3-venv python3-dev build-essential pamtester

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y audit jq python3 python3-pip python3-devel gcc pamtester

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y audit jq python3 python3-pip python3-devel gcc pamtester

OS-specific helpers (optional but handy):

  • Fedora/RHEL authselect:
sudo dnf install -y authselect
  • openSUSE pam-config:
sudo zypper install -y pam-config
  • Debian/Ubuntu pam-auth-update lives in libpam-runtime (usually installed). If needed:
sudo apt install -y libpam-runtime

Step 1: Watch PAM for silent changes (auditd)

Track edits to PAM configuration and related security files. Create a persistent auditd rule:

sudo tee /etc/audit/rules.d/pam.rules >/dev/null <<'EOF'
-w /etc/pam.d/ -p wa -k pam_stack
-w /etc/security/ -p wa -k pam_security
EOF

# Load the new rules
if command -v augenrules >/dev/null; then
  sudo augenrules --load
else
  sudo systemctl restart auditd || sudo service auditd restart
fi

Quick check for any recent PAM edits:

sudo ausearch -k pam_stack -k pam_security --start today --format text

Why this matters: even a well-meaning admin can introduce risk. Change signals + time window make for fast incident triage.

Step 2: Baseline your PAM stack and lock it in

  • Snapshot your current PAM files:
sudo mkdir -p /var/lib/pam-audit
sudo find /etc/pam.d -type f -print0 | sudo xargs -0 sha256sum | sudo tee /var/lib/pam-audit/baseline.sha256
  • Capture a readable inventory:
sudo grep -R "^[^#]" /etc/pam.d | sed 's/\t/ /g' | sudo tee /var/lib/pam-audit/pam-inventory.txt
  • Validate with OS tools:
    • Fedora/RHEL: authselect current
    • openSUSE/SLE: sudo pam-config --list-modules
    • Debian/Ubuntu: sudo pam-auth-update --package

On any deviation (hash mismatches, unexpected module orders, or an inserted pam_permit), you’ll have evidence and context.

Step 3: Normalize PAM/auth events from journald

We’ll extract relevant PAM/auth records from journald into line-delimited JSON (NDJSON). This creates structured input for AI/anomaly analysis.

journalctl --since "24 hours ago" -o json \
| jq -r '
  select(.SYSLOG_IDENTIFIER
    | test("sshd|sudo|su|login|systemd-logind|gdm-password|lightdm|sssd|pam_unix|pam_sss")) 
  | {
      ts: (.__REALTIME_TIMESTAMP|tonumber/1000000|gmtime|strftime("%Y-%m-%dT%H:%M:%SZ")),
      host: .HOSTNAME,
      unit: .SYSLOG_IDENTIFIER,
      pid: ._PID,
      msg: .MESSAGE
    }
  | @json
' | tee /var/lib/pam-audit/pam-events.ndjson

Tip: If your distro logs auth to files (e.g., /var/log/auth.log or /var/log/secure), you can ingest those too. Journald is preferred for consistency and JSON export.

Step 4: Add a lightweight AI anomaly detector

We’ll use a short Python script with IsolationForest to flag “unusual” auth patterns (e.g., many failures from one IP across many users, sudden success-after-fail patterns, late-night spikes). It parses common sshd and pam_unix/pam_sss messages.

Save this as pam_ai_audit.py:

#!/usr/bin/env python3
import json, re, sys, math
from collections import defaultdict, Counter
from datetime import datetime
from sklearn.ensemble import IsolationForest

# Regex patterns for common messages (Debian/Ubuntu/Fedora/EL variants)
RE_SSH_FAIL = re.compile(r'(Failed password for (invalid user )?(?P<user>\S+) from (?P<ip>\d{1,3}(?:\.\d{1,3}){3}))')
RE_SSH_OK   = re.compile(r'(Accepted (password|publickey) for (?P<user>\S+) from (?P<ip>\d{1,3}(?:\.\d{1,3}){3}))')
RE_PAM_FAIL = re.compile(r'authentication failure;.*user=(?P<user>\S*) .*rhost=(?P<ip>\S*)')

def parse_line(rec):
    msg = rec.get("msg","")
    unit = rec.get("unit","")
    ts   = rec.get("ts")
    dt = None
    try:
        dt = datetime.fromisoformat(ts.replace("Z","+00:00"))
    except Exception:
        pass

    user = None; ip = None; action = None; service = None
    m = RE_SSH_FAIL.search(msg)
    if m:
        user = m.group("user")
        ip = m.group("ip")
        action = "fail"
        service = "sshd"
    else:
        m = RE_SSH_OK.search(msg)
        if m:
            user = m.group("user")
            ip = m.group("ip")
            action = "ok"
            service = "sshd"
        else:
            if "sudo" in unit or "su" in unit or "login" in unit or "pam_unix" in unit or "pam_sss" in unit:
                m = RE_PAM_FAIL.search(msg)
                if m:
                    user = (m.group("user") or "").strip() or "unknown"
                    ip = (m.group("ip") or "").strip()
                    ip = ip if ip and ip != "=" else "local"
                    action = "fail"
                    service = "pam"
    if not (user or ip or action):
        return None

    hour = dt.hour if dt else None
    return {
        "user": user or "unknown",
        "ip": ip or "local",
        "action": action,
        "service": service or unit,
        "hour": hour
    }

def vectorize(agg):
    # Features per (ip,user,service) key:
    # [total, fails, oks, fail_ratio, active_hours, unique_days]
    total = agg["total"]
    fails = agg["fails"]
    oks = agg["oks"]
    ratio = fails/total if total else 0
    hours = len(agg["hours"])
    days = len(agg["days"])
    return [total, fails, oks, ratio, hours, days]

def main(path):
    events = []
    with open(path) as f:
        for line in f:
            try:
                rec = json.loads(line)
            except Exception:
                continue
            p = parse_line(rec)
            if p:
                # derive day bucket from timestamp if present
                ts = rec.get("ts")
                day = ts[:10] if ts else "na"
                p["day"] = day
                events.append(p)

    # Aggregate by (ip,user,service)
    buckets = defaultdict(lambda: {"total":0, "fails":0, "oks":0, "hours":set(), "days":set()})
    for e in events:
        key = (e["ip"], e["user"], e["service"])
        b = buckets[key]
        b["total"] += 1
        if e["action"] == "fail":
            b["fails"] += 1
        else:
            b["oks"] += 1
        if e["hour"] is not None:
            b["hours"].add(e["hour"])
        b["days"].add(e["day"])

    if not buckets:
        print("No parsable PAM/auth events found.")
        return

    keys = list(buckets.keys())
    X = [vectorize(buckets[k]) for k in keys]

    # Fit IsolationForest
    clf = IsolationForest(
        n_estimators=200, contamination=0.03, random_state=42
    )
    clf.fit(X)
    scores = clf.score_samples(X)  # more negative = more anomalous

    # Rank anomalies
    ranked = sorted(zip(keys, X, scores), key=lambda t: t[2])
    print("=== Suspicious auth principals (ip,user,service) ===")
    for (ip,user,svc), feats, s in ranked[:20]:
        total, fails, oks, ratio, hours, days = feats
        print(f"- {ip} {user} {svc} | total={total} fails={fails} ok={oks} fail_ratio={ratio:.2f} hours={hours} days={days} score={s:.3f}")

    # Simple heuristics worth highlighting
    print("\n=== Heuristics ===")
    # 1) Many users from one IP (possible password spraying)
    by_ip_users = defaultdict(set)
    for (ip,user,svc) in keys:
        if ip != "local":
            by_ip_users[ip].add(user)
    for ip, users in sorted(by_ip_users.items(), key=lambda x: -len(x[1]))[:10]:
        if len(users) >= 5:
            print(f"* IP {ip} targeted {len(users)} users (possible spraying).")

    # 2) Success after multiple failures (possible brute-force success)
    for (ip,user,svc), feats in zip(keys, X):
        total, fails, oks, ratio, hours, days = feats
        if fails >= 5 and oks >= 1:
            print(f"* {ip} {user} {svc} had {fails} fails then {oks} success (investigate).")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: pam_ai_audit.py /path/to/pam-events.ndjson")
        sys.exit(1)
    main(sys.argv[1])

Install Python deps:

python3 -m pip install --user scikit-learn

Run it:

python3 pam_ai_audit.py /var/lib/pam-audit/pam-events.ndjson

You’ll get:

  • Anomaly-ranked principals (ip, user, service) with quick stats

  • Heuristic flags for password spraying and success-after-fail

Note: This is intentionally lightweight—no agents or daemons—and runs anywhere Python does. You can expand parsers for local conventions.

Step 5: Safely generate test events with pamtester

Use pamtester to create known-good and known-bad attempts and validate your pipeline:

  • Trigger a failed sshd auth (enter a wrong password at prompt):
pamtester sshd "$USER" authenticate
  • Trigger a failed sudo auth:
pamtester sudo "$USER" authenticate

Then re-export events and re-run the AI audit:

journalctl --since "5 minutes ago" -o json | jq -r 'select(.SYSLOG_IDENTIFIER|test("sshd|sudo|pam_unix|pam_sss")) | {ts:(.__REALTIME_TIMESTAMP|tonumber/1000000|gmtime|strftime("%Y-%m-%dT%H:%M:%SZ")),host:.HOSTNAME,unit:.SYSLOG_IDENTIFIER,pid:._PID,msg:.MESSAGE} | @json' \
| tee -a /var/lib/pam-audit/pam-events.ndjson

python3 pam_ai_audit.py /var/lib/pam-audit/pam-events.ndjson

You should see your test entries influence stats and, if abnormal enough, get flagged.

Real-world signals to watch

  • Password spraying: One IP touches many users with failures. Your heuristic catches it; the anomaly score should drop for that IP.

  • Sudden late-night auth bursts: A quiet server suddenly authenticates across many hours; IsolationForest flags the shift.

  • Success after long failure streaks: Potential compromising guess; gets an explicit heuristic flag.

  • PAM stack drift: auditd reveals changes to /etc/pam.d or /etc/security that don’t match your baseline hash set.

Operationalizing it (quick wins)

  • Cron or systemd timer every hour:
0 * * * * root journalctl --since "-65 minutes" -o json | jq -r 'select(.SYSLOG_IDENTIFIER|test("sshd|sudo|su|pam_unix|pam_sss")) | {ts:(.__REALTIME_TIMESTAMP|tonumber/1000000|gmtime|strftime("%Y-%m-%dT%H:%M:%SZ")),host:.HOSTNAME,unit:.SYSLOG_IDENTIFIER,pid:._PID,msg:.MESSAGE} | @json' >> /var/lib/pam-audit/pam-events.ndjson && /usr/bin/python3 /usr/local/bin/pam_ai_audit.py /var/lib/pam-audit/pam-events.ndjson | logger -t pam-ai-audit
  • Alert on change events:
sudo ausearch -k pam_stack -k pam_security --start recent --format short | logger -t pam-audit-change
  • Keep data lean: rotate /var/lib/pam-audit files with logrotate.

Conclusion and next steps

You don’t need a full-blown SIEM to get meaningful, AI-assisted PAM audits. With auditd watching your controls, journald exporting clean signals, and a small IsolationForest pass, you can spot unusual auth behavior and config drift quickly—using standard Linux tooling and Bash-friendly workflows.

Next steps:

  • Tune the regex patterns and features for your environment (e.g., include sssd, Kerberos, or VPN gateways).

  • Wire the output into your alerting path (email, Slack, syslog to a central collector).

  • Extend the model: add per-host baselines, holiday/weekend profiles, or service-specific detectors.

If you’d like a follow-up, I can share:

  • A systemd unit + timer for hands-free hourly audits

  • A richer parser covering more PAM modules

  • A dashboard-ready CSV summary for SIEM ingestion

Stay ahead of auth noise—let AI do the sifting, and keep your PAM stack trustworthy.

Related posts