Posted on
• Artificial Intelligence

Artificial Intelligence Configuration Drift Detection

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Bash-first AI: Detecting Configuration Drift Before It Bites You

Ever sworn “nobody touched that server” right before a 2 a.m. outage? Configuration drift is the silent culprit: tiny, untracked changes that compound into downtime, security incidents, and compliance gaps. The good news: you don’t need a data center’s worth of tooling to get ahead of it. With a few reliable Linux tools and a small dose of AI, you can baseline, watch, and flag risky drift from the command line.

This guide shows you how to:

  • Capture reproducible baselines of critical system state

  • Automate drift snapshots with Bash

  • Score drift with an anomaly detector (Isolation Forest)

  • Get practical alerts and human-friendly summaries

All commands and code are Bash-first, with minimal Python for the AI bit.

Why this matters (and why AI helps)

  • Not all changes are bad. Patching day looks different from a quiet Sunday. AI helps you learn “normal” change patterns from your own hosts and elevates the weird ones.

  • Compliance and audits love repeatable evidence. Baselines plus versioned, machine-readable diffs reduce audit time and blast radius.

  • Security signal over noise. Most drift is benign; AI-based scoring highlights unusual service enables, package churn, or configuration spikes that deserve human eyes.

What you’ll build

  • A snapshot script that captures:

    • /etc file checksums
    • Enabled services
    • Installed packages
    • Listening ports
    • Firewall rules

  • A feature extractor that compares snapshots and produces metrics

  • A tiny Python script that flags anomalies using Isolation Forest

  • Optional automation via cron or inotify

Prerequisites and installation

We’ll use standard CLI tools plus Python’s scientific stack.

Install required packages:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git jq inotify-tools python3 python3-pip
# Optional if you want cron scheduling:
sudo apt install -y cron
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y git jq inotify-tools python3 python3-pip
# Optional cron (enable/start service):
sudo dnf install -y cronie
sudo systemctl enable --now crond
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y git jq inotify-tools python3 python3-pip
# Optional cron:
sudo zypper install -y cron

Install Python packages (user scope):

python3 -m pip install --user --upgrade pip
python3 -m pip install --user scikit-learn pandas numpy

Make sure ~/.local/bin is in your PATH if needed:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
. ~/.bashrc

1) Build a clean baseline (repeatable snapshots)

Create a directory for your snapshots and a Bash script to capture system state.

Create /usr/local/bin/ai-drift-snapshot.sh:

#!/usr/bin/env bash
set -euo pipefail

SNAPDIR="${SNAPDIR:-/var/lib/ai-drift/snapshots}"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
CUR="$SNAPDIR/$TS"

sudo mkdir -p "$CUR"
sudo chown "$(id -u)":"$(id -g)" "$CUR"

pkg_list() {
  if command -v dpkg-query >/dev/null 2>&1; then
    dpkg-query -W -f='${Package}\t${Version}\n' | sort
  elif command -v rpm >/dev/null 2>&1; then
    rpm -qa --qf '%{NAME}\t%{EPOCHNUM}:%{VERSION}-%{RELEASE}.%{ARCH}\n' | sort
  else
    echo -e "unknown-pkgmgr\t0"
  fi
}

echo "# snapshot: $TS" > "$CUR/meta.txt"
uname -a >> "$CUR/meta.txt" || true
cat /etc/os-release >> "$CUR/meta.txt" 2>/dev/null || true

# Package inventory
pkg_list > "$CUR/packages.tsv"

# Systemd unit files (enabled/disabled/etc.)
if command -v systemctl >/dev/null 2>&1; then
  systemctl list-unit-files --type=service --no-legend --no-pager > "$CUR/systemd-unit-files.txt" || true
fi

# Checksums of /etc (skip some private/cert dirs to reduce noise)
sudo find /etc -xdev -type f \
  -not -path '/etc/ssl/private/*' \
  -not -path '/etc/pki/*' \
  -not -path '/etc/ssh/ssh_host_*' \
  -exec sha256sum {} + | sort > "$CUR/etc.sha256"

# Root crontab (if any)
crontab -l > "$CUR/crontab.root" 2>/dev/null || true

# Listening sockets
ss -tulpen > "$CUR/ports.txt" 2>/dev/null || true

# Firewall rules (iptables/nft)
iptables-save > "$CUR/iptables.save" 2>/dev/null || true
nft list ruleset > "$CUR/nft.txt" 2>/dev/null || true

# Disk overview (helps spot sudden growth)
df -hT > "$CUR/df.txt" 2>/dev/null || true

# Maintain a symlink to "latest"
ln -sfn "$CUR" "$SNAPDIR/latest"

echo "Snapshot saved to: $CUR"

Make it executable and run it:

sudo install -m 0755 ai-drift-snapshot.sh /usr/local/bin/ai-drift-snapshot.sh
sudo /usr/local/bin/ai-drift-snapshot.sh

Commit snapshots to a local Git repo if you want time travel and off-host backups:

sudo bash -c '
  set -e
  SNAPDIR=/var/lib/ai-drift/snapshots
  cd "$SNAPDIR"
  git init -q || true
  git add .
  git commit -m "baseline $(date -u +%F)" || true
'

2) Automate capture (cron and/or inotify)

Option A: periodic snapshots with cron (e.g., every 6 hours):

( crontab -l 2>/dev/null; echo "0 */6 * * * sudo /usr/local/bin/ai-drift-snapshot.sh >> /var/log/ai-drift.log 2>&1" ) | crontab -

Option B: watch /etc for live changes and snapshot after bursts:

#!/usr/bin/env bash
# /usr/local/bin/ai-drift-watch.sh
set -euo pipefail
DEBOUNCE=60
LAST=0
inotifywait -m -r -e modify,create,delete,move --format '%T %w%f %e' --timefmt '%F %T' /etc | while read -r line; do
  NOW=$(date +%s)
  if (( NOW - LAST > DEBOUNCE )); then
    echo "Change in /etc detected: $line"
    sudo /usr/local/bin/ai-drift-snapshot.sh
    LAST="$NOW"
  fi
done

Run it as a service if desired, or in a tmux/screen session:

sudo install -m 0755 ai-drift-watch.sh /usr/local/bin/ai-drift-watch.sh
/usr/local/bin/ai-drift-watch.sh >> /var/log/ai-drift.log 2>&1 &

3) Extract features and score drift with AI

We’ll convert snapshot diffs into simple numeric features, then let an Isolation Forest learn what “normal” looks like for your host.

Create /usr/local/bin/ai-drift-features.sh:

#!/usr/bin/env bash
set -euo pipefail

SNAPDIR="${SNAPDIR:-/var/lib/ai-drift/snapshots}"
OUTCSV="${OUTCSV:-$SNAPDIR/features.csv}"

# Find previous and current snapshots
mapfile -t snaps < <(find "$SNAPDIR" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' 2>/dev/null | sort)
if (( ${#snaps[@]} < 2 )); then
  echo "Need at least two snapshots to compute features." >&2
  exit 1
fi
CUR="${snaps[-1]}"
PREV="${snaps[-2]}"

p="$SNAPDIR/$PREV"
c="$SNAPDIR/$CUR"

# etc checksum diff lines (content changes + additions/removals)
etc_diff=$(diff -U 0 "$p/etc.sha256" "$c/etc.sha256" 2>/dev/null | grep -E '^[+-][0-9a-f]' | wc -l || echo 0)

# Enabled services churn
enabled_prev=$(awk '$2=="enabled"{print $1}' "$p/systemd-unit-files.txt" 2>/dev/null | sort -u || true)
enabled_curr=$(awk '$2=="enabled"{print $1}' "$c/systemd-unit-files.txt" 2>/dev/null | sort -u || true)
tmpd="$(mktemp -d)"
printf '%s\n' $enabled_prev > "$tmpd/p.txt"
printf '%s\n' $enabled_curr > "$tmpd/c.txt"
new_services=$(comm -13 "$tmpd/p.txt" "$tmpd/c.txt" | wc -l || echo 0)
removed_services=$(comm -23 "$tmpd/p.txt" "$tmpd/c.txt" | wc -l || echo 0)

# Package adds/removes (names only)
cut -f1 "$p/packages.tsv" | sort > "$tmpd/pkgs_p.txt"
cut -f1 "$c/packages.tsv" | sort > "$tmpd/pkgs_c.txt"
new_pkgs=$(comm -13 "$tmpd/pkgs_p.txt" "$tmpd/pkgs_c.txt" | wc -l || echo 0)
removed_pkgs=$(comm -23 "$tmpd/pkgs_p.txt" "$tmpd/pkgs_c.txt" | wc -l || echo 0)

# Listening sockets count
ports_prev=$(grep -E 'LISTEN|UNCONN' "$p/ports.txt" 2>/dev/null | wc -l || echo 0)
ports_curr=$(grep -E 'LISTEN|UNCONN' "$c/ports.txt" 2>/dev/null | wc -l || echo 0)
ports_delta=$(( ports_curr - ports_prev ))

# Time features
ts="$CUR"
hour=$(date -u -d "${ts:0:8} ${ts:9:2}:${ts:11:2}:${ts:13:2}Z" +%H 2>/dev/null || date -u +%H)

# Write header if missing
if [ ! -f "$OUTCSV" ]; then
  echo "timestamp,etc_diff,new_services,removed_services,new_pkgs,removed_pkgs,ports_delta,hour" > "$OUTCSV"
fi

echo "$CUR,$etc_diff,$new_services,$removed_services,$new_pkgs,$removed_pkgs,$ports_delta,$hour" >> "$OUTCSV"
rm -rf "$tmpd"

echo "Features appended for $CUR -> $OUTCSV"

Create /usr/local/bin/ai-drift-detect.py:

#!/usr/bin/env python3
import sys
import json
import pandas as pd
from sklearn.ensemble import IsolationForest

if len(sys.argv) < 2:
    print("Usage: ai-drift-detect.py /var/lib/ai-drift/snapshots/features.csv")
    sys.exit(1)

csv_path = sys.argv[1]
df = pd.read_csv(csv_path)

if len(df) < 10:
    # Not enough history; fallback to rule-of-thumb
    latest = df.iloc[-1]
    rough = (latest['etc_diff'] > 10) or (latest['new_services'] > 0) or (latest['new_pkgs'] > 5)
    out = {
        "timestamp": latest['timestamp'],
        "anomaly": bool(rough),
        "reason": "heuristic" if rough else "insufficient_history",
        "etc_diff": int(latest['etc_diff']),
        "new_services": int(latest['new_services']),
        "removed_services": int(latest['removed_services']),
        "new_pkgs": int(latest['new_pkgs']),
        "removed_pkgs": int(latest['removed_pkgs']),
        "ports_delta": int(latest['ports_delta']),
        "hour": int(latest['hour'])
    }
    print(json.dumps(out, indent=2))
    sys.exit(0)

# Train on all but last row
train = df.iloc[:-1, 1:]  # exclude timestamp
test = df.iloc[-1:, 1:]

model = IsolationForest(
    n_estimators=200,
    contamination=0.1,
    random_state=42
)
model.fit(train)
pred = model.predict(test)[0]    # 1 normal, -1 anomaly
score = model.decision_function(test)[0]  # higher is more normal

latest = df.iloc[-1]
out = {
    "timestamp": latest['timestamp'],
    "anomaly": True if pred == -1 else False,
    "score": float(score),
    "etc_diff": int(latest['etc_diff']),
    "new_services": int(latest['new_services']),
    "removed_services": int(latest['removed_services']),
    "new_pkgs": int(latest['new_pkgs']),
    "removed_pkgs": int(latest['removed_pkgs']),
    "ports_delta": int(latest['ports_delta']),
    "hour": int(latest['hour'])
}
print(json.dumps(out, indent=2))

Install and wire it together:

sudo install -m 0755 ai-drift-features.sh /usr/local/bin/ai-drift-features.sh
sudo install -m 0755 ai-drift-detect.py /usr/local/bin/ai-drift-detect.py

Usage after at least two snapshots:

sudo /usr/local/bin/ai-drift-snapshot.sh
sudo /usr/local/bin/ai-drift-snapshot.sh

/usr/local/bin/ai-drift-features.sh
/usr/local/bin/ai-drift-detect.py /var/lib/ai-drift/snapshots/features.csv

Automate post-snapshot processing by appending this to your cron entry:

0 */6 * * * sudo /usr/local/bin/ai-drift-snapshot.sh && /usr/local/bin/ai-drift-features.sh && /usr/local/bin/ai-drift-detect.py /var/lib/ai-drift/snapshots/features.csv >> /var/log/ai-drift.log 2>&1

4) Real-world example: the “harmless” tweak

Scenario:

  • A teammate “temporarily” enables a custom systemd service and opens a new port.

  • A minor hotfix also tweaks multiple /etc files.

Your detector output:

{
  "timestamp": "20250117T031500Z",
  "anomaly": true,
  "score": -0.12,
  "etc_diff": 27,
  "new_services": 1,
  "removed_services": 0,
  "new_pkgs": 0,
  "removed_pkgs": 0,
  "ports_delta": 1,
  "hour": 03
}

Interpretation: unusual config churn outside change windows, a newly enabled service, and a new listening socket. That’s worth investigation before it becomes a 2 a.m. page.

Production tips and hardening

  • Whitelist expected change windows: add a simple guard in ai-drift-detect.py to relax scores during maintenance windows.

  • Git-remote your snapshots: push to a central, append-only repo for tamper-evidence.

  • Tune features: consider adding kernel params (sysctl -a), sudoers diffs, container images, or specific high-signal files (e.g., sshd_config).

  • Reduce noise: exclude cert rotations and known ephemeral files from /etc checksuming.

  • Permissions: snapshots may contain sensitive file paths. Restrict SNAPDIR to root and carefully manage backups.

  • Systemd timers: prefer systemd timers over cron where possible for better visibility and logs.

Conclusion and next steps

Configuration drift isn’t going away—but with a Bash-first workflow and a tiny AI model, you can turn chaos into signal. Start by:

1) Deploying the snapshot script on one staging host today
2) Running for a week to build history
3) Reviewing flagged anomalies and tuning exclusions
4) Rolling out to more hosts and centralizing snapshots

If you found this useful, try extending the feature set or wiring alerts into your chat/issue system. Your future self—sleeping through the night—will thank you.

Related posts