Posted on
• Artificial Intelligence

Detecting Suspicious Activity Using Bash and Artificial Intelligence

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Detecting Suspicious Activity Using Bash and Artificial Intelligence

If you woke up tomorrow to find your server was probing strange IPs or running a new binary from /tmp, would you know? Most Linux boxes don’t ship with a full-blown EDR. But they do have Bash, logs, and a treasure trove of signals. In this guide, we’ll turn those signals into an AI-powered anomaly detector using simple Bash and a tiny dose of machine learning.

You’ll get a working example that:

  • Collects security-relevant metrics with Bash (no agents required)

  • Learns a baseline from your system

  • Flags unusual behavior in near-real-time

Why Bash + AI is a valid approach

  • Bash is everywhere: It’s guaranteed to work on most Linux servers with minimal dependencies.

  • Logs tell the truth: SSH auth attempts, sudo usage, ports, and processes are a rich dataset.

  • AI is a complement: Rules (like fail2ban) catch the known bad; anomaly detection catches the weird new things you didn’t anticipate.

We’ll build a small, understandable pipeline that’s easy to tune and extend. No black boxes or heavyweight frameworks.

What you’ll build

  • A Bash collector that, every few minutes, records:

    • Failed SSH attempts and unique source IPs
    • Sudo command attempts
    • New listening ports
    • Root-owned processes not in a whitelist
    • New outbound ports

  • A Python anomaly detector (Isolation Forest) that learns your “normal”

  • An alert path (syslog/console) when something looks suspicious

Prerequisites and installation

We’ll need:

  • Python 3 and pip

  • iproute2 (for ss)

  • Optional: inotify-tools (for file monitoring extensions), jq (for JSON output), git (to sync configs)

Use your distro’s package manager:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip iproute2 inotify-tools jq git

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y python3 python3-pip python3-virtualenv iproute inotify-tools jq git

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-pip python3-virtualenv iproute2 inotify-tools jq git

Create a project directory and a Python environment, then install ML deps:

mkdir -p ~/bash-ai-detector && cd ~/bash-ai-detector

# Python virtual environment (apt systems may require python3-venv; others can use built-in venv)
python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
pip install scikit-learn pandas numpy joblib

Note:

  • Reading complete logs may require root or membership in the “adm” group. Run the collector with sudo if you’re not seeing expected events.

Step 1: Generate baselines (known-good state)

We’ll maintain a few “known” lists:

  • Known listening ports

  • Whitelisted root-owned process names

  • Known outbound destination ports

These help the model focus on changes.

Initialize them with your current reality:

mkdir -p ~/.local/share/bash-ai-detector
ss -lntH | awk '{split($4,a,":"); print a[length(a)]}' | sort -n | uniq > ~/.local/share/bash-ai-detector/known_listening_ports.txt

ps -eo user,comm --no-headers | awk '$1=="root"{print $2}' | sort -u > ~/.local/share/bash-ai-detector/root_proc_whitelist.txt

ss -ntH state established | awk '{split($5,a,":"); print a[length(a)]}' | sort -n | uniq > ~/.local/share/bash-ai-detector/known_outbound_ports.txt

Review and edit these files to reflect your environment (e.g., allowlist known daemons/bind ports).

Step 2: The Bash collector

Save this as collector.sh and make it executable (chmod +x collector.sh). It appends features to a CSV that the model will learn from and score.

#!/usr/bin/env bash
set -euo pipefail

# Configuration
WINDOW_MIN="${WINDOW_MIN:-5}"   # sliding window size
SINCE_OPT="-$WINDOW_MIN min"
DATA_DIR="${DATA_DIR:-$HOME/.local/share/bash-ai-detector}"
LOG_DIR="${LOG_DIR:-$DATA_DIR}"
METRICS_CSV="$DATA_DIR/metrics.csv"

KNOWN_LISTEN="$DATA_DIR/known_listening_ports.txt"
ROOT_WHITELIST="$DATA_DIR/root_proc_whitelist.txt"
KNOWN_OUTBOUND="$DATA_DIR/known_outbound_ports.txt"

mkdir -p "$DATA_DIR"

# Ensure baselines exist
init_baselines() {
  [[ -f "$KNOWN_LISTEN" ]] || ss -lntH | awk '{split($4,a,":"); print a[length(a)]}' | sort -n | uniq > "$KNOWN_LISTEN"
  [[ -f "$ROOT_WHITELIST" ]] || ps -eo user,comm --no-headers | awk '$1=="root"{print $2}' | sort -u > "$ROOT_WHITELIST"
  [[ -f "$KNOWN_OUTBOUND" ]] || ss -ntH state established | awk '{split($5,a,":"); print a[length(a)]}' | sort -n | uniq > "$KNOWN_OUTBOUND"
}

journal_failed_ssh() {
  # Count failed SSH logins (systemd-journald is widely available)
  # Check both ssh and sshd units, aggregate output.
  journalctl -q -S "$SINCE_OPT" -u ssh -u sshd 2>/dev/null | grep -c "Failed password" || true
}

journal_failed_ssh_unique_ips() {
  journalctl -q -S "$SINCE_OPT" -u ssh -u sshd 2>/dev/null \
    | awk '/Failed password/ {for (i=1;i<=NF;i++) if ($i=="from") print $(i+1)}' \
    | sort -u | wc -l || true
}

journal_sudo_cmds() {
  # Count sudo command events
  # Try facility tag first, then fallback
  local c=0
  c=$(journalctl -q -S "$SINCE_OPT" -t sudo 2>/dev/null | grep -c "COMMAND=" || true)
  if [[ "$c" -eq 0 ]]; then
    c=$(journalctl -q -S "$SINCE_OPT" 2>/dev/null | grep -c "sudo:.*COMMAND=" || true)
  fi
  echo "$c"
}

new_listening_ports() {
  tmp_cur=$(mktemp)
  ss -lntH | awk '{split($4,a,":"); print a[length(a)]}' | sort -n | uniq > "$tmp_cur"
  # ports in current but not in known
  comm -13 "$KNOWN_LISTEN" "$tmp_cur" | wc -l
  rm -f "$tmp_cur"
}

unknown_root_procs() {
  tmp_cur=$(mktemp)
  ps -eo user,comm --no-headers | awk '$1=="root"{print $2}' | sort -u > "$tmp_cur"
  comm -13 "$ROOT_WHITELIST" "$tmp_cur" | wc -l
  rm -f "$tmp_cur"
}

new_outbound_ports() {
  tmp_cur=$(mktemp)
  ss -ntH state established | awk '{split($5,a,":"); print a[length(a)]}' | sort -n | uniq > "$tmp_cur"
  comm -13 "$KNOWN_OUTBOUND" "$tmp_cur" | wc -l
  rm -f "$tmp_cur"
}

write_header_if_needed() {
  if [[ ! -f "$METRICS_CSV" ]]; then
    echo "timestamp,failed_ssh,failed_ssh_unique_ips,sudo_cmds,new_listening_ports,unknown_root_procs,new_outbound_ports" > "$METRICS_CSV"
  fi
}

main() {
  init_baselines
  write_header_if_needed

  # If journald is unavailable, warn (fallbacks would need custom parsing by distro)
  if ! command -v journalctl >/dev/null 2>&1; then
    echo "Warning: journalctl not found; SSH/sudo counts may be incomplete without custom log parsing." >&2
  fi

  local ts failed uniq_ips sudo_cmds nlisten nroot nout
  ts=$(date -Iseconds)
  failed=$(journal_failed_ssh)
  uniq_ips=$(journal_failed_ssh_unique_ips)
  sudo_cmds=$(journal_sudo_cmds)
  nlisten=$(new_listening_ports)
  nroot=$(unknown_root_procs)
  nout=$(new_outbound_ports)

  echo "$ts,$failed,$uniq_ips,$sudo_cmds,$nlisten,$nroot,$nout" >> "$METRICS_CSV"

  # Optional: log a summary
  logger -t bash-ai-detector "Collected: failed=$failed uniq_ips=$uniq_ips sudo=$sudo_cmds new_listen=$nlisten unknown_root=$nroot new_out=$nout"
}

if [[ "${1:-}" == "--init" ]]; then
  init_baselines
  write_header_if_needed
  echo "Initialized baselines and metrics file at $DATA_DIR"
  exit 0
fi

main

Tips:

  • Run once with ./collector.sh --init to seed baseline files and CSV header.

  • Schedule with cron to collect every 5 minutes:

    crontab -e

    Add:

    */5 * * * * /bin/bash -lc 'source ~/bash-ai-detector/.venv/bin/activate && ~/bash-ai-detector/collector.sh' >> ~/.local/share/bash-ai-detector/collector.log 2>&1

On systems without journald access for your user, run via root’s crontab (sudo crontab -e) or add your user to the adm group.

Step 3: Train the anomaly detection model

After you’ve collected a reasonable baseline (e.g., a few hours to a day of normal operation), train an Isolation Forest.

Save as train_anomaly.py:

#!/usr/bin/env python3
import argparse
import os
import joblib
import pandas as pd
from sklearn.ensemble import IsolationForest

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--data", default=os.path.expanduser("~/.local/share/bash-ai-detector/metrics.csv"))
    ap.add_argument("--model", default=os.path.expanduser("~/.local/share/bash-ai-detector/model.pkl"))
    ap.add_argument("--contamination", type=float, default=0.01, help="Estimated fraction of anomalies")
    ap.add_argument("--train_rows", type=int, default=0, help="Use first N rows for training; 0 = all")
    args = ap.parse_args()

    df = pd.read_csv(args.data)
    if len(df) < 20:
        raise SystemExit("Not enough data to train (need at least ~20 rows).")

    features = ["failed_ssh","failed_ssh_unique_ips","sudo_cmds","new_listening_ports","unknown_root_procs","new_outbound_ports"]
    X = df[features]

    if args.train_rows and args.train_rows < len(X):
        X = X.iloc[:args.train_rows]

    model = IsolationForest(
        n_estimators=200,
        contamination=args.contamination,
        random_state=42
    )
    model.fit(X)

    joblib.dump({"model": model, "features": features}, args.model)
    print(f"Saved model to {args.model}, trained on {len(X)} rows.")

if __name__ == "__main__":
    main()

Train it (from your venv):

python train_anomaly.py --contamination 0.02

Tuning note:

  • contamination ~0.005–0.02 works for most servers. Lower = fewer alerts, higher = more sensitive.

Step 4: Score new data and alert

Save as score_anomaly.py:

#!/usr/bin/env python3
import argparse
import os
import sys
import json
import joblib
import pandas as pd

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--data", default=os.path.expanduser("~/.local/share/bash-ai-detector/metrics.csv"))
    ap.add_argument("--model", default=os.path.expanduser("~/.local/share/bash-ai-detector/model.pkl"))
    ap.add_argument("--threshold", type=float, default=-0.1, help="Decision function threshold; lower is more anomalous")
    args = ap.parse_args()

    pack = joblib.load(args.model)
    model = pack["model"]
    features = pack["features"]

    df = pd.read_csv(args.data)
    last = df.tail(1).copy()

    score = float(model.decision_function(last[features])[0])   # higher is more normal
    is_anom = score < args.threshold

    out = {
        "timestamp": last["timestamp"].values[0],
        "score": round(score, 4),
        "threshold": args.threshold,
        "failed_ssh": int(last["failed_ssh"].values[0]),
        "failed_ssh_unique_ips": int(last["failed_ssh_unique_ips"].values[0]),
        "sudo_cmds": int(last["sudo_cmds"].values[0]),
        "new_listening_ports": int(last["new_listening_ports"].values[0]),
        "unknown_root_procs": int(last["unknown_root_procs"].values[0]),
        "new_outbound_ports": int(last["new_outbound_ports"].values[0]),
        "anomalous": bool(is_anom)
    }
    print(json.dumps(out))
    sys.exit(2 if is_anom else 0)

if __name__ == "__main__":
    main()

Wire scoring into your collection run (append this to the end of collector.sh or add as a separate cron entry right after collection). Example as a separate cron line that runs after collector:

*/5 * * * * /bin/bash -lc 'source ~/bash-ai-detector/.venv/bin/activate && python ~/bash-ai-detector/score_anomaly.py' \
  | tee -a ~/.local/share/bash-ai-detector/alerts.jsonl \
  | jq -c 'select(.anomalous==true)' \
  | while read -r line; do logger -t bash-ai-detector "ALERT: $line"; done

  • If you don’t have jq installed, remove the jq line and handle parsing yourself.

  • You can also send alerts to email/Slack/Webhooks in that while-loop.

3–5 actionable ideas and real-world examples

1) Start with a tight baseline, then loosen

  • Let the model learn from a “clean” period (e.g., during business hours) before exposing it to unusual workloads.

  • Tune --contamination and --threshold if you get too many/few alerts.

2) Alert on “new” rather than “any”

  • The diffs against known lists (ports, root processes, outbound ports) cut noise dramatically.

  • Example: A new listener on port 45219 by a binary in /tmp should be loud. Investigate with:

    ss -lptn 'sport = :45219'
ls -l /proc/$(pgrep -f your-binary)/exe
sha256sum /proc/$(pgrep -f your-binary)/exe

3) Spikes in SSH failures and unique IPs

  • Dictionary attacks show up as rising failed_ssh and failed_ssh_unique_ips. You’ll see the anomaly even before an account is brute-forced.

  • Follow-up actions:

    • Geo-block or rate-limit at the firewall
    • Tighten SSH: disable password logins, move to keys, consider port-knocking or TOTP
    • Consider adding fail2ban as a rule-based companion

4) Unusual sudo activity outside norms

  • Sudden rises in sudo_cmds during odd hours can indicate account misuse.

  • Cross-check users and commands:

    journalctl -S "-1 hour" -t sudo | sed -n 's/.*COMMAND=//p'

5) Keep your allowlists current

  • When you install a new service:

    • Add its listening port to known_listening_ports.txt
    • Add the daemon name to root_proc_whitelist.txt
    • If it makes outbound connections (e.g., a proxy), add expected ports to known_outbound_ports.txt

  • This reduces false positives and keeps your model focused.

Optional extensions

  • File integrity monitoring: Use inotify-tools to watch for executable files appearing in /tmp or changes in /etc:

    inotifywait -m -e create,modify,attrib,move,delete /etc /tmp

    Feed counts of suspicious events as new features in collector.sh.

  • Per-process features: Extend the collector to hash unknown executables, record parent/child relationships, and count networked processes.

  • Multi-host: rsync or ship the CSV to a central box for training/scoring, then broadcast model updates back out.

Troubleshooting

  • No SSH/sudo events appear:

    • Ensure journalctl is present and the cron job runs with sufficient permissions.
    • RHEL/SUSE log files: /var/log/secure; Debian/Ubuntu: /var/log/auth.log. You can add file-based parsing if journald is not used.

  • Too many anomalies:

    • Increase --threshold toward 0.0 (less sensitive).
    • Decrease --contamination (model expects fewer anomalies).
    • Update the allowlists.

  • Too quiet:

    • Lower --threshold (e.g., -0.2).
    • Increase --contamination (e.g., 0.03).

Conclusion and next steps (CTA)

You now have a lightweight, transparent, and extendable anomaly detector built from the Linux primitives you already trust—Bash, logs, and a small ML model. It won’t replace a full security stack, but it can dramatically shorten your time-to-signal when something unusual happens.

Next steps:

  • Deploy collector and scorer via cron on your servers.

  • Let the model learn your baseline, then tune thresholds.

  • Extend features for your environment (file changes, DNS anomalies, process lineage).

  • Combine with rule-based tools (fail2ban, auditd) for defense-in-depth.

If you found this useful, try it on a non-critical host today. Share your tuned features and lessons learned—others will benefit from your patterns!

Related posts