Artificial Intelligence SSH Security: Let Your Server Learn to Fight Back

Bots hammer SSH 24/7. They’re faster than ever, they rotate IPs, and they love “low-and-slow” attempts that sneak under static rules. If your logs are full of “Failed password” messages, you’re not alone. The good news: you don’t need a SOC or a data lake to get smarter. In this guide, you’ll wire up a tiny machine-learning assist to your Linux box with a few lines of Bash and Python—so your server adapts to attack patterns in real time.

What you’ll get:

• Clear reasons why AI-driven SSH defense is worth it (and when it’s not).

• A practical, copy-paste setup using systemd, journalctl, nftables, and a lightweight ML model.

• 3–5 actionable steps to harden, detect, and auto-block attackers—with install commands for apt, dnf, and zypper.

---

Why AI for SSH Defense?

• Attackers evolve. Static thresholds (e.g., ban after N failures) miss slow-distributed attacks. Simple anomaly detection learns what “normal” events look like for your host and flags the weird stuff—like a sudden flurry, username spraying, or unusual timing.

• Defense in depth. AI-scoring complements classics like key-based auth and fail2ban. Keep your basics; add learning on top.

• Practical and local. You don’t need to stream logs to the cloud. A small local model (IsolationForest) can score events in milliseconds.

Caveats:

• Start conservative to avoid false positives.

• Keep an allowlist (office VPN, CI runners).

• Monitor early results; tune thresholds gradually.

---

Step 1: Baseline SSH Hardening (Do This First)

1) Disable password logins and require keys:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?ChallengeResponseAuthentication.*/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?KbdInteractiveAuthentication.*/KbdInteractiveAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart sshd || sudo systemctl restart ssh

2) Tighten auth parameters:

sudo bash -c 'cat >> /etc/ssh/sshd_config' <<'EOF'
MaxAuthTries 3
LoginGraceTime 20
MaxStartups 10:30:60
EOF
sudo systemctl restart sshd || sudo systemctl restart ssh

3) Optional but helpful: move SSH off 22 (doesn’t replace security, but reduces noise).

sudo sed -i 's/^#\?Port .*/Port 2222/' /etc/ssh/sshd_config
sudo systemctl restart sshd || sudo systemctl restart ssh

---

Step 2: Install the Pieces (apt, dnf, zypper)

We’ll use:

• systemd-journald for logs

• nftables for fast, time-limited blocking

• Python 3 + pip for a tiny ML model

• Optional: fail2ban for classic pattern bans

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip nftables fail2ban
# OpenSSH server (if needed):
sudo apt install -y openssh-server

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y python3 python3-pip nftables fail2ban
# OpenSSH server (if needed):
sudo dnf install -y openssh-server

openSUSE Leap/Tumbleweed or SLE (zypper):

sudo zypper refresh
sudo zypper install -y python3 python3-pip nftables fail2ban
# OpenSSH server (if needed; package name is "openssh" on SUSE):
sudo zypper install -y openssh

Enable services if they’re not already on:

sudo systemctl enable --now nftables
sudo systemctl enable --now sshd || sudo systemctl enable --now ssh
# Optional:
sudo systemctl enable --now fail2ban

---

Step 3: Create an AI-Assisted SSH Guard

We’ll build a small pipeline:

• Follow ssh/sshd logs from journalctl.

• Extract failed login events and compute simple features per IP.

• Score with a small ML model and heuristics.

• Block bad IPs quickly using nftables with timeouts.

Directory layout:

/opt/ssh-ai/
  venv/                 # Python virtualenv
  score.py              # ML + heuristic scorer
  ssh-ai-guard.sh       # Bash watcher and auto-block
  allowlist.txt         # IPs never to ban (one per line)

Set up environment and Python deps:

sudo mkdir -p /opt/ssh-ai
sudo python3 -m venv /opt/ssh-ai/venv
sudo /opt/ssh-ai/venv/bin/pip install --upgrade pip
sudo /opt/ssh-ai/venv/bin/pip install scikit-learn
sudo touch /opt/ssh-ai/allowlist.txt
sudo chmod 640 /opt/ssh-ai/allowlist.txt

nftables rule bootstrap (auto-created by our script too)

We’ll ensure at runtime that a deny set and input rule exist. No need to edit /etc/nftables.conf for this demo.

The ML scorer: /opt/ssh-ai/score.py

• Uses a conservative heuristic first.

• Falls back to IsolationForest when it has enough observations.

#!/usr/bin/env python3
import os, sys, time, json, math
from pathlib import Path

# Lazy import only if available
try:
    from sklearn.ensemble import IsolationForest
    HAVE_SK = True
except Exception:
    HAVE_SK = False

BASE = Path("/opt/ssh-ai")
OBS = BASE / "observations.csv"
MODEL = BASE / "model.joblib"

def safe_float(x, default=0.0):
    try: return float(x)
    except: return default

def load_data():
    rows = []
    if OBS.exists():
        with OBS.open() as f:
            for line in f:
                parts = line.strip().split(",")
                if len(parts) != 6: continue
                # ts, ip, cnt60, cnt300, users60, avg_gap, hour
                rows.append([float(parts[2]), float(parts[3]), float(parts[4]), float(parts[5]), float(parts[6]) if len(parts)>6 else 0.0])
    return rows

def save_obs(ip, cnt60, cnt300, users60, avg_gap, hour):
    ts = time.time()
    with OBS.open("a") as f:
        f.write(f"{ts},{ip},{cnt60},{cnt300},{users60},{avg_gap},{hour}\n")

def heuristic(cnt60, cnt300, users60, avg_gap):
    # Conservative thresholds you can tune:
    # Quick burst
    if cnt60 >= 5: return True, "heuristic_cnt60>=5"
    # Sustained probing
    if cnt300 >= 12: return True, "heuristic_cnt300>=12"
    # Username spray
    if users60 >= 3: return True, "heuristic_users60>=3"
    # Very tight spacing implies automation
    if cnt60 >= 3 and avg_gap > 0 and avg_gap < 5: return True, "heuristic_gap<5s"
    return False, None

def ml_predict(sample, rows):
    if not HAVE_SK or len(rows) < 200:
        return 0, "ml_unavailable_or_insufficient_data"
    # Train simple IsolationForest (quick and dirty; retrains per call)
    try:
        from joblib import dump
        X = rows[-2000:]  # recent window
        iso = IsolationForest(n_estimators=100, contamination=0.03, random_state=42)
        iso.fit(X)
        pred = iso.predict([sample])[0]  # -1 anomaly, 1 normal
        score = iso.decision_function([sample])[0]  # lower = more anomalous
        reason = f"iso_pred={pred},score={score:.4f}"
        return (-1 if pred == -1 else 1), reason
    except Exception as e:
        return 0, f"ml_error:{e}"

def main():
    if len(sys.argv) < 7:
        print("PASS insufficient_args")
        sys.exit(0)
    ip = sys.argv[1]
    cnt60 = safe_float(sys.argv[2])
    cnt300 = safe_float(sys.argv[3])
    users60 = safe_float(sys.argv[4])
    avg_gap = safe_float(sys.argv[5])
    hour = safe_float(sys.argv[6])

    # Log observation
    save_obs(ip, cnt60, cnt300, users60, avg_gap, hour)

    # Heuristic first
    do_ban, why = heuristic(cnt60, cnt300, users60, avg_gap)
    if do_ban:
        print(f"BAN {why}")
        return

    # ML second
    rows = load_data()
    pred, why_ml = ml_predict([cnt60, cnt300, users60, avg_gap, hour], rows)
    if pred == -1:
        print(f"BAN {why_ml}")
    else:
        print(f"PASS {why_ml}")

if __name__ == "__main__":
    main()

sudo tee /opt/ssh-ai/score.py >/dev/null <<'PY'
[PASTE THE CONTENTS ABOVE]
PY
sudo chmod +x /opt/ssh-ai/score.py

The watcher + auto-blocker: /opt/ssh-ai/ssh-ai-guard.sh

• Follows ssh/sshd logs in real time.

• Computes features per IP over sliding windows.

• Calls the scorer; bans via nftables set with timeout.

#!/usr/bin/env bash
set -euo pipefail

VENV=/opt/ssh-ai/venv
SCORER=/opt/ssh-ai/score.py
STATE_DIR=/run/ssh-ai
EVENTS_FILE="$STATE_DIR/events.log"
ALLOWLIST=/opt/ssh-ai/allowlist.txt
BAN_TIMEOUT="2h"    # adjust as you like
WINDOW_SHORT=60
WINDOW_LONG=300

mkdir -p "$STATE_DIR"
touch "$EVENTS_FILE"

# Ensure nftables building blocks exist (idempotent)
ensure_nft() {
  sudo nft list table inet filter >/dev/null 2>&1 || sudo nft add table inet filter
  sudo nft list chain inet filter input >/dev/null 2>&1 || sudo nft add chain inet filter input '{ type filter hook input priority 0; }'
  sudo nft list set inet filter ssh_deny >/dev/null 2>&1 || sudo nft 'add set inet filter ssh_deny { type ipv4_addr; flags timeout; }'
  # Drop rule referencing the set (insert once)
  if ! sudo nft list chain inet filter input | grep -q 'ip saddr @ssh_deny drop'; then
    sudo nft insert rule inet filter input tcp dport 22 ip saddr @ssh_deny drop
  fi
}
ensure_nft

is_allowed() {
  local ip="$1"
  [[ -f "$ALLOWLIST" ]] && grep -qE "^${ip}$" "$ALLOWLIST"
}

prune_old() {
  local now epoch min_epoch
  now=$(date +%s)
  min_epoch=$(( now - WINDOW_LONG - 5 ))
  awk -v m="$min_epoch" '$1 >= m' "$EVENTS_FILE" > "${EVENTS_FILE}.new" || true
  mv "${EVENTS_FILE}.new" "$EVENTS_FILE"
}

features_for_ip() {
  local ip="$1"
  local now=$(date +%s)
  local short_from=$(( now - WINDOW_SHORT ))
  local long_from=$(( now - WINDOW_LONG ))

  local cnt60 cnt300 users60 avg_gap
  cnt60=$(awk -v ip="$ip" -v s="$short_from" '$1>=s && $2==ip {c++} END{print c+0}' "$EVENTS_FILE")
  cnt300=$(awk -v ip="$ip" -v s="$long_from"  '$1>=s && $2==ip {c++} END{print c+0}' "$EVENTS_FILE")
  users60=$(awk -v ip="$ip" -v s="$short_from" '$1>=s && $2==ip {u[$3]=1} END{print length(u)}' "$EVENTS_FILE")

  # Average inter-arrival gap over long window (rough)
  local gaps sum n prev t
  sum=0; n=0; prev=""
  while read -r t; do
    if [[ -n "$prev" ]]; then
      gaps=$(( t - prev ))
      if (( gaps >= 0 )); then sum=$(( sum + gaps )); n=$(( n + 1 )); fi
    fi
    prev="$t"
  done < <(awk -v ip="$ip" -v s="$long_from" '$1>=s && $2==ip {print $1}' "$EVENTS_FILE" | sort -n)
  if (( n > 0 )); then
    avg_gap=$(awk -v s="$sum" -v n="$n" 'BEGIN{printf("%.3f", s/n)}')
  else
    avg_gap="0"
  fi

  echo "$cnt60" "$cnt300" "$users60" "$avg_gap"
}

ban_ip() {
  local ip="$1"
  sudo nft add element inet filter ssh_deny { $ip timeout $BAN_TIMEOUT; } || true
  logger -t ssh-ai "Banned $ip for $BAN_TIMEOUT"
}

# Follow journald for ssh/sshd; use short-unix to get epoch timestamp
journalctl -o short-unix -u ssh -u sshd -f | \
while IFS= read -r line; do
  # Keep only failed attempts or invalid users
  [[ "$line" =~ Failed\ password|Invalid\ user ]] || continue

  # Extract epoch (first token, may be SEC.USEC)
  epoch_raw="${line%% *}"
  epoch="${epoch_raw%%.*}"

  # Extract IPv4 (demo focuses on IPv4; extend for IPv6 if needed)
  ip=$(echo "$line" | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | head -n1)
  [[ -n "${ip:-}" ]] || continue

  # Extract username heuristic
  user=$(echo "$line" | sed -nE 's/.*(Failed password for (invalid user )?|Invalid user )([^ ]+).*/\3/p')
  [[ -n "${user:-}" ]] || user="-"

  # Allowlist check
  if is_allowed "$ip"; then
    continue
  fi

  # Append event and prune
  echo "$epoch $ip $user" >> "$EVENTS_FILE"
  prune_old

  # Compute features and score
  read -r f60 f300 u60 gap < <(features_for_ip "$ip")
  hour=$(date +%H)
  result=$("$VENV/bin/python3" "$SCORER" "$ip" "$f60" "$f300" "$u60" "$gap" "$hour" || echo "PASS scorer_error")

  action=${result%% *}
  reason=${result#* }

  if [[ "$action" == "BAN" ]]; then
    ban_ip "$ip"
  fi
done

sudo tee /opt/ssh-ai/ssh-ai-guard.sh >/dev/null <<'SH'
[PASTE THE CONTENTS ABOVE]
SH
sudo chmod +x /opt/ssh-ai/ssh-ai-guard.sh

Run it as a systemd service

sudo tee /etc/systemd/system/ssh-ai-guard.service >/dev/null <<'UNIT'
[Unit]
Description=AI-assisted SSH Guard (log watcher + nftables)
After=network-online.target systemd-journald.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/opt/ssh-ai/ssh-ai-guard.sh
Restart=always
RestartSec=2s
User=root
AmbientCapabilities=CAP_NET_ADMIN
CapabilityBoundingSet=CAP_NET_ADMIN
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now ssh-ai-guard.service

Tip: If your distro uses firewalld on top of nftables, this set-based drop rule still works, because it hooks in the inet filter/input chain. If you have custom nftables rules, ensure no later “accept” shadows this drop.

---

Step 4: Real-World Tuning and Examples

• Allowlisting:

  • Put trusted IPs or CIDRs (expanded to IPs) into /opt/ssh-ai/allowlist.txt (one per line). The guard skips those entirely.

• Test locally:

  • From another host, attempt a few failed logins to simulate bursts:

  ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@your.server
  

  • Watch bans appear:

  sudo watch -n1 'nft list set inet filter ssh_deny'
  sudo journalctl -u ssh-ai-guard -f
  

• See fewer “low-and-slow” penetrations:

  • Username spraying (admin, test, deploy, ubuntu) across a few minutes often evades naive limits. The users60 feature plus ML scoring catches this pattern even when attempts are spaced out.

• Persisting across reboots:

  • The service re-creates the nftables set/rule on startup and repopulates bans as new events come in. If you want static persistence of the table + rule at boot (before the service starts), you can add an include to /etc/nftables.conf, but for many hosts the dynamic approach is fine.

• Combine with fail2ban (optional):

  • Let fail2ban handle classic fast bursts with well-tested filters, and keep the AI guard for subtle anomalies.
  • Installation:
  • apt: sudo apt install -y fail2ban
  • dnf: sudo dnf install -y fail2ban
  • zypper: sudo zypper install -y fail2ban
  • Then sudo systemctl enable --now fail2ban and tune /etc/fail2ban/jail.local.

---

Step 5: Safety, Observability, Next Steps

• Start strict basics first (keys only). Then turn on the AI guard with a higher threshold (e.g., raise heuristic limits) and monitor logs.

• Log reasons:

  • The scorer outputs why it banned an IP (e.g., heuristic_cnt60>=5, iso_pred=-1). You can extend ssh-ai-guard.sh to write CSV or send metrics to Prometheus/ELK.

• Extend to IPv6:

  • Add a second set for ipv6_addr and a parallel rule:

  sudo nft list set inet filter ssh_deny6 >/dev/null 2>&1 || sudo nft 'add set inet filter ssh_deny6 { type ipv6_addr; flags timeout; }'
  sudo nft insert rule inet filter input tcp dport 22 ip6 saddr @ssh_deny6 drop
  

  • Update ban_ip() to add to ssh_deny6 for IPv6 addresses.

• Smarter features:

  • Country or ASN (requires geoip), ratio of successes to attempts, diurnal patterns (hour-based baselines), and per-user anomaly scores.

---

Conclusion and Call to Action

Static SSH defenses stop the obvious stuff; learning-based detection catches the rest. With a few commands and small scripts, your server can:

• Observe SSH behavior in real time.

• Learn what’s normal for your host.

• Block abnormal sources automatically with fast, time-limited bans.

Your next steps: 1) Apply Step 1 hardening immediately. 2) Install the dependencies with apt/dnf/zypper. 3) Deploy the AI guard service and watch your ban set grow while brute-force noise drops. 4) Tune thresholds and allowlist, then extend to IPv6 or SIEM as needed.

If this reduced your nightly “Failed password” flood, consider packaging your tweaks into a role or container and sharing it with your team. Smarter SSH starts here—ship it.