Posted on
• Artificial Intelligence

Artificial Intelligence Phishing Detection

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Phishing Detection From Your Linux Terminal

If you’ve ever tried to “grep” your way out of a phishing storm, you know the feeling: attackers iterate faster than your handcrafted rules. Links hide behind URL shorteners, look-alike domains use punycode tricks, and attachments pack payloads into nested archives. In this post, we’ll build a practical, AI-assisted phishing detection pipeline you can run from Bash, combining open threat feeds, lightweight machine intelligence, and proven security tools.

What you’ll get:

  • A Bash-first workflow to score URLs and emails for phishing risk

  • A small Python helper that applies ML-style features and outputs reasons

  • Integration with threat feeds, DKIM/DMARC verification, and malware scanning

  • Copy-pasteable install and run steps for apt, dnf, and zypper

Why it’s worth your time:

  • Phishing adapts too quickly for static rules alone

  • AI-style features catch signals like obfuscation, domain tricks, and risky URL patterns

  • Running locally keeps sensitive content inside your perimeter

  • The pipeline is modular—swap, extend, or automate as you like

Prerequisites: Install the CLI and libraries

We’ll use curl, jq, ripgrep, YARA, ClamAV, OpenDKIM, OpenDMARC, Python 3, and a virtualenv for a tiny Python helper.

On Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq ripgrep yara clamav opendkim-tools opendmarc mpack python3 python3-pip python3-venv git

On Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y curl jq ripgrep yara clamav opendkim-tools opendmarc mpack python3 python3-pip python3-virtualenv git

On openSUSE/SLE (zypper):

sudo zypper install -y curl jq ripgrep yara clamav opendkim-tools opendmarc mpack python3 python3-pip python3-virtualenv git

Initialize ClamAV signatures (optional but recommended):

sudo freshclam

Create a project folder and Python virtual environment:

mkdir -p ~/phish-lab/{bin,data,work}
cd ~/phish-lab
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install tldextract idna

Note: If your distro requires a CA/SSL bundle for Python, ensure it’s installed (e.g., ca-certificates).

Step 1: Pull community phishing feeds and normalize

We’ll aggregate URLs from OpenPhish and URLHaus into a single local blocklist.

Create bin/fetch_feeds.sh:

#!/usr/bin/env bash
set -euo pipefail
ROOT="${ROOT:-$HOME/phish-lab}"
DATA="$ROOT/data"
TMP="$ROOT/work"
mkdir -p "$DATA" "$TMP"

# Feeds: community OpenPhish and URLHaus recent URLs
OPENPHISH_URL="https://openphish.com/feed.txt"
URLHAUS_CSV="https://urlhaus.abuse.ch/downloads/urlhaus_recent.csv"

curl -fsSL "$OPENPHISH_URL" -o "$TMP/openphish.txt"

curl -fsSL "$URLHAUS_CSV" -o "$TMP/urlhaus.csv"
# Extract URLs from URLHaus CSV (skip comments and header)
awk -F, 'BEGIN{OFS=","} !/^#/ && NR>1 {print $2}' "$TMP/urlhaus.csv" \
  | sed 's/^"//;s/"$//' > "$TMP/urlhaus.txt"

# Normalize, de-dupe, and keep http(s) only
cat "$TMP/openphish.txt" "$TMP/urlhaus.txt" \
  | grep -Eoi 'https?://[^ ]+' \
  | sed 's/[[:space:]]//g' \
  | sort -u > "$DATA/blocklist_urls.txt"

echo "Wrote $(wc -l < "$DATA/blocklist_urls.txt") blocklisted URLs to $DATA/blocklist_urls.txt"

Make it executable and run:

chmod +x bin/fetch_feeds.sh
bin/fetch_feeds.sh

Pro tip: Schedule regular refresh with cron or a systemd timer.

Step 2: Add a lightweight AI URL scorer

We’ll create a small Python tool that:

  • Extracts ML-style features from each URL (length, punycode, IP literals, entropy hints, etc.)

  • Applies a logistic-style scoring to estimate phishing probability

  • Explains the score with human-readable reasons

  • Optionally short-circuits to “known-bad” if the URL is on our blocklist

Create bin/url_ai.py:

#!/usr/bin/env python3
import sys, re, json, math, os
from urllib.parse import urlparse, parse_qs
import idna
try:
    import tldextract
except ImportError:
    print("Missing dependency: tldextract. Activate venv and run: pip install tldextract idna", file=sys.stderr)
    sys.exit(2)

ROOT = os.environ.get("ROOT", os.path.expanduser("~/phish-lab"))
BLOCKLIST = os.path.join(ROOT, "data", "blocklist_urls.txt")
blocklisted = set()
if os.path.exists(BLOCKLIST):
    with open(BLOCKLIST, "r", encoding="utf-8", errors="ignore") as f:
        blocklisted = {line.strip() for line in f if line.strip()}

ip_regex = re.compile(r'^(?:\d{1,3}\.){3}\d{1,3}$')
ipv6_regex = re.compile(r'^\[?[0-9a-f:]+\]?$',
                        re.IGNORECASE)

suspicious_tlds = {
    "zip","mov","click","link","top","xyz","gq","tk","ml","cf","work","fit",
    "cam","quest","country","kim","men","loan","download","review","accountants"
}
suspicious_keywords = ["login","verify","update","secure","confirm","password","invoice","webscr","support","helpdesk"]

def is_ip_literal(host):
    if not host: return False
    host = host.strip("[]")
    return bool(ip_regex.match(host) or ipv6_regex.match(host))

def pct_ratio(url):
    total = max(len(url), 1)
    return url.count('%') / total

def extract_features(u):
    reasons = []
    try:
        p = urlparse(u)
    except Exception:
        return None, ["unparseable"], 0.99
    scheme = p.scheme.lower()
    netloc = p.netloc.lower()
    host = netloc.split('@')[-1].split(':')[0]
    path = p.path or ""
    query = p.query or ""
    frag = p.fragment or ""
    full = u.strip()

    ext = tldextract.extract(u)
    domain = ".".join([x for x in [ext.domain, ext.suffix] if x])
    subdomain = ext.subdomain or ""

    features = {}
    features["len_url"] = len(full)
    features["has_at"] = 1 if "@" in netloc or "@" in path else 0
    features["has_ip"] = 1 if is_ip_literal(host) else 0
    features["dots_host"] = host.count(".")
    features["hyphens_host"] = host.count("-")
    features["digits_host"] = sum(ch.isdigit() for ch in host)
    features["pct_ratio"] = pct_ratio(full)
    features["has_punycode"] = 1 if "xn--" in host else 0
    features["path_len"] = len(path)
    features["query_len"] = len(query)
    features["num_params"] = len(parse_qs(query))
    features["frag_len"] = len(frag)
    tld = ext.suffix.lower() if ext.suffix else ""
    features["sus_tld"] = 1 if tld in suspicious_tlds else 0
    kw = sum(1 for k in suspicious_keywords if k in full.lower())
    features["kw_hits"] = kw
    features["sub_len"] = len(subdomain)

    # Reasons
    if features["has_ip"]: reasons.append("URL uses raw IP")
    if features["has_punycode"]: reasons.append("Punycode in hostname")
    if features["sus_tld"]: reasons.append(f"Suspicious TLD: .{tld}")
    if features["kw_hits"] > 0: reasons.append("Sensitive keywords in URL")
    if features["pct_ratio"] > 0.03: reasons.append("High percent-encoding")
    if features["has_at"]: reasons.append("@ present in URL")
    if features["dots_host"] >= 4: reasons.append("Many dots in hostname")
    if features["hyphens_host"] >= 3: reasons.append("Many hyphens in hostname")
    if features["digits_host"] >= 4: reasons.append("Many digits in hostname")
    if features["path_len"] > 60: reasons.append("Long path")
    if features["num_params"] >= 5: reasons.append("Many query parameters")

    return features, reasons, None

# Heuristic weights approximating a logistic regressor
WEIGHTS = {
    "bias": -3.0,
    "len_url": 0.003,       # long URLs are riskier
    "has_at": 1.5,
    "has_ip": 1.8,
    "dots_host": 0.35,
    "hyphens_host": 0.25,
    "digits_host": 0.18,
    "pct_ratio": 40.0,      # percent-encoding spikes are strong signals
    "has_punycode": 1.6,
    "path_len": 0.01,
    "query_len": 0.01,
    "num_params": 0.3,
    "frag_len": 0.006,
    "sus_tld": 1.4,
    "kw_hits": 0.9,
    "sub_len": 0.02
}

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def score(features):
    z = WEIGHTS["bias"]
    for k, w in WEIGHTS.items():
        if k == "bias": continue
        z += w * features.get(k, 0)
    return sigmoid(z)

def main():
    urls = []
    if len(sys.argv) > 1 and sys.argv[1] != "-":
        with open(sys.argv[1], "r", encoding="utf-8", errors="ignore") as f:
            urls = [l.strip() for l in f if l.strip()]
    else:
        urls = [l.strip() for l in sys.stdin if l.strip()]

    for u in urls:
        label = "unknown"
        # Feed short-circuit
        if u in blocklisted:
            out = {"url": u, "score": 99.0, "label": "phishy", "reasons": ["Listed in threat feed"]}
            print(json.dumps(out, ensure_ascii=False))
            continue

        feats, reasons, parse_err_prob = extract_features(u)
        if feats is None:
            out = {"url": u, "score": 95.0 if parse_err_prob else 50.0, "label": "phishy", "reasons": reasons}
            print(json.dumps(out, ensure_ascii=False))
            continue

        p = score(feats)
        pct = round(p * 100, 2)
        if pct >= 60:
            label = "phishy"
        elif pct <= 30:
            label = "likely-benign"
        else:
            label = "uncertain"
        out = {"url": u, "score": pct, "label": label, "reasons": reasons}
        print(json.dumps(out, ensure_ascii=False))

if __name__ == "__main__":
    main()

Make it executable:

chmod +x bin/url_ai.py

Test with a couple of URLs:

printf "%s\n" \
  "http://xn--paypol-6ve.com/security/verify" \
  "https://accounts.google.com/signin" \
  "http://198.51.100.23/login?user=you" \
| bin/url_ai.py | jq .

Step 3: A Bash-first email/URL scanner that ties it together

We’ll build a wrapper that:

  • Accepts either a raw .eml file or a list of URLs

  • Extracts URLs from the message

  • Checks feeds and AI score

  • Optionally validates DKIM/DMARC and scans attachments with ClamAV/YARA

Create bin/phish_scan.sh:

#!/usr/bin/env bash
set -euo pipefail

ROOT="${ROOT:-$HOME/phish-lab}"
BIN="$ROOT/bin"
DATA="$ROOT/data"
WORK="$ROOT/work"
mkdir -p "$WORK"

usage() {
  echo "Usage:"
  echo "  $0 --eml message.eml"
  echo "  $0 --urls urls.txt"
  exit 1
}

mode=""
input=""
while [[ $# -gt 0 ]]; do
  case "$1" in
    --eml) mode="eml"; input="$2"; shift 2;;
    --urls) mode="urls"; input="$2"; shift 2;;
    *) usage;;
  esac
done

[[ -z "$mode" || -z "$input" ]] && usage

extract_urls() {
  # Basic URL regex; for better extraction consider 'urlextract' in Python
  grep -Eo 'https?://[^)"'\'"[:space:]]+' "$1" \
    | sed 's/[>),.;:]*$//' \
    | sort -u
}

scan_urls() {
  local urlfile="$1"
  echo "== URL Scoring =="
  "$BIN/url_ai.py" "$urlfile" \
    | jq -r '. | "\(.label)\t\(.score)\t\(.url)\t\(.reasons|join("; "))"'
}

if [[ "$mode" == "urls" ]]; then
  tmpu="$WORK/urls.$$.txt"
  sort -u "$input" > "$tmpu"
  scan_urls "$tmpu"
  rm -f "$tmpu"
  exit 0
fi

# Mode: EML
EML="$input"
if [[ ! -f "$EML" ]]; then
  echo "Missing file: $EML" >&2; exit 2
fi

echo "== Extracting URLs from $(basename "$EML") =="
tmpurls="$WORK/extracted.$$.txt"
extract_urls "$EML" > "$tmpurls" || true
wc -l "$tmpurls"

if command -v opendkim-testmsg >/dev/null 2>&1; then
  echo "== DKIM verification =="
  # Will exit non-zero on failure; capture status and continue
  if opendkim-testmsg -d < "$EML" >/dev/null 2>&1; then
    echo "DKIM: pass"
  else
    echo "DKIM: fail or none"
  fi
fi

if command -v opendmarc >/dev/null 2>&1 || command -v opendmarc-test >/dev/null 2>&1; then
  echo "== DMARC (basic) =="
  if command -v opendmarc-test >/dev/null 2>&1; then
    if opendmarc-test < "$EML" >/dev/null 2>&1; then
      echo "DMARC: pass"
    else
      echo "DMARC: fail or none"
    fi
  else
    echo "DMARC: tool not available (install opendmarc for deeper checks)"
  fi
fi

# Optionally unpack and scan attachments
ATT_DIR="$WORK/att.$$.d"
mkdir -p "$ATT_DIR"
if command -v munpack >/dev/null 2>&1; then
  echo "== Extracting attachments =="
  (cd "$ATT_DIR" && munpack -q ../"$EML" >/dev/null 2>&1 || true)
  if compgen -G "$ATT_DIR/*" > /dev/null; then
    echo "Attachments extracted to $ATT_DIR"
    if command -v clamscan >/dev/null 2>&1; then
      echo "== ClamAV scan =="
      clamscan -ri "$ATT_DIR" || true
    fi
    if command -v yara >/dev/null 2>&1; then
      echo "== YARA quick scan =="
      # Quick heuristic: detect base64-heavy JS/VBS
      cat > "$WORK/simple_phish_rules.yar" <<'YAR'
rule Suspicious_Encoded_Script {
  strings:
    $a = /[A-Za-z0-9+\/]{200,}={0,2}/
    $b = "FromCharCode" nocase
    $c = "WScript.Shell" nocase
  condition:
    (uint16(0) == 0x5a4d or filesize < 2MB) and 1 of ($a,$b,$c)
}
YAR
      yara -r "$WORK/simple_phish_rules.yar" "$ATT_DIR" || true
    fi
  else
    echo "No attachments extracted (or not MIME-encoded)."
  fi
else
  echo "munpack not found; skip attachment extraction (install mpack)"
fi

if [[ -s "$tmpurls" ]]; then
  scan_urls "$tmpurls"
else
  echo "No URLs found in message."
fi

rm -f "$tmpurls"

Make it executable:

chmod +x bin/phish_scan.sh

Try it on a saved email:

bin/phish_scan.sh --eml suspicious.eml

Or on a URL list:

printf "%s\n" "https://secure-update.example.top/login" "https://example.com/" > /tmp/u.txt
bin/phish_scan.sh --urls /tmp/u.txt

Step 4: Improve results with 3–5 practical tactics

1) Combine feeds with AI scoring

  • Feeds catch known-bad quickly; AI-style scoring flags look-alikes and zero-day variants.

  • Keep feeds fresh:

    bin/fetch_feeds.sh

    Put it in cron (e.g., run hourly).

2) Seed allowlists and cut false positives

  • If you have a known-good domain list (your company, vendors), prefer them.

  • Add a quick allowlist check to bin/url_ai.py (before scoring), or filter in Bash:

    grep -v -F -f data/allow_domains.txt urls.txt | bin/url_ai.py

3) Strengthen feature extraction

  • Expand suspicious_tlds and suspicious_keywords in url_ai.py with what you see in your telemetry.

  • Add features: homograph detection, Levenshtein distance to known brands, URL shortening domains expansion (resolving HEAD Location with curl -I).

  • Example to unshorten:

    curl -sIL https://bit.ly/xyz | awk -F': ' '/^location:/ {print $2}' | tail -n1

4) Verify email authenticity signals

  • DKIM and DMARC checking helps demote obviously spoofed messages.

  • If you run Postfix/Exim, integrate OpenDKIM/OpenDMARC milters server-wide; for offline checks, opendkim-testmsg and opendmarc-test help.

5) Scan attachments proactively

  • clamscan catches many known malware families in attachments.

  • YARA fills the gap with custom rules tailored to your environment.

  • Keep rules small and targeted to reduce noise; start with the high-signal patterns you encounter.

Real-world example pattern: A message with “DocuSign” theme using a .zip attachment and a link to hxxps://docusign-secure-update.xyz/s/doc.html will usually:

  • Fail DMARC or arrive with no DKIM

  • Score high due to suspicious TLD, punycode/IP literals, long encoded URLs

  • Trigger YARA if the extracted HTML/JS is heavily base64-encoded

Step 5: Automate and integrate

  • Run as a procmail or sieve filter: pipe inbound emails through bin/phish_scan.sh, then add headers like X-Phish-Score based on the JSON labels/scores.

  • Use exit codes: treat “phishy” over 80 as quarantine, 60–80 as review, under 30 as pass with logging.

  • Log to JSONL for SIEM ingestion:

    bin/url_ai.py urls.txt >> logs/url_scores.jsonl

Conclusion and Next Steps (CTA)

You now have a Bash-first, AI-assisted phishing detector that:

  • Ingests live threat intelligence

  • Scores URLs with explainable, ML-style features

  • Verifies DKIM/DMARC and scans attachments

  • Works entirely on your Linux box with familiar CLI tools

Where to go from here:

  • Expand features and tuning in bin/url_ai.py to match your environment

  • Integrate with your MTA or ticketing system for automated triage

  • Add a simple web or TUI front end for analysts

  • Build a small labeled corpus of your own “ham vs phish” and replace heuristic weights with a trained model

If you found this useful, wire it into cron today, then iterate weekly based on what your telemetry shows. The best phishing defense is both adaptive and automated—start simple, ship it, and keep improving.

Related posts