Posted on
• Artificial Intelligence

Artificial Intelligence Log Analysis for Security

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Log Analysis for Security (with Bash-Friendly Workflows)

Attackers don’t knock—they rattle your doors until one opens. Your logs already record the story. The problem is volume: thousands to millions of lines per day, per host. Artificial Intelligence (AI) can turn that torrent into timely, actionable insights by finding patterns and anomalies you’d miss by eye or with static rules.

In this article you’ll:

  • Understand why AI belongs in your Linux log workflow

  • Set up a small, local toolkit (no cloud required)

  • Run a quick anomaly detector for SSH brute-forcing

  • Reduce log noise with template mining

  • Wire it up to simple alerts

All examples are shell- and ops-friendly, with installation instructions for apt, dnf, and zypper.

Why use AI for log analysis?

  • Pattern recognition at scale: Machine learning can quickly find recurring structures in messy, semi-structured logs.

  • Unsupervised detection: Even with no labels, algorithms like Isolation Forest can flag “weird” behavior—e.g., an IP behaving unlike others.

  • Noise reduction: Template mining collapses near-duplicate messages into digestible patterns so analysts focus on what changed.

  • Local and scriptable: You can run everything on a single box, under a Linux user, and glue it together with Bash.

1) Prepare your toolbox

We’ll use Python (for ML), pip/venv (to isolate packages), and jq (for JSON in Bash).

Install prerequisites:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-pip python3-venv jq build-essential python3-dev
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-pip jq gcc python3-devel
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y python3 python3-pip jq gcc python3-devel

Create a Python virtual environment:

python3 -m venv ~/.venvs/ailogs
source ~/.venvs/ailogs/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn drain3

Ensure your user can read the relevant logs. On systemd systems, journalctl works via sudo or by adding your user to the systemd-journal group:

sudo usermod -aG systemd-journal "$USER"
# Log out and back in for group change to take effect.

For file-based logs (e.g., /var/log/auth.log, /var/log/secure), make sure you have read permission (typically requires sudo).

2) Quick-win: Detect SSH anomalies with Isolation Forest

Brute force and credential-stuffing attempts show up clearly in SSH auth logs. We’ll parse recent SSH logs, extract per-IP features, and flag suspicious ones.

Create detect_ssh_anomalies.py:

#!/usr/bin/env python3
import sys, re, math
from collections import defaultdict
from datetime import datetime
import pandas as pd
from sklearn.ensemble import IsolationForest

ip_re = re.compile(r'(\d{1,3}\.){3}\d{1,3}')
failed_re = re.compile(r'Failed password')
invalid_user_re = re.compile(r'Invalid user')
accepted_re = re.compile(r'Accepted (password|publickey)')
user_re = re.compile(r'for(?: invalid user)? ([a-zA-Z0-9._-]+)')

# Feature store per IP
stats = defaultdict(lambda: {
    'failed': 0,
    'invalid_user': 0,
    'accepted': 0,
    'users': set()
})

for line in sys.stdin:
    ip_m = ip_re.search(line)
    if not ip_m:
        continue
    ip = ip_m.group(0)

    if failed_re.search(line):
        stats[ip]['failed'] += 1
    if invalid_user_re.search(line):
        stats[ip]['invalid_user'] += 1
    if accepted_re.search(line):
        stats[ip]['accepted'] += 1
    u = user_re.search(line)
    if u:
        stats[ip]['users'].add(u.group(1))

# Build feature frame
rows = []
for ip, s in stats.items():
    rows.append({
        'ip': ip,
        'failed': s['failed'],
        'invalid_user': s['invalid_user'],
        'accepted': s['accepted'],
        'distinct_users': len(s['users']),
        'fail_to_accept_ratio': (s['failed'] / (s['accepted']+1)),
        'attacky_score': s['failed'] + 2*s['invalid_user'] + 0.5*s['distinct_users']
    })

if not rows:
    sys.exit(0)

df = pd.DataFrame(rows)

# Train simple IsolationForest (unsupervised)
model = IsolationForest(
    n_estimators=200,
    contamination=0.05,  # tune to your environment
    random_state=42
)
features = df[['failed','invalid_user','accepted','distinct_users','fail_to_accept_ratio','attacky_score']]
model.fit(features)
pred = model.predict(features)               # -1 anomalous, 1 normal
score = model.decision_function(features)    # lower is more anomalous

df['iforest_pred'] = pred
df['iforest_score'] = score

# Sort by most anomalous (lowest score) and print top
out = df.sort_values('iforest_score').reset_index(drop=True)

# Pretty print CSV to allow easy Bash parsing
print("ip,failed,invalid_user,accepted,distinct_users,fail_to_accept_ratio,attacky_score,iforest_pred,iforest_score")
for _, r in out.iterrows():
    print(f"{r.ip},{int(r.failed)},{int(r.invalid_user)},{int(r.accepted)},{int(r.distinct_users)},"
          f"{r.fail_to_accept_ratio:.3f},{r.attacky_score:.3f},{int(r.iforest_pred)},{r.iforest_score:.6f}")

Make it executable:

chmod +x detect_ssh_anomalies.py

Run it against the last 24 hours of SSH logs (systemd-journald):

sudo journalctl -u ssh -S -24h --no-pager | ./detect_ssh_anomalies.py
# If your unit is sshd (Fedora/RHEL):
sudo journalctl -u sshd -S -24h --no-pager | ./detect_ssh_anomalies.py

Or against file-based logs:

sudo cat /var/log/auth.log | ./detect_ssh_anomalies.py        # Debian/Ubuntu
sudo cat /var/log/secure | ./detect_ssh_anomalies.py          # RHEL/Fedora/SUSE

Interpretation:

  • iforest_pred = -1 means “anomalous”

  • Lower iforest_score = more suspicious

  • Use attacky_score and/or iforest_score as simple thresholds for alerting

Schedule daily with a systemd timer or cron to get a shortlist of suspect IPs for blocking or further triage.

3) Reduce noise with log template mining (Drain3)

Logs are notoriously repetitive with minor differences (IDs, IPs). Template mining clusters similar lines into templates so you can spot “new” or “rare” patterns quickly.

We’ll use Drain3 (installed earlier) to read logs from stdin and print new/changed templates.

Create template_miner.py:

#!/usr/bin/env python3
import sys
from drain3 import TemplateMiner
from drain3.template_miner_config import TemplateMinerConfig

config = TemplateMinerConfig()
config.load_default()
# Tweak if needed:
# config.drain_sim_th = 0.4  # similarity threshold
# config.drain_depth = 5

miner = TemplateMiner(config=config)

seen_templates = {}

for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    r = miner.add_log_message(line)
    tid = r["cluster_id"]
    tmpl = r["template_mined"]
    occ = r["cluster_size"]

    # Print when a new template forms or grows beyond a small threshold
    if tid not in seen_templates:
        seen_templates[tid] = occ
        print(f"NEW_TEMPLATE id={tid} size={occ} tmpl={tmpl}")
    else:
        prev = seen_templates[tid]
        if occ in (2, 5, 10, 50) or occ % 100 == 0:
            print(f"TEMPLATE_UPDATE id={tid} size={occ} tmpl={tmpl}")
        seen_templates[tid] = occ

Make it executable:

chmod +x template_miner.py

Run it on, say, NGINX access logs in real-time:

sudo tail -F /var/log/nginx/access.log | ./template_miner.py

What to look for:

  • NEW_TEMPLATE indicates previously unseen events—often worthy of attention

  • Rapid TEMPLATE_UPDATE growth may indicate an emerging issue or attack pattern

  • Use this to build suppression rules or to focus on outliers while compressing the rest

4) Turn insights into alerts (Bash + curl + jq)

Send anomalies to Slack or any webhook once thresholds are tripped.

Example: Alert top anomalous SSH IPs if any are found:

# Export your Slack Incoming Webhook URL first
# export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"

suspicious=$(sudo journalctl -u ssh -S -24h --no-pager \
  | ./detect_ssh_anomalies.py \
  | awk -F, 'NR>1 && $8=="-1"{print}' \
  | head -n 10)

if [ -n "$suspicious" ]; then
  payload=$(jq -n --arg text "SSH anomalies detected:\n$suspicious" '{text:$text}')
  curl -s -X POST -H 'Content-type: application/json' \
    --data "$payload" "$SLACK_WEBHOOK_URL" >/dev/null && echo "Alert sent."
fi

Optional: Summarize with a local LLM for human-friendly context. If you want to try this, install Ollama (single script):

curl -fsSL https://ollama.com/install.sh | sh
# Then run a small model, e.g., mistral
ollama pull mistral

Then:

summary=$(printf "%s" "$suspicious" | ollama run mistral "Summarize these SSH anomalies for a SOC analyst:")
echo "$summary"

Keep this optional—don’t block alerts on LLM availability.

Real-world pointers

  • Calibrate contamination: Start with contamination=0.05 (5%) and adjust until you get 1–10 high-quality anomalies per day per host.

  • Separate by service: Train per log type/service (SSH vs. web vs. DB) for crisper signals.

  • Baseline by time/window: Nightly jobs vs. business hours look different. Consider training by window or feeding “hour” as a feature.

  • Keep a blocklist/allowlist: Pipe confirmed bad IPs into firewall automation, and maintain an allowlist for known scanners or maintenance hosts.

  • Version your pipelines: Save your scripts/config in git; note when you change thresholds or features.

Conclusion and Next Steps

You don’t need a massive SIEM to get value from AI in log analysis. With a few Bash-friendly scripts and open-source Python libraries, you can:

  • Flag suspicious SSH sources

  • Collapse repetitive logs into readable templates

  • Ship only the important stuff to your eyes (or a chat channel)

Next steps:

  • Put the SSH anomaly job behind a systemd timer

  • Add parsers for web server logs and DB audit logs

  • Expand features (geo-IP, ASN, time-of-day) for stronger detection

  • Feed outputs into your existing SIEM or ticketing system

Security thrives on fast feedback. Start with the SSH detector today, tune it over a week, and you’ll have an immediately useful, locally controlled AI assistant for your logs.

Related posts