Posted on
• Artificial Intelligence

Artificial Intelligence Log Analysis Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Log Analysis Projects (for Bash-Friendly Linux Users)

Ever felt like your logs are trying to tell you something, but you don’t speak “10GB of text per day”? You’re not alone. Logs are full of operational gold—security signals, performance regressions, and early warnings. The problem is volume and variability. This article shows you how to use practical AI techniques with Bash and Python to turn raw logs into reliable insights—without buying a SIEM or running a massive cluster.

You’ll learn 3 hands-on projects:

  • Anomaly detection for Nginx/Apache access logs

  • SSH brute-force detection using robust statistics

  • Template mining to de-duplicate noisy logs and find “new” errors

Each example comes with minimal, production-minded commands and installation instructions for apt, dnf, and zypper.

Why AI for Log Analysis?

  • Scale and speed: AI models (even simple ones) find patterns and outliers faster than ad-hoc greps or dashboards.

  • Fewer false positives: Instead of 100 regex alerts, you learn the “shape” of normal behavior and get alerted only when something breaks that shape.

  • Adaptable to new signals: Logs evolve; models retrain. You don’t have to rewrite rules for every service update.

Prerequisites: One-Time Setup

We’ll use Python with a virtual environment for ML, plus jq and ripgrep to slice-and-dice logs in Bash.

Choose the command that matches your distro:

APT (Debian/Ubuntu):

sudo apt update
sudo apt install -y python3 python3-venv python3-pip build-essential jq ripgrep

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y python3 python3-pip python3-virtualenv gcc gcc-c++ make python3-devel jq ripgrep

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y python3 python3-pip python3-virtualenv gcc gcc-c++ make python3-devel jq ripgrep

Create a Python virtual environment and install libraries:

python3 -m venv ~/.venvs/logai
source ~/.venvs/logai/bin/activate
pip install --upgrade pip
pip install numpy pandas scikit-learn matplotlib drain3 ujson

Recommended workspace:

mkdir -p ~/log-ai && cd ~/log-ai

Note: You may need read access to system logs. Use sudo where appropriate or adjust group membership (e.g., adm on Debian/Ubuntu).

Project 1: Access-Log Anomaly Detection (Isolation Forest)

Goal: Flag unusual web requests without writing dozens of brittle rules.

What it does:

  • Parses Nginx/Apache “combined” logs

  • Builds simple numeric features (method, status, path length/depth, hour, bytes)

  • Trains an IsolationForest to score each line; surfaces the most anomalous

Save as nginx_anomaly.py:

#!/usr/bin/env python3
import re
import sys
import argparse
import pandas as pd
import numpy as np
from datetime import datetime
from sklearn.ensemble import IsolationForest

# Regex for "combined" format: 
# $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent ...
LOG_RE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>[A-Z]+)\s(?P<path>[^" ]*)(?: [^"]*)?" (?P<status>\d{3}) (?P<bytes>\d+|-)\b'
)

def parse_time(s):
    # Example: 12/Mar/2026:19:55:32 +0000
    try:
        return datetime.strptime(s, "%d/%b/%Y:%H:%M:%S %z")
    except Exception:
        return None

def features_from_line(line):
    m = LOG_RE.match(line)
    if not m:
        return None
    d = m.groupdict()
    dt = parse_time(d["time"])
    hour = dt.hour if dt else -1
    method = d["method"]
    method_map = {"GET":0, "POST":1, "PUT":2, "DELETE":3, "HEAD":4, "PATCH":5, "OPTIONS":6}
    mcode = method_map.get(method, 7)
    path = d["path"] or "/"
    path_len = len(path)
    # depth: number of segments ("/a/b" -> 2). Normalize "/" to 0.
    p = path.strip("/")
    depth = 0 if p == "" else p.count("/") + 1
    status = int(d["status"])
    bytes_sent = 0 if d["bytes"] == "-" else int(d["bytes"])
    return [hour, mcode, path_len, depth, status, bytes_sent], d, line

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--log", required=True, help="Path to access log")
    ap.add_argument("--top", type=int, default=50, help="Show N most anomalous lines")
    ap.add_argument("--seed", type=int, default=42)
    args = ap.parse_args()

    feats = []
    meta = []
    raw = []
    with open(args.log, "r", errors="ignore") as f:
        for line in f:
            out = features_from_line(line.rstrip("\n"))
            if out:
                x, d, l = out
                feats.append(x)
                meta.append(d)
                raw.append(l)

    if not feats:
        print("No parseable lines found. Check your log format.", file=sys.stderr)
        sys.exit(1)

    X = np.array(feats, dtype=float)

    # IsolationForest: robust unsupervised anomaly detector
    iso = IsolationForest(n_estimators=200, contamination="auto", random_state=args.seed)
    iso.fit(X)
    scores = iso.score_samples(X)  # higher is more normal; lower is more anomalous

    df = pd.DataFrame(meta)
    df["score"] = scores
    df["raw"] = raw
    # Show lowest scores first (most anomalous)
    df = df.sort_values("score").head(args.top)

    for i, row in df.iterrows():
        print(f"[score={row['score']:.4f}] {row['raw']}")

if __name__ == "__main__":
    main()

Run it:

# Example paths; adjust for Apache vs Nginx
# Nginx: /var/log/nginx/access.log
# Apache: /var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (RHEL/Fedora)
sudo python3 nginx_anomaly.py --log /var/log/nginx/access.log --top 40

Tips:

  • If your log format deviates, tweak the regex near LOG_RE.

  • Consider running this daily and storing results in a file:

sudo python3 nginx_anomaly.py --log /var/log/nginx/access.log --top 200 | tee -a anomalies-access.log

Project 2: SSH Brute-Force/Outlier IP Detection (Robust Stats)

Goal: Identify suspicious source IPs hitting SSH without complicated correlation rules.

We’ll:

  • Pull the last 24h of SSH logs

  • Count failed attempts by IP

  • Use a robust z-score (Median Absolute Deviation) to flag outliers

Export logs (journalctl or file-based):

# Systemd journals
sudo journalctl -u ssh -u sshd --since "24 hours ago" --output short-iso > ssh-24h.log
# Or Debian/Ubuntu file-based:
# sudo cp /var/log/auth.log ssh-24h.log

Save as ssh_bruteforce.py:

#!/usr/bin/env python3
import re
import sys
from statistics import median

FAIL_RE = re.compile(r'(Failed password|Invalid user).*(?:from|rhost=)\s(?P<ip>\d+\.\d+\.\d+\.\d+)')
# Optional: Success lines to compute success/fail ratio
OK_RE = re.compile(r'Accepted (?:password|publickey).*(?:from|rhost=)\s(?P<ip>\d+\.\d+\.\d+\.\d+)')

def mad_zscores(counts):
    vals = list(counts.values())
    if not vals:
        return {}
    med = median(vals)
    abs_dev = [abs(x - med) for x in vals]
    mad = median(abs_dev) or 1e-9
    z = {}
    for k, x in counts.items():
        z[k] = 0.6745 * (x - med) / mad
    return z

def main(path):
    fails = {}
    oks = {}
    with open(path, "r", errors="ignore") as f:
        for line in f:
            m = FAIL_RE.search(line)
            if m:
                ip = m.group("ip")
                fails[ip] = fails.get(ip, 0) + 1
                continue
            m2 = OK_RE.search(line)
            if m2:
                ip = m2.group("ip")
                oks[ip] = oks.get(ip, 0) + 1

    z = mad_zscores(fails)
    # Flags: high z-score or high absolute count
    alerts = [(ip, fails.get(ip,0), z.get(ip,0.0), oks.get(ip,0)) for ip in fails]
    alerts.sort(key=lambda t: (t[2], t[1]), reverse=True)

    print("ip,fail_count,robust_z,ok_count")
    for ip, fc, rz, ok in alerts:
        if fc >= 10 or rz >= 3:  # tune thresholds
            print(f"{ip},{fc},{rz:.2f},{ok}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: ssh_bruteforce.py <logfile>", file=sys.stderr)
        sys.exit(1)
    main(sys.argv[1])

Run it:

python3 ssh_bruteforce.py ssh-24h.log | column -t -s,

Pure-Bash quick check (handy for triage):

rg -oN '(?<=from )\d+\.\d+\.\d+\.\d+' ssh-24h.log | sort | uniq -c | sort -nr | head

Project 3: Log Template Mining (Drain3) to Reduce Noise and Surface “New” Errors

Goal: Automatically group similar log lines into templates, cut noise, and alert on never-seen patterns.

Why it matters: Production logs often have dynamic values (IDs, IPs). Template mining reduces millions of lines to a manageable set of patterns. “New template” often equals “new failure mode.”

Save as drain_stream.py:

#!/usr/bin/env python3
import sys
import json
from drain3 import TemplateMiner
from drain3.template_miner_config import TemplateMinerConfig
from drain3.file_persistence import FilePersistence

# Config with default settings; adjust as desired
config = TemplateMinerConfig()
persistence = FilePersistence("drain3_state.json")
miner = TemplateMiner(persistence, config)

for line in sys.stdin:
    line = line.strip()
    if not line:
        continue
    r = miner.add_log_message(line)
    # r['change_type']: "None", "cluster_created", "cluster_template_changed"
    if r["change_type"] != "None":
        out = {
            "change_type": r["change_type"],
            "template": r["cluster"]["template_mined"],
            "size": r["cluster"]["size"],
        }
        print(json.dumps(out, ensure_ascii=False))
        sys.stdout.flush()

Stream your logs into it:

# Follow system logs with minimal formatting and pipe into Drain3
sudo journalctl -f -o cat | python3 drain_stream.py | tee drain3-events.jsonl

Interpretation:

  • You’ll see JSON messages when a new template is discovered or a template changes significantly.

  • These events are excellent alert candidates. For example, send lines with "change_type": "cluster_created" to your alerting channel.

Optional: You can pre-seed the miner with a day’s worth of logs before going live:

sudo journalctl --since "1 day ago" -o cat | python3 drain_stream.py > /dev/null

Operationalizing: Scheduling and Hygiene

  • Cron (daily anomaly scan for access logs):
# Edit root’s crontab
sudo crontab -e
# Run at 01:25 daily and append to a dated file
25 1 * * * /usr/bin/python3 /home/youruser/log-ai/nginx_anomaly.py --log /var/log/nginx/access.log --top 200 >> /var/log/nginx/anomaly-$(date +\%F).log 2>&1
  • Systemd service/timer (for Drain3 streaming):
# /etc/systemd/system/drain3.service
[Unit]
Description=Drain3 Log Template Miner

[Service]
ExecStart=/bin/bash -lc 'source ~/.venvs/logai/bin/activate && journalctl -f -o cat | python3 /home/youruser/log-ai/drain_stream.py >> /var/log/drain3-events.jsonl'
Restart=always

# /etc/systemd/system/drain3.timer
[Unit]
Description=Start Drain3 at boot

[Timer]
OnBootSec=15sec
Unit=drain3.service

[Install]
WantedBy=multi-user.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now drain3.service
  • Keep data light:
    • Rotate output logs with logrotate.
    • Store only features or templates, not raw PII where not required.
    • Document thresholds and retraining cadence.

Real-World Tips

  • Start simple: IsolationForest and MAD-based outliers catch a surprising amount of real issues.

  • Keep parsers resilient: Logs change. Write regex to fail gracefully and log parse rates.

  • Thresholds are context: Tune per environment; watch false positives for a week before paging.

  • Use Bash to pre-filter: It’s faster to pipe a small, relevant slice into Python than to parse gigabytes unfiltered.

Handy Bash filters:

# Fetch only 5xx lines, last day
sudo journalctl --since "1 day ago" -u nginx -o cat | rg 'HTTP/1\.[01]" 5\d{2}'

# Extract suspicious user agents or long URLs
rg -n 'curl|bot|wget|Mozilla.*\?[^ ]{200,}' /var/log/nginx/access.log

Conclusion and Call to Action

You don’t need a full-blown data platform to get meaningful AI-powered insights from logs. With a few Bash-friendly tools and small Python scripts, you can:

  • Find unusual web requests without handcrafting rules

  • Catch SSH brute-force attempts early

  • Collapse noisy logs into stable templates and detect “new” errors in real-time

Your next step: 1) Install the prerequisites (apt/dnf/zypper commands above). 2) Run Project 1 on your access logs and review the top anomalies. 3) Enable Project 2 daily and tune thresholds for your SSH profile. 4) Pilot Project 3 on one service and alert on new templates for a week.

If you want a packaged repo with these scripts, reply and I’ll assemble a ready-to-clone project with Makefile and systemd units.

Related posts