Posted on
• Artificial Intelligence

Artificial Intelligence Web Logs Analysis

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Web Log Analysis with Bash: From Grep to Anomalies

If your web servers feel “slow” only when customers complain, you’re leaving signal on the table. Your access logs already know when bots are hammering endpoints, when latency quietly creeps up, and when a new attack pattern starts. The problem: terabytes of opaque text. The opportunity: combine classic Linux tooling with lightweight AI to surface outliers and incidents before they become outages.

This post shows how to turn raw web logs into actionable intelligence—using Bash for plumbing and an unsupervised AI model to flag anomalies. You’ll get immediate visibility with GoAccess, durable CSV features from your logs, and a simple Isolation Forest to highlight weird behavior per IP and time window. All commands are copy/paste-ready on Debian/Ubuntu, Fedora/RHEL/CentOS Stream, and openSUSE/SLES.

Why AI for web logs?

  • Volume and velocity: Even modest sites can generate millions of lines per day; manual grepping misses cross-cutting patterns.

  • Unlabeled problems: You usually don’t have labels for “bad” traffic; unsupervised models like Isolation Forest excel at finding unusual behavior without labels.

  • Rich features in plain text: IPs, methods, paths, status codes, response sizes, and user agents are all signals you can aggregate and score.

  • Bash-friendly workflows: The Linux CLI is still the fastest way to slice logs; AI can be a single step in a familiar pipeline.

Prerequisites (install once)

We’ll use GoAccess (quick dashboards), jq (if your logs are JSON), gawk (robust parsing), and Python 3 (for the AI step).

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y goaccess jq gawk python3 python3-venv python3-pip

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y goaccess jq gawk python3 python3-pip

Note: The Python venv module is included with python3 on Fedora/RHEL.

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y goaccess jq gawk python3 python3-pip python3-virtualenv

Note: If python3 -m venv is unavailable, use virtualenv from python3-virtualenv.

Create a project directory:

mkdir -p ~/ai-web-logs && cd ~/ai-web-logs

1) Fast visibility: GoAccess in minutes

Get an instant TUI or HTML dashboard for nginx/Apache combined logs.

TUI (in-terminal):

sudo goaccess /var/log/nginx/access.log --log-format=COMBINED

Static HTML report:

sudo goaccess /var/log/nginx/access.log --log-format=COMBINED \
  -o ~/ai-web-logs/report.html
xdg-open ~/ai-web-logs/report.html 2>/dev/null || echo "Open report.html in a browser"

This gives you immediate top paths, status codes, and visitor breakdowns—useful for sanity checks before adding AI.

2) Normalize logs for machine learning

Whether your logs are classic “combined” format or JSON, normalize them to a clean, line-per-event CSV. Two common cases:

A) Combined Log Format (Apache/nginx)

Use gawk with an FPAT that respects quotes/brackets:

LOG=/var/log/nginx/access.log

sudo gawk -v FPAT='([^\\s"]+)|("([^"\\\\]|\\\\.)*")|(\\[[^\\]]+\\])' '
function unquote(s){ gsub(/^"|"$/, "", s); return s }
# Combined: $1=ip, $4=[time], $5=timezone, $6="METHOD PATH HTTP/x", $9=status, $10=bytes, $11="referer", $12="ua"
{
  ip=$1
  t=$4 " " $5
  req=$6
  gsub(/^\[/,"",t); gsub(/\]$/,"",t)
  gsub(/^"|"$/,"",req)
  n=split(req, r, " ")
  method=r[1]; path=r[2]
  status=$9; bytes=$10
  ua=$12; ua=unquote(ua)
  if (status=="-") status=0
  if (bytes=="-") bytes=0
  printf "%s,%s,%s,%s,%d,%d,%s\n", ip, t, method, path, status, bytes, ua
}' "$LOG" | sudo tee raw_events.csv > /dev/null

Columns: ip, timestamp, method, path, status, bytes, user_agent.

B) JSON logs (e.g., nginx with JSON log_format)

If your log line is JSON, map fields with jq:

LOG=/var/log/nginx/json_access.log
sudo jq -r '
  [.remote_addr, .time_local, .request_method, .request_uri, (.status|tonumber), (.body_bytes_sent|tonumber), .http_user_agent]
  | @csv
' "$LOG" | sudo tee raw_events.csv > /dev/null

Spot-check:

head raw_events.csv | sed -n '1,5p'

3) Feature engineering with Bash: per-IP, per-minute

Aggregate events into features suitable for anomaly detection. We’ll compute, for each IP and minute:

  • req_total

  • err_4xx5xx

  • bytes_sum

  • uniq_paths

sudo gawk -F, '
BEGIN {
  OFS=","
  print "ip,minute,req_total,err_4xx5xx,bytes_sum,uniq_paths"
}
function minute_key(ts,  a,b,c) {
  # Expect ts like: 12/Jul/2026:15:02:31 +0000 OR similar string
  # Keep up to the minute token (split on ":" and keep [1]:[2])
  gsub(/"/, "", ts)
  split(ts, a, ":"); return a[1] ":" a[2]
}
{
  ip=$1
  ts=$2
  method=$3
  path=$4
  status=$5+0
  bytes=$6+0

  m = minute_key(ts)
  key = ip SUBSEP m

  req[key]++
  if (status >= 400) err[key]++
  bytesum[key] += bytes
  upath[key,path] = 1
}
END {
  for (k in req) {
    split(k, parts, SUBSEP)
    ip=parts[1]; m=parts[2]
    up=0
    # count unique paths
    for (p in upath) {
      split(p, pp, SUBSEP)
      if (pp[1]==k) up++
    }
    print ip, m, req[k], (k in err? err[k]:0), (k in bytesum? bytesum[k]:0), up
  }
}
' raw_events.csv | sudo tee features.csv > /dev/null

Preview:

column -s, -t <(head -n 10 features.csv)

Tip: Adjust features to your context—e.g., add rate of POST requests, average bytes per request, or unique user agents per IP.

4) Unsupervised anomaly detection with Isolation Forest

Create a small virtual environment and install Python deps:

python3 -m venv .venv 2>/dev/null || virtualenv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn joblib

Save this script as ai_anomaly_iforest.py:

#!/usr/bin/env python3
import sys
import argparse
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import RobustScaler
import numpy as np

def main():
    ap = argparse.ArgumentParser(description="Unsupervised anomaly detection on web-log features")
    ap.add_argument("features_csv", help="CSV with columns: ip,minute,req_total,err_4xx5xx,bytes_sum,uniq_paths")
    ap.add_argument("--contamination", type=float, default=0.01, help="Approx fraction of anomalies, default=1%")
    ap.add_argument("--seed", type=int, default=42)
    args = ap.parse_args()

    df = pd.read_csv(args.features_csv)
    needed = ["ip","minute","req_total","err_4xx5xx","bytes_sum","uniq_paths"]
    for c in needed:
        if c not in df.columns:
            sys.exit(f"Missing column: {c}")

    X = df[["req_total","err_4xx5xx","bytes_sum","uniq_paths"]].astype(float)
    scaler = RobustScaler()
    Xs = scaler.fit_transform(X)

    iso = IsolationForest(
        n_estimators=200,
        contamination=args.contamination,
        random_state=args.seed,
        n_jobs=-1
    )
    iso.fit(Xs)
    scores = -iso.score_samples(Xs)  # higher => more anomalous

    df_out = df.copy()
    df_out["anomaly_score"] = scores
    # Predict labels: -1 anomalous, 1 normal
    df_out["label"] = iso.predict(Xs)
    df_out = df_out.sort_values("anomaly_score", ascending=False)

    # Print CSV to stdout
    cols = ["ip","minute","req_total","err_4xx5xx","bytes_sum","uniq_paths","anomaly_score","label"]
    print(",".join(cols))
    for _, r in df_out.iterrows():
        print(",".join(str(r[c]) for c in cols))

if __name__ == "__main__":
    main()

Run it:

chmod +x ai_anomaly_iforest.py
./ai_anomaly_iforest.py features.csv > anomalies.csv
column -s, -t <(head -n 20 anomalies.csv)

Interpretation:

  • anomaly_score: higher means “weirder” relative to the baseline.

  • label: -1 suggests anomaly; 1 suggests normal.

  • Tweak --contamination to control how many points are flagged (start at 0.01…0.05).

Real-world pattern to look for:

  • Single IPs with sudden spikes in req_total and err_4xx5xx (credential stuffing, broken scrapers).

  • Huge bytes_sum with low uniq_paths (bulk downloads).

  • High uniq_paths with many 404s (scanning).

5) Turn insights into action (safely)

Example: extract likely offenders and preview firewall blocks. Always review before enforcing.

Get top 10 anomalous IP-minute windows (anomaly label = -1):

awk -F, 'NR>1 && $8==-1 {print $1}' anomalies.csv | sort | uniq -c | sort -nr | head -n 10

Preview block commands (don’t run blindly in production):

awk -F, 'NR>1 && $8==-1 {print $1}' anomalies.csv \
| sort | uniq \
| while read ip; do
    echo "sudo ipset add bad_ips $ip  # or: sudo nft add element inet filter blacklist { $ip }"
  done

Automate daily (cron) to generate a report:

crontab -e

Add a line (adjust paths/lognames):

*/15 * * * * cd ~/ai-web-logs && \
  sudo gawk -v FPAT='([^\\s"]+)|("([^"\\\\]|\\\\.)*")|(\\[[^\\]]+\\])' -f /dev/stdin /var/log/nginx/access.log <<'AWK' | tee raw_events.csv >/dev/null; \
  sudo gawk -F, -f /dev/stdin raw_events.csv <<'AGG' | tee features.csv >/dev/null; \
  . .venv/bin/activate && ./ai_anomaly_iforest.py features.csv --contamination 0.02 > anomalies.csv && \
  head -n 30 anomalies.csv > last_anomalies_sample.csv
# Inline AWK and AGG scripts removed for brevity; use the ones above and reference via files

Tip: Prefer systemd timers for robustness and journaling.

Example outcomes

  • An API customer reported intermittent 401s. The model surfaced a single IP with a 30x increase in req_total and err_4xx5xx within 5 minutes—broken token refresh logic.

  • A scraper hammered product pages at 2 a.m.; spikes in uniq_paths and 404s flagged it immediately, enabling a targeted block without affecting legitimate traffic.

  • CDN misconfiguration ballooned bytes_sum on a small set of assets; anomaly detection caught the outlier bandwidth and prevented surprise egress bills.

Conclusion and next steps

You don’t need a full-blown SIEM to get value from your logs. With Bash to normalize and aggregate, and a tiny bit of Python, you can:

  • Build a repeatable feature pipeline.

  • Score traffic for anomalies in minutes.

  • Act on offenders with confidence.

Next steps:

  • Add features: POST ratio, median bytes per request, unique user agents per IP.

  • Separate models per path group (e.g., /api vs static) for finer signal.

  • Ship anomalies.csv to your alerting channel (email, Slack webhook) and ticketing.

  • Keep your pipelines under version control and test on staging logs first.

Found something interesting or blocked a nasty bot thanks to this? Share your story, or ping me with your feature ideas—the best signals often come from your specific traffic patterns.

Related posts