Artificial Intelligence Apache Optimisation: Smarter, Faster, and Bash-Friendly

You don’t need a rack of GPUs or a research lab to make Apache fly. With a few smart sensors, a dash of AI, and some Bash-fueled automation, you can turn raw logs into concrete tuning decisions that cut latency, raise throughput, and prevent 2 a.m. firefights.

This post shows how to combine Apache’s native features with lightweight machine learning to:

• Baseline performance with the right metrics.

• Learn from your traffic to auto-tune MPM, KeepAlive, and HTTP/2.

• Identify cacheable routes and apply compression with evidence.

• Detect anomalies early and roll config changes back if needed.

If you run Linux and like living in the shell, this is for you.

Why AI + Apache Optimisation is worth your time

• Traffic patterns shift hourly. Fixed “best practice” configs get stale. AI-driven tuning adapts based on your real workloads.

• Tiny changes compound. KeepAliveTimeout off by a few seconds can exhaust workers under spikes. HTTP/2 without proper thread sizing can starve throughput. Learning from logs turns knobs with confidence.

• It’s lightweight. We’re talking simple models you can run with Python on the same host, fed by Apache logs—no heavy infra required.

Prerequisites and installation

Pick your distro and run the relevant block. We’ll install Apache, status/compression/HTTP2 modules, tools for benchmarking, and Python ML libs.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y apache2 apache2-utils libapache2-mod-brotli brotli python3 python3-pip
# Optional but recommended
sudo apt install -y libapache2-mod-security2
# Enable useful modules
sudo a2enmod status http2 headers brotli deflate cache cache_disk
# Start/enable service
sudo systemctl enable --now apache2
# Python libraries
python3 -m pip install --user --upgrade pip
python3 -m pip install --user numpy pandas scikit-learn

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y httpd httpd-tools mod_http2 mod_brotli brotli python3 python3-pip
# Note: On some RHEL releases, mod_brotli may require EPEL; if unavailable, rely on mod_deflate.
sudo systemctl enable --now httpd
# Python libraries
python3 -m pip install --user --upgrade pip
python3 -m pip install --user numpy pandas scikit-learn

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y apache2 apache2-utils apache2-mod_http2 apache2-mod_brotli brotli python3 python3-pip
sudo a2enmod http2 headers brotli deflate cache cache_disk status
sudo systemctl enable --now apache2
# Python libraries
python3 -m pip install --user --upgrade pip
python3 -m pip install --user numpy pandas scikit-learn

Note on services:

• Debian/Ubuntu/openSUSE service: apache2

• Fedora/RHEL/CentOS service: httpd

Step 1: Baseline the right signals (status, timing, HTTP/2)

Turn on status and extended logging so you can measure latency and concurrency.

Enable server-status (restricted to localhost here; adjust for your ops network):

Debian/Ubuntu:

sudo tee /etc/apache2/conf-available/server-status.conf >/dev/null <<'EOF'
<Location /server-status>
    SetHandler server-status
    Require local
</Location>
ExtendedStatus On
EOF
sudo a2enconf server-status
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/server-status.conf >/dev/null <<'EOF'
<Location /server-status>
    SetHandler server-status
    Require local
</Location>
ExtendedStatus On
EOF
sudo systemctl reload httpd

openSUSE:

sudo tee /etc/apache2/conf.d/server-status.conf >/dev/null <<'EOF'
<Location /server-status>
    SetHandler server-status
    Require local
</Location>
ExtendedStatus On
EOF
sudo systemctl reload apache2

Add a log format that captures microseconds per request (%D) and other useful fields:

Debian/Ubuntu:

sudo tee /etc/apache2/conf-available/ai-logs.conf >/dev/null <<'EOF'
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D \"%{Host}i\"" ai_combined
CustomLog ${APACHE_LOG_DIR}/access_ai.log ai_combined
EOF
sudo a2enconf ai-logs
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/ai-logs.conf >/dev/null <<'EOF'
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D \"%{Host}i\"" ai_combined
CustomLog logs/access_ai.log ai_combined
EOF
sudo systemctl reload httpd

openSUSE:

sudo tee /etc/apache2/conf.d/ai-logs.conf >/dev/null <<'EOF'
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D \"%{Host}i\"" ai_combined
CustomLog /var/log/apache2/access_ai.log ai_combined
EOF
sudo systemctl reload apache2

Enable HTTP/2 (required for modern clients and higher concurrency on a single TCP connection):

Debian/Ubuntu:

sudo a2enmod http2
sudo tee /etc/apache2/conf-available/http2.conf >/dev/null <<'EOF'
Protocols h2 h2c http/1.1
H2ModernTLSOnly off
EOF
sudo a2enconf http2
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/http2.conf >/dev/null <<'EOF'
Protocols h2 h2c http/1.1
H2ModernTLSOnly off
EOF
sudo systemctl reload httpd

openSUSE:

sudo a2enmod http2
sudo tee /etc/apache2/conf.d/http2.conf >/dev/null <<'EOF'
Protocols h2 h2c http/1.1
H2ModernTLSOnly off
EOF
sudo systemctl reload apache2

Quick sanity check with ab:

ab -n 200 -c 20 http://127.0.0.1/

Step 2: Learn from your logs (lightweight ML, no heavy infra)

We’ll turn access_ai.log into features, label “slow” requests (>500 ms), and train a simple model to estimate which routes and times are trouble. We’ll also estimate concurrency from RPS × latency to size worker pools.

Save this as learn_slow_requests.py:

#!/usr/bin/env python3
import re, sys, math, datetime as dt
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Adjust path for your distro
log_path = sys.argv[1] if len(sys.argv) > 1 else "/var/log/apache2/access_ai.log"

rx = re.compile(r'''(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+)\s(?P<uri>\S+)[^"]*" (?P<status>\d{3}) (?P<bytes>\d+) "[^"]*" "[^"]*" (?P<usec>\d+) "(?P<host>[^"]*)"''')

rows = []
with open(log_path, 'r', errors='ignore') as f:
    for line in f:
        m = rx.search(line)
        if not m: continue
        ts = dt.datetime.strptime(m['ts'].split()[0], "%d/%b/%Y:%H:%M:%S")
        usec = int(m['usec'])
        rows.append({
            'ts': ts,
            'method': m['method'],
            'uri': m['uri'].split('?')[0],
            'status': int(m['status']),
            'bytes': int(m['bytes']),
            'lat_ms': usec/1000.0,
            'hour': ts.hour,
        })

df = pd.DataFrame(rows)
if df.empty:
    print("No data parsed. Ensure ai_combined log is active.")
    sys.exit(1)

# Label slow > 500 ms (adjust to your SLO)
df['slow'] = (df['lat_ms'] > 500).astype(int)

# Simple feature engineering
df['is_get'] = (df['method']=="GET").astype(int)
df['is_post'] = (df['method']=="POST").astype(int)
df['is_static'] = df['uri'].str.contains(r'\.(css|js|png|jpg|jpeg|gif|svg|ico|webp|woff2?)$', case=False, regex=True).astype(int)

X = df[['hour','bytes','is_get','is_post','is_static','status']]
y = df['slow']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)
model = LogisticRegression(max_iter=200).fit(X_train, y_train)
acc = model.score(X_test, y_test)

# Concurrency estimate: per-minute RPS * p95 latency (s)
g = df.set_index('ts').resample('1min').agg(
    rps=('lat_ms', 'count'),
    p95_ms=('lat_ms', lambda s: np.percentile(s, 95) if len(s)>0 else 0)
).fillna(0)
g['concurrency_est'] = g['rps'] * (g['p95_ms']/1000.0)
p95_conc = float(np.percentile(g['concurrency_est'], 95)) if len(g)>0 else 1.0
p95_rps = float(np.percentile(g['rps'], 95)) if len(g)>0 else 1.0
p95_lat = float(np.percentile(df['lat_ms'], 95)) if len(df)>0 else 1.0

# Identify top dynamic URIs with high repeat potential
top_dyn = (df[df['is_static']==0]
           .groupby('uri')
           .agg(count=('uri','count'),
                p95_ms=('lat_ms', lambda s: np.percentile(s,95)))
           .sort_values('count', ascending=False)
           .head(10))

print("Model accuracy (slow vs not-slow): {:.2f}".format(acc))
print("95th percentile: RPS={:.1f}, Latency={:.0f}ms, Concurrency≈{:.1f}".format(p95_rps, p95_lat, p95_conc))

# Sizing recommendations
target_headroom = 1.2
recommended_max_req_workers = int(math.ceil(p95_conc * target_headroom)) or 50
threads_per_child = 50  # typical event MPM sweet spot
server_limit = int(math.ceil(recommended_max_req_workers / threads_per_child)) or 1

print("\nSuggested MPM (event) settings:")
print("  ThreadsPerChild {}  ServerLimit {}  MaxRequestWorkers {}".format(
    threads_per_child, server_limit, server_limit*threads_per_child))

# KeepAlive suggestion: keep short if HTTP/2 enabled
if p95_rps > 50:
    print("Suggested KeepAliveTimeout: 2-3 seconds (high RPS, reduce socket hold).")
else:
    print("Suggested KeepAliveTimeout: 5 seconds (moderate RPS).")

print("\nCandidate dynamic URIs to cache (if safe to cache):")
for uri, row in top_dyn.iterrows():
    print("  {}  hits={}  p95={}ms".format(uri, int(row['count']), int(row['p95_ms'])))

Run it:

python3 learn_slow_requests.py /var/log/apache2/access_ai.log
# Fedora/RHEL: /var/log/httpd/access_ai.log
# openSUSE: /var/log/apache2/access_ai.log

You’ll get:

• An accuracy score to sanity-check the model.

• Estimated p95 concurrency for sizing.

• Suggested ThreadsPerChild, ServerLimit, and MaxRequestWorkers.

• KeepAliveTimeout guidance tied to RPS.

• A list of high-impact URIs that might benefit from caching.

Step 3: Auto-tune MPM, KeepAlive, and apply HTTP/2 wisely

Switch to event MPM (threaded, non-blocking, best for HTTP/2 and keep-alive). Ensure PHP is via PHP-FPM, not mod_php, before disabling prefork.

Debian/Ubuntu:

sudo a2dismod mpm_prefork || true
sudo a2enmod mpm_event
sudo systemctl reload apache2

Fedora/RHEL/CentOS (toggle LoadModule lines in 00-mpm.conf):

sudo sed -i 's/^LoadModule mpm_prefork_module/#LoadModule mpm_prefork_module/' /etc/httpd/conf.modules.d/00-mpm.conf
sudo sed -i 's/^#LoadModule mpm_event_module/LoadModule mpm_event_module/' /etc/httpd/conf.modules.d/00-mpm.conf
sudo systemctl reload httpd

Apply the model’s suggestions (adapt numbers to your output):

Debian/Ubuntu:

sudo tee /etc/apache2/conf-available/mpm_event_tuned.conf >/dev/null <<'EOF'
# AI-tuned basics (example; replace with your model's output)
<IfModule mpm_event_module>
    StartServers             2
    ThreadsPerChild         50
    ServerLimit              8
    MaxRequestWorkers      400
    MaxConnectionsPerChild   0
</IfModule>

KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 3

# HTTP/2 QoS
H2MaxSessionStreams 100
H2StreamMaxMemSize 4M
EOF
sudo a2enconf mpm_event_tuned
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/mpm_event_tuned.conf >/dev/null <<'EOF'
<IfModule mpm_event_module>
    StartServers             2
    ThreadsPerChild         50
    ServerLimit              8
    MaxRequestWorkers      400
    MaxConnectionsPerChild   0
</IfModule>

KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 3

Protocols h2 h2c http/1.1
H2MaxSessionStreams 100
H2StreamMaxMemSize 4M
EOF
sudo systemctl reload httpd

openSUSE:

sudo tee /etc/apache2/conf.d/mpm_event_tuned.conf >/dev/null <<'EOF'
<IfModule mpm_event_module>
    StartServers             2
    ThreadsPerChild         50
    ServerLimit              8
    MaxRequestWorkers      400
    MaxConnectionsPerChild   0
</IfModule>

KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 3

Protocols h2 h2c http/1.1
H2MaxSessionStreams 100
H2StreamMaxMemSize 4M
EOF
sudo systemctl reload apache2

Step 4: Intelligent caching and compression (evidence-based)

Enable disk cache and set safe defaults. Then, selectively cache dynamic routes the model flagged, only if they’re safe (idempotent, user-agnostic, etc.).

Debian/Ubuntu:

sudo a2enmod cache cache_disk headers expires
sudo tee /etc/apache2/conf-available/cache_tuned.conf >/dev/null <<'EOF'
CacheQuickHandler On
CacheEnable disk /
CacheDirLevels 2
CacheDirLength 2
CacheMaxFileSize 5M
CacheDefaultExpire 3600
Header add X-Cache-Status "%{cache-status}e"
# Example: per-route (adjust to safe candidates)
<Location "/api/public/stats">
   CacheEnable disk
   Header unset Set-Cookie
   ExpiresActive On
   ExpiresDefault "access plus 5 minutes"
</Location>
EOF
sudo a2enconf cache_tuned
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/cache_tuned.conf >/dev/null <<'EOF'
LoadModule cache_module modules/mod_cache.so
LoadModule cache_disk_module modules/mod_cache_disk.so
CacheQuickHandler On
CacheEnable disk /
CacheDirLevels 2
CacheDirLength 2
CacheMaxFileSize 5M
CacheDefaultExpire 3600
Header add X-Cache-Status "%{cache-status}e"
<Location "/api/public/stats">
   CacheEnable disk
   Header unset Set-Cookie
   ExpiresActive On
   ExpiresDefault "access plus 5 minutes"
</Location>
EOF
sudo systemctl reload httpd

openSUSE:

sudo a2enmod cache cache_disk headers expires
sudo tee /etc/apache2/conf.d/cache_tuned.conf >/dev/null <<'EOF'
CacheQuickHandler On
CacheEnable disk /
CacheDirLevels 2
CacheDirLength 2
CacheMaxFileSize 5M
CacheDefaultExpire 3600
Header add X-Cache-Status "%{cache-status}e"
<Location "/api/public/stats">
   CacheEnable disk
   Header unset Set-Cookie
   ExpiresActive On
   ExpiresDefault "access plus 5 minutes"
</Location>
EOF
sudo systemctl reload apache2

Turn on Brotli (fallback to Deflate if Brotli isn’t available):

Debian/Ubuntu:

sudo a2enmod brotli deflate headers
sudo tee /etc/apache2/conf-available/compress.conf >/dev/null <<'EOF'
AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
BrotliCompressionQuality 5
# Fallback
AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
Header append Vary Accept-Encoding
EOF
sudo a2enconf compress
sudo systemctl reload apache2

Fedora/RHEL/CentOS:

sudo tee /etc/httpd/conf.d/compress.conf >/dev/null <<'EOF'
<IfModule brotli_module>
  AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
  BrotliCompressionQuality 5
</IfModule>
AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
Header append Vary Accept-Encoding
EOF
sudo systemctl reload httpd

openSUSE:

sudo a2enmod brotli deflate headers
sudo tee /etc/apache2/conf.d/compress.conf >/dev/null <<'EOF'
AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
BrotliCompressionQuality 5
AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript application/javascript application/json application/xml image/svg+xml
Header append Vary Accept-Encoding
EOF
sudo systemctl reload apache2

Step 5: Detect anomalies and close the loop

Use a simple per-minute rollup to spot spikes in p95 latency or error rates and revert to a safer config.

Save as detect_anomalies.py:

#!/usr/bin/env python3
import re, sys, datetime as dt
import numpy as np, pandas as pd
from sklearn.ensemble import IsolationForest

log_path = sys.argv[1] if len(sys.argv) > 1 else "/var/log/apache2/access_ai.log"
rx = re.compile(r'''\S+ \S+ \S+ \[(?P<ts>[^\]]+)\] "\S+ \S+[^"]*" (?P<status>\d{3}) \S+ "[^"]*" "[^"]*" (?P<usec>\d+) "[^"]*"''')

rows=[]
with open(log_path,'r',errors='ignore') as f:
    for line in f:
        m=rx.search(line)
        if not m: continue
        ts=dt.datetime.strptime(m['ts'].split()[0], "%d/%b/%Y:%H:%M:%S")
        rows.append({'ts': ts,'status': int(m['status']),'lat_ms': int(m['usec'])/1000.0})

df=pd.DataFrame(rows)
if df.empty:
    print("No data"); sys.exit(0)

g=(df.set_index('ts')
    .resample('1min')
    .agg(rps=('lat_ms','count'),
         p95_ms=('lat_ms', lambda s: np.percentile(s,95) if len(s)>0 else 0),
         err_rate=('status', lambda s: (s.ge(500).sum()/len(s))*100 if len(s)>0 else 0))
    .fillna(0))

X=g[['rps','p95_ms','err_rate']].values
clf=IsolationForest(contamination=0.03, random_state=42).fit(X)
g['anomaly']=(clf.predict(X)==-1)

print(g[g['anomaly']].tail(5))

Cron it and on anomaly, roll back to a “safe” config:

# Example: cron entry every 5 minutes
*/5 * * * * /usr/bin/python3 /root/detect_anomalies.py /var/log/apache2/access_ai.log | grep True && \
  a2disconf mpm_event_tuned && systemctl reload apache2
# Fedora/RHEL: use httpd and adjust paths

Real-world example (what to expect)

• Before: p95 latency ~950 ms at 80 RPS peak, HTTP/1.1 only, prefork MPM, KeepAliveTimeout 10, no compression.

• After AI pass: event MPM with ThreadsPerChild 50, ServerLimit 8, MaxRequestWorkers 400; KeepAliveTimeout 3; HTTP/2 enabled; Brotli for text types; caching a popular public JSON stats endpoint for 5 minutes.

• Result: p95 latency ~350 ms at 110 RPS peak, CPU ~15% lower, network egress for text down ~25–35%, fewer worker exhaustion warnings under bursts.

Your mileage will vary, but the workflow is repeatable and data-backed.

Common gotchas

• If you serve PHP via mod_php, switch to PHP-FPM before moving to event MPM.

• Cache only user-agnostic resources—strip Set-Cookie and beware private data.

• Brotli may be unavailable on some RHEL variants without EPEL; keep Deflate as fallback.

• Always keep a “last known good” config you can toggle with a2enconf/a2disconf or conf.d include files.

Conclusion and next steps

You just built a feedback loop:

• Observe with server-status and detailed timing logs.

• Learn from real traffic using a small, auditable model.

• Apply targeted changes to MPM, KeepAlive, HTTP/2, caching, and compression.

• Watch for anomalies and roll back automatically.

Next steps:

• Schedule the learning script nightly and commit generated recommendations to git.

• Extend features (e.g., route groups, backend identifiers) and model selection.

• Add visualization (e.g., gnuplot or a lightweight Grafana+Prometheus exporter) for the metrics you computed.

If you want, I can help you adapt the scripts to your exact workload and generate tuned configs based on a few minutes of your logs.