Posted on
• Artificial Intelligence

Artificial Intelligence Reverse Proxy Analysis

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Reverse Proxy Analysis: Control, Observe, and Optimize Your LLM Traffic

If your AI bill surprises you each month or your team loses hours debugging flaky model calls, you’re not alone. As AI adoption explodes, the simple “call the model endpoint directly” approach becomes a liability. A reverse proxy in front of your AI providers and local model servers gives you a single, battle-tested control plane to secure keys, enforce rate limits, fail over across providers, and analyze real usage. In this article, we’ll build a practical AI reverse proxy, show how to observe it from Bash, and give you steps you can apply today on any Linux distro.

Why a reverse proxy for AI is worth it

  • Security and governance: Centralize API key handling, restrict access, and log all calls for compliance without refactoring every client.

  • Reliability and portability: Fail over between cloud AI (e.g., OpenAI/Azure) and on-prem models (vLLM/Ollama) without changing clients.

  • Cost and performance control: Rate-limit abusers, split-traffic to compare models, and cache/short-circuit health checks.

  • Observability: Structured logs and metrics let you see P95 latency, error spikes, and usage by endpoint or team.

The best part: you can do this with standard Linux tools (Nginx/HAProxy/Caddy), a few config snippets, and Bash-friendly analysis.

1) Install the tooling (Nginx/HAProxy/Caddy + CLI analyzers)

You only need one proxy to start. Nginx is a common default, but HAProxy and Caddy are excellent too. We’ll also use jq and GoAccess for log analysis.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y nginx haproxy caddy jq goaccess certbot python3-certbot-nginx

Fedora/RHEL/CentOS Stream (dnf):

# On RHEL/CentOS you may need: sudo dnf install -y epel-release
sudo dnf install -y nginx haproxy caddy jq goaccess certbot python3-certbot-nginx

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y nginx haproxy caddy jq goaccess certbot python3-certbot-nginx

Enable and start what you choose (example: Nginx):

sudo systemctl enable --now nginx

Optional: open the firewall (example using firewalld):

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

2) Core Nginx setup for AI endpoints (secure, model-aware, observable)

This Nginx config does the following:

  • Terminates TLS (optional snippet for Let’s Encrypt shown below)

  • Proxies OpenAI-compatible traffic to multiple backends (cloud and local)

  • Adds per-key rate limiting (by Authorization header)

  • Produces both human-friendly and JSON logs for Bash analysis

  • Preps for A/B split and failover between providers

Create a site config (replace your domain and backends):

sudo mkdir -p /etc/nginx/ai.d
sudo nano /etc/nginx/ai.d/ai.conf

Paste:

# /etc/nginx/ai.d/ai.conf

# 1) Define upstreams: a cloud AI provider and a local LLM server (e.g., vLLM or Ollama proxy)
upstream openai_api {
    zone openai_zone 64k;
    server api.openai.com:443 max_fails=3 fail_timeout=30s;
    keepalive 64;
}

upstream local_llm {
    zone local_zone 64k;
    # Example local server: vLLM/OpenAI-compatible on 127.0.0.1:8000
    server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

# 2) Rate limit per API key (Authorization: Bearer ...)
# Store ~10MB of keys, limit to 60 requests/minute per key.
limit_req_zone $http_authorization zone=per_key:10m rate=60r/m;

# 3) Structured logging for analysis
# Combined format (for GoAccess)
log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"';

# JSON format (for jq and custom tooling)
log_format ai_json escape=json
  '{ "time":"$time_iso8601", "remote_addr":"$remote_addr", '
  '"method":"$request_method", "path":"$uri", "status":$status, '
  '"bytes":$bytes_sent, "req_time":$request_time, '
  '"upstream":"$upstream_addr", "upstream_time":"$upstream_response_time", '
  '"request_id":"$request_id", "content_length":"$http_content_length", '
  '"client":"$http_user_agent" }';

# 4) Choose a backend dynamically: 70% to cloud, 30% to local (A/B)
# We use a split to a variable that contains a full "http://upstream" string.
split_clients "$request_id$remote_addr" $ai_backend {
  70% "https://openai_api";
  *   "http://local_llm";
}

server {
    listen 80;
    server_name ai.yourdomain.com;

    # Let’s Encrypt HTTP-01 challenge
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    # Redirect to HTTPS
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name ai.yourdomain.com;

    # TLS certs (replace with your actual cert files or use certbot below)
    ssl_certificate     /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem;

    # Access logs
    access_log /var/log/nginx/ai_access.log main;
    access_log /var/log/nginx/ai_access.json ai_json;
    error_log  /var/log/nginx/ai_error.log warn;

    # Security and proxy tuning
    client_max_body_size 10m;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Request-ID $request_id;

    # Enforce presence of Authorization header
    if ($http_authorization = "") {
        return 401;
    }

    location /v1/ {
        # Per-key rate limit, bursts allowed but with no delay
        limit_req zone=per_key burst=10 nodelay;

        # Timeouts and retries; fail over on gateway/timeout errors
        proxy_read_timeout  300s;
        proxy_connect_timeout 30s;
        proxy_send_timeout  300s;
        proxy_next_upstream error timeout http_502 http_503 http_504;

        # Choose backend; for HTTPS upstream we must pass to the upstream group with proper scheme.
        # If proxying to OpenAI directly, ensure TLS SNI and ALPN:
        proxy_ssl_server_name on;

        # Some AI endpoints require Authorization to be passed through:
        proxy_set_header Authorization $http_authorization;

        # Route by A/B decision
        proxy_pass $ai_backend;
    }
}

Enable and test:

sudo ln -s /etc/nginx/ai.d/ai.conf /etc/nginx/conf.d/ai.conf
sudo nginx -t
sudo systemctl reload nginx

Obtain and install TLS certificate with Certbot (optional but recommended):

# Prepare webroot directory for ACME challenges
sudo mkdir -p /var/www/html

# Issue and install cert; Certbot will edit TLS paths for you
sudo certbot --nginx -d ai.yourdomain.com --redirect --agree-tos -m you@example.com

Notes:

  • Swap openai_api for any HTTPS upstream (Azure OpenAI, Anthropic via a gateway, etc.).

  • Swap local_llm for vLLM/Ollama/OpenAI-compatible servers you run internally.

  • Move the split_clients weights to bias traffic to the cheapest/fastest backend, then measure.

Real-world example: When a cloud outage hits, proxy_next_upstream fails over to the local LLM for 502/504 errors, so clients remain functional.

3) Alternative: HAProxy snippet for per-key rate limiting and headers

If you prefer HAProxy, this skeleton shows header-based auth enforcement and per-key throttling via stick tables:

# /etc/haproxy/haproxy.cfg (add to your frontend/backend)

global
  log /dev/log local0
  maxconn 50000

defaults
  mode http
  timeout connect 30s
  timeout client  300s
  timeout server  300s
  option httplog

frontend ai_front
  bind *:443 ssl crt /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem crt-key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem
  http-response add-header X-Proxy-Request-ID %[unique-id]
  acl missing_auth hdr_cnt(Authorization) eq 0
  http-request deny status 401 if missing_auth

  # Track per-key req rate (Authorization header used as key)
  stick-table type string len 128 size 1m expire 1m store http_req_rate(60s)
  http-request track-sc0 hdr(Authorization)
  acl too_fast sc_http_req_rate(0) gt 60
  http-request deny status 429 if too_fast

  # Simple routing: 70% cloud, 30% local
  use_backend openai_api if { rand(100) lt 70 }
  default_backend local_llm

backend openai_api
  server s1 api.openai.com:443 ssl verify none alpn h2,http/1.1

backend local_llm
  server s1 127.0.0.1:8000

Reload:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

4) Analyze AI traffic from Bash: latency, error spikes, and hot endpoints

You now have two access logs from Nginx:

  • /var/log/nginx/ai_access.log (combined format, GoAccess-friendly)

  • /var/log/nginx/ai_access.json (NDJSON, jq-friendly)

Examples:

Top called endpoints:

awk '{print $7}' /var/log/nginx/ai_access.log | sort | uniq -c | sort -nr | head

Error rate in the last 5 minutes:

sudo journalctl -u nginx --since "5 min ago" | grep -c " 5[0-9][0-9] "

P95 request time from the JSON log (simple awk percentile):

cat /var/log/nginx/ai_access.json \
  | jq -r '.req_time' \
  | sort -n \
  | awk ' {a[NR]=$1} END{ if (NR>0) {p=int(0.95*NR); if(p<1)p=1; print a[p]} }'

Requests per minute trend:

grep "$(date '+%d/%b/%Y:%H:%M')" /var/log/nginx/ai_access.log \
  | awk '{print substr($4,2,17)}' \
  | sort | uniq -c | tail -n 20

Interactive dashboard with GoAccess (serves a live HTML report on port 7890):

sudo goaccess /var/log/nginx/ai_access.log \
  --log-format=COMBINED \
  --real-time-html -o /var/www/html/ai-report.html

# Serve the report with any static server, or:
sudo python3 -m http.server 7890 --directory /var/www/html

Tip: If your upstreams add headers like X-Usage-Tokens or X-Model, you can log them by adding to the JSON log format:

"$upstream_http_x_usage_tokens"

Then slice by token usage per route/team with jq.

Log rotation (avoid giant files):

sudo nano /etc/logrotate.d/nginx-ai

/var/log/nginx/ai_access.log /var/log/nginx/ai_access.json {
  daily
  rotate 14
  compress
  missingok
  notifempty
  create 0640 www-data adm
  sharedscripts
  postrotate
    [ -s /run/nginx.pid ] && kill -USR1 $(cat /run/nginx.pid)
  endscript
}

5) Practical hardening and routing patterns

  • Lock down who can call your proxy:

    • Restrict by IP/CIDR:
    allow 10.0.0.0/8;
allow 192.168.0.0/16;
deny all;

  • Normalize upstream timeouts for long generations (e.g., 300s) to avoid premature 504s.

  • Use request IDs everywhere (we added X-Request-ID) to correlate client logs and proxy logs.

  • Test failover: intentionally drop the cloud upstream to see traffic move to local LLM seamlessly.

  • Separate per-team prefixes: route /teamA/v1 to a specific upstream, apply a stricter per-key rate limit with a second limit_req_zone, and allocate budgets operationally.

Real-world example: A platform team split 10% of chat completions to a cheaper local LLM for low-priority workloads while keeping 90% on a premium provider for critical paths. They dropped average costs by 22% in the first week, then tuned the split further as latency and quality data accumulated.

Troubleshooting quick hits

  • 401s immediately: ensure clients send Authorization: Bearer and that your proxy doesn’t strip it.

  • 502/504s: raise proxy_read_timeout and confirm your upstream supports streaming if you need it.

  • TLS SNI issues upstream: set proxy_ssl_server_name on; (as shown) and proxy_ssl_name if the hostname differs.

  • CORS for browser apps: add:

    add_header Access-Control-Allow-Origin *;
add_header Access-Control-Allow-Headers Authorization,Content-Type;
if ($request_method = OPTIONS) { return 204; }

Conclusion and next steps

Reverse proxies turn AI chaos into control: one endpoint, many providers, with security, limits, failover, and clear analytics. Start with the Nginx config above, point a single client at it, and watch your logs. In an afternoon you can:

  • Protect keys and stop runaway costs

  • Observe latency and error rates

  • Split and compare models without changing client code

Call to action:

  • Install Nginx and jq/goaccess on your distro using the commands above.

  • Drop in the sample config, reload, and send a test request.

  • Add one improvement at a time: rate limits, A/B routing, then dashboards.

Your AI stack will be faster to debug, cheaper to run, and easier to govern—without touching every application.

Related posts