Artificial Intelligence Query Performance Analysis from the Linux Shell

If your AI app feels fast in dev but crawls in prod, it’s usually not the model—it’s the query pipeline. The good news: you don’t need a dashboard or heavyweight APM to diagnose it. With a few standard Linux tools and some Bash, you can measure, explain, and fix most performance issues in AI endpoints—local or remote.

This article gives you a practical, repeatable way to analyze AI query performance from the command line. You’ll learn what to measure, how to collect it, and how to act on it.

Why this matters

• AI requests are compound operations: prompt serialization, network, authentication, scheduling, inference, token generation/streaming, and response assembly. Small inefficiencies add up.

• Latency spikes (p95/p99) kill UX and SLAs more than mean latency does.

• Throughput and concurrency drive cost and capacity planning.

• A simple, scriptable test harness lets you reproduce issues, compare changes, and enforce performance baselines in CI.

What you’ll install

We’ll use only widely available packages.

• curl: low-level HTTP client with fine-grained timing

• jq: JSON processing

• ApacheBench (ab): quick concurrency/load test

• GNU parallel: drive controlled concurrency

• sysstat (pidstat, iostat, sar): CPU, I/O, and network observability

• iperf3: network throughput sanity check

• time: measure process time

Install on your distro (choose one):

apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y curl jq apache2-utils iperf3 sysstat time parallel
sudo systemctl enable --now sysstat

dnf (Fedora/RHEL/CentOS Stream):

sudo dnf install -y curl jq httpd-tools iperf3 sysstat time parallel
sudo systemctl enable --now sysstat

zypper (openSUSE/SLES):

sudo zypper install -y curl jq apache2-utils iperf3 sysstat time parallel
sudo systemctl enable --now sysstat

Note: ApacheBench is provided by apache2-utils (apt/zypper) or httpd-tools (dnf).

---

Step 1: Build a minimal, repeatable workload

Define your target endpoint and prompts. These examples assume an OpenAI-compatible JSON API, but the approach works for any HTTP endpoint.

Export your settings:

export AI_URL="http://localhost:8000/v1/chat/completions"  # or your remote endpoint
export AI_KEY=""                                           # e.g. "sk-..." if required
export MODEL="my-model"                                    # change to match your server

Create a prompts file:

cat > prompts.txt << 'EOF'
Explain the moon phases in one sentence.
List three Linux performance tools and when to use them.
Summarize the difference between batch and streaming AI inference.
Give a 2-line Bash one-liner example that parses JSON.
What is the trade-off between max_tokens and latency?
EOF

Create a one-request script that sends a single prompt and emits newline-delimited JSON (NDJSON) with curl timing:

cat > one_request.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

PROMPT="${1:-Explain the moon phases in one sentence.}"
AI_URL="${AI_URL:-http://localhost:8000/v1/chat/completions}"
AI_KEY="${AI_KEY:-}"
MODEL="${MODEL:-my-model}"

payload="$(jq -nc --arg m "$MODEL" --arg p "$PROMPT" \
  '{model:$m, messages:[{role:"user", content:$p}], stream:false, max_tokens:128, temperature:0}')"

# Emit key timing fields as NDJSON. Drop body to keep logs small.
curl -sS -X POST "$AI_URL" \
  -H "Content-Type: application/json" \
  ${AI_KEY:+-H "Authorization: Bearer '"$AI_KEY"'"} \
  --data "$payload" \
  -o /dev/null \
  -w '{"ts":"%{time_total}","ttfb":"%{time_starttransfer}","connect":"%{time_connect}","dns":"%{time_namelookup}","bytes":%{size_download},"code":%{http_code}}\n'
EOF
chmod +x one_request.sh

Smoke-test:

./one_request.sh

You should see one JSON line with fields like total time, TTFB, status code.

---

Step 2: Establish baselines and latency distributions

Run multiple requests to get statistically meaningful results.

Single-threaded (sequential):

> results.ndjson
while read -r p; do ./one_request.sh "$p" >> results.ndjson; done < prompts.txt

Concurrent with GNU parallel (10 workers, 200 total requests, sampling prompts):

shuf -r -n 200 prompts.txt | parallel -j 10 ./one_request.sh >> results.ndjson

Summarize with jq, sort, and awk:

# Extract totals, compute p50/p95/p99 (seconds), and mean.
jq -r '.ts' results.ndjson | sort -n | awk '
  {a[NR]=$1; sum+=$1} END{
    if (NR==0) exit;
    p=function(p){i=int((NR*p)+0.5); if(i<1)i=1; if(i>NR)i=NR; return a[i]};
    printf("count=%d mean=%.4fs p50=%.4fs p95=%.4fs p99=%.4fs\n",
      NR, sum/NR, p(0.50), p(0.95), p(0.99));
  }'

Check where time is going by averaging phases:

jq -r '[.dns,.connect,.ttfb,.ts] | @tsv' results.ndjson | \
awk -F'\t' '{dns+=$1; conn+=$2; ttfb+=$3; total+=$4; n++}
END{printf("avg_dns=%.4fs avg_connect=%.4fs avg_ttfb=%.4fs avg_total=%.4fs\n", dns/n, conn/n, ttfb/n, total/n)}'

Optional: a quick concurrency/load probe with ApacheBench (great for non-streaming endpoints and fixed payloads):

cat > payload.json <<'EOF'
{"model":"my-model","messages":[{"role":"user","content":"Summarize this in 1 sentence."}],"stream":false,"max_tokens":64,"temperature":0}
EOF

# 1000 requests, concurrency 50, keep-alive on
ab -k -n 1000 -c 50 -p payload.json -T application/json "$AI_URL"

Note: ab doesn’t support HTTP/2 or server-sent events; results are still useful for quick HTTP/1.1 latency/throughput baselines.

What good looks like:

• p95 within your SLO

• TTFB roughly equals model scheduling + first token generation + network

• total ≈ TTFB + remaining tokens; if total » TTFB with small outputs, suspect network, chunking, or server-side buffering

---

Step 3: Observe the system while you load it

While the test runs, correlate system metrics every 1s. Replace my_server with your process name or PID.

CPU, run queue, context switches by process:

pidstat -urd -h 1
# Or target a PID: pidstat -urd -p $(pgrep -d, -f my_server) 1

Disk I/O saturation (await, svctm, util%):

iostat -xz 1

Network device throughput and errors:

sar -n DEV 1

If you use an NVIDIA GPU for inference (optional):

nvidia-smi dmon -s pucm -d 1

Correlate symptoms:

• High CPU% + high run queue: CPU-bound; consider batch size, quantization, or more workers

• High GPU util + low SM occupancy or memory throttle: tune batch/token settings, check PCIe/bus limits

• High iowait: storage bottleneck (model paging, logging)

• Network drops/retransmits: SLOs dominated by transport issues

---

Step 4: Sanity-check the network path

Is the network the bottleneck or just noisy?

Measure raw throughput and latency headroom (where possible you need server-side iperf3):

# Server (on remote host with access):
iperf3 -s

# Client:
iperf3 -c <server_ip_or_dns>

Check TLS/connect overhead with curl:

curl -sS -o /dev/null -w 'dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s starttransfer=%{time_starttransfer}s total=%{time_total}s\n' "$AI_URL"

Tips:

• Prefer persistent connections (HTTP/1.1 keep-alive). For ab use -k. For curl, reuse connections by sending many requests to the same host in a single process.

• If you can enable HTTP/2 on your server, multiplexing reduces head-of-line blocking for concurrent prompts.

---

Step 5: Apply the optimization checklist

• Concurrency and worker pools

  • Match concurrency to your real workload. If p95 grows superlinearly with -c, you are saturating. Increase workers, shard across instances, or reduce per-request cost.

• Token and output control

  • Lower max_tokens, temperature, and stop sequences to reduce generation time when acceptable.

• Batching and scheduling (server-side)

  • Micro-batching can improve GPU utilization; measure p50/p95 impacts and cap queueing delay.

• Streaming vs non-streaming

  • Streaming (SSE/chunked) lowers TTFB at the cost of more network events; validate client-side rendering overhead and backpressure.

• Keep-alive and compression

  • Ensure keep-alive is on. For JSON payloads, enable gzip/deflate if large prompts (verify CPU overhead vs win).

• Hot paths and cold starts

  • Warm the model once before measuring; separate cold-start metrics from steady-state. Pin models to memory and avoid lazy loads where feasible.

• Logging and I/O

  • Excessive per-request logging (sync fs) inflates tail latency. Batch or sample logs; move to async.

Real-world example: reducing p95 by 40%

• Baseline: p50 0.8s, p95 2.2s at c=20; pidstat shows frequent context switches and 30% iowait during traffic spikes.

• Action: Disable synchronous request logging and move to buffered logs; preload model on startup; enable keep-alive.

• Result: p50 0.6s, p95 1.3s at c=20; iowait drops near zero; ab RPS improves by ~25%.

---

Putting it together: a quick, scriptable harness

Full example run (10 workers, 300 requests), then summarize:

> results.ndjson
shuf -r -n 300 prompts.txt | parallel -j 10 ./one_request.sh >> results.ndjson

echo "Latency distribution:"
jq -r '.ts' results.ndjson | sort -n | awk '
  {a[NR]=$1; sum+=$1} END{
    if (NR==0) exit;
    p=function(p){i=int((NR*p)+0.5); if(i<1)i=1; if(i>NR)i=NR; return a[i]};
    printf("count=%d mean=%.3fs p50=%.3fs p90=%.3fs p95=%.3fs p99=%.3fs\n",
      NR, sum/NR, p(0.50), p(0.90), p(0.95), p(0.99));
  }'

echo "Phase averages:"
jq -r '[.dns,.connect,.ttfb,.ts] | @tsv' results.ndjson | \
awk -F'\t' '{dns+=$1; conn+=$2; ttfb+=$3; total+=$4; n++}
END{printf("avg_dns=%.3fs avg_connect=%.3fs avg_ttfb=%.3fs avg_total=%.3fs\n", dns/n, conn/n, ttfb/n, total/n)}'

---

Conclusion and next steps

AI query performance is measurable, explainable, and improvable with the Linux tools you already have. Start by: 1) Capturing per-request timings with curl, 2) Driving realistic concurrency with parallel or ab, 3) Watching system and network metrics with sysstat and iperf3, 4) Iterating on a focused optimization checklist.

Call to action:

• Turn this into a repo-local “perf harness” for your service.

• Add it to CI to block regressions on p95/p99.

• Share your before/after numbers with your team and make performance a feature, not an afterthought.

If you want a follow-up, ask for a shell-only dashboard script that tails results.ndjson and renders live percentiles and resource metrics side by side.