Posted on
• Artificial Intelligence

Artificial Intelligence Performance Baselines

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Performance Baselines on Linux: A Bash-First Playbook

Ever noticed your model running 20–30% slower “this week” and no one can agree on why? Kernel updates, BLAS swaps, power profiles, or a sneaky driver change can crater AI performance without touching a single line of code. The fix is not guesswork; it’s reproducible, Linux-native baselining.

This post gives you a practical, Bash-first workflow to create and maintain AI performance baselines. You’ll install a minimal toolkit, fingerprint your machine, define stable metrics, run a real model benchmark, and store results for easy comparison.

Why baselines matter (and why Linux helps)

  • AI stacks are layered: kernel, drivers, compilers, BLAS, runtimes, frameworks, models. Any layer can shift performance.

  • Linux gives you first-class introspection and control: CPU affinity, NUMA locality, kernel counters, and standardized package management.

  • Baselines turn “feels slower” into “p50 latency regressed 12.4% after kernel 6.8.x; IPC dropped; L3 misses climbed 18%.” That’s how teams fix problems fast.

1) Install the minimal benchmarking toolkit

You’ll use:

  • hyperfine: quick, statistically sound wall-clock benchmarking

  • perf: CPU performance counters (cycles, instructions, cache, branch)

  • numactl: NUMA and CPU pinning

  • lm-sensors: temperature and power (where supported)

  • Python + pip: to run a simple ONNX Runtime model benchmark

  • git: to version your baseline data

  • podman: optional, for containerized, repeatable environments

Install with your package manager:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip git hyperfine numactl lm-sensors podman \
  linux-tools-common linux-tools-generic linux-tools-$(uname -r)

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y \
  python3 python3-pip git hyperfine numactl perf lm_sensors podman

openSUSE Leap/Tumbleweed (zypper):

sudo zypper install -y \
  python3 python3-pip git hyperfine numactl perf lm_sensors podman

First-time sensors setup (optional but useful):

sudo sensors-detect --auto
sensors

perf permissions (if you get “permission denied”):

# Temporarily lower restrictions (resets on reboot)
echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid

2) Freeze your environment and fingerprint the machine

Do this before you run a single benchmark; record it with your results.

Machine fingerprint:

# OS, kernel, CPU, memory, NUMA topology, microcode
cat /etc/os-release
uname -a
lscpu
free -h
numactl -H
dmesg | grep -i microcode | tail -n 1 || true

Pin CPU/NUMA placement for reproducibility:

# Example: pin to CPU cores 0–7 on NUMA node 0 (adjust to your system)
export CPUSET=0-7
export NUMANODE=0

Control threading (most AI libs respect these):

export OMP_NUM_THREADS=8
export MKL_NUM_THREADS=8
export OPENBLAS_NUM_THREADS=8
export NUMEXPR_NUM_THREADS=8
# Optional: stable thread placement for Intel MKL/OpenMP
export KMP_AFFINITY=granularity=fine,compact,1,0

Optional power stability (if allowed):

# Force "performance" scaling governor (root; may vary by distro)
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

3) Define your core metrics and a repeatable run command

Track at least:

  • Latency (p50, p90) and throughput (samples/sec)

  • CPU counters (cycles, instructions, IPC, cache-misses)

  • Peak memory and temperature (helps spot throttling)

Wrap your workload with hyperfine for wall-clock numbers:

hyperfine --warmup 3 --runs 10 --style basic 'taskset -c ${CPUSET} numactl -N ${NUMANODE} -m ${NUMANODE} ./run_workload.sh'

Add perf stat for micro-architectural insight:

perf stat -e cycles,instructions,cache-misses,branches,branch-misses \
  taskset -c ${CPUSET} numactl -N ${NUMANODE} -m ${NUMANODE} ./run_workload.sh

Quick thermal check while running:

watch -n 1 sensors

4) Real-world example: ONNX Runtime ResNet50 (CPU)

You’ll benchmark image inference on CPU using ONNX Runtime. Results generalize to other models/runtimes.

Create a Python virtual environment:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install onnxruntime numpy

Download a model and create a tiny runner:

curl -L -o resnet50-v1-12.onnx https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx

Create bench_onnx.py:

#!/usr/bin/env python3
import argparse, time, statistics, os
import numpy as np
import onnxruntime as ort

def run(batch, iters, warmup):
    so = ort.SessionOptions()
    sess = ort.InferenceSession("resnet50-v1-12.onnx", sess_options=so, providers=["CPUExecutionProvider"])
    input_name = sess.get_inputs()[0].name
    x = np.random.rand(batch, 3, 224, 224).astype(np.float32)

    # Warmup
    for _ in range(warmup):
        sess.run(None, {input_name: x})

    times = []
    for _ in range(iters):
        t0 = time.perf_counter()
        sess.run(None, {input_name: x})
        times.append(time.perf_counter() - t0)

    lat_ms = [t*1000 for t in times]
    p50 = statistics.median(lat_ms)
    p90 = np.percentile(lat_ms, 90)
    thr = (batch * iters) / sum(times)
    print(f"batch={batch} iters={iters} p50_ms={p50:.2f} p90_ms={p90:.2f} thr_sps={thr:.2f}")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--batch", type=int, default=1)
    ap.add_argument("--iters", type=int, default=50)
    ap.add_argument("--warmup", type=int, default=10)
    args = ap.parse_args()
    # Optional: fixed seed for determinism of data shapes (not timing)
    np.random.seed(int(os.environ.get("BASELINE_SEED", "1234")))
    run(args.batch, args.iters, args.warmup)

Make it executable:

chmod +x bench_onnx.py

Baseline run (pin CPU/NUMA, set threads):

export CPUSET=0-7
export NUMANODE=0
export OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8

taskset -c ${CPUSET} numactl -N ${NUMANODE} -m ${NUMANODE} \
  ./bench_onnx.py --batch 1 --iters 100

Compare batches and thread counts with hyperfine:

hyperfine --warmup 3 --runs 10 --export-json baseline.json \
  'OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 taskset -c 0-3 numactl -N 0 -m 0 ./bench_onnx.py --batch 1 --iters 50' \
  'OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 1 --iters 50' \
  'OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 8 --iters 50'

Add perf context to a single representative command:

perf stat -e cycles,instructions,cache-misses,branches,branch-misses \
  taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 8 --iters 200

Tip: Repeat the baseline after any of these change:

  • Kernel or microcode updates

  • BLAS/oneDNN/MKL/OpenBLAS updates

  • ONNX Runtime/PyTorch/TensorFlow upgrades

  • Power/thermal firmware changes

  • Docker/Podman base image updates

Optional: run in a container for stronger isolation:

podman run --rm -it -v "$(pwd)":/work -w /work docker.io/library/python:3.11-slim bash -lc '
  apt-get update && apt-get install -y --no-install-recommends numactl && \
  pip install --no-cache-dir onnxruntime numpy && \
  OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 \
  taskset -c 0-7 numactl -N 0 -m 0 python bench_onnx.py --batch 8 --iters 200
'

5) Store and compare results like code

Make a simple baseline directory that includes environment and results:

stamp=$(date +%Y%m%d_%H%M%S)
host=$(hostname -s)
mkdir -p baselines/${host}_${stamp}
# Fingerprint
{
  echo "# OS/KERNEL"
  cat /etc/os-release
  uname -a
  echo "# CPU/MEM/NUMA"
  lscpu
  free -h
  numactl -H
} > baselines/${host}_${stamp}/system.txt

# Metrics
hyperfine --warmup 3 --runs 10 --export-json baselines/${host}_${stamp}/hyperfine.json \
  'OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 1 --iters 100' \
  'OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 8 --iters 100'

# perf (single representative run)
perf stat -x, -o baselines/${host}_${stamp}/perf.csv \
  -e cycles,instructions,cache-misses,branches,branch-misses \
  taskset -c 0-7 numactl -N 0 -m 0 ./bench_onnx.py --batch 8 --iters 500

Track over time with git:

git init
git add baselines
git commit -m "Add first AI baseline ${host} ${stamp}"

Later, to see regressions, diff hyperfine JSONs (jq helps) or just re-run and compare p50/p90/throughput and perf counters. If p50 latency regresses while IPC drops and cache-misses climb, suspect memory locality or a library change. If temps spike, check throttling.

Extra: What about GPUs?

This post focused on CPU to stay distro-portable. For GPU baselines, keep the same structure:

  • Use vendor-recommended drivers and monitoring tools (nvidia-smi for NVIDIA, rocm-smi for AMD, intel_gpu_top for Intel).

  • Pin driver/runtime versions and collect GPU clocks, power, and memory bandwidth data during runs.

  • Keep CPU pinning and NUMA locality in mind; CPU-side input pipelines can bottleneck GPU throughput.

Because GPU driver installs vary by distro and vendor repos, follow your vendor’s official instructions and then integrate those tools into the same hyperfine/perf/sensors workflow.

Conclusion and next steps

Performance confidence is a feature. With a small, Linux-native toolkit and a few disciplined steps, you can detect regressions early, explain them with data, and ship with speed.

Your next move:

  • Copy the commands above into a baseline.sh script and run it today on your primary training/inference hosts.

  • Commit system fingerprints and results to git.

  • Re-run after each weekly update window or CI image refresh.

  • Extend the example to your real workloads (PyTorch, TensorFlow, tokenizers, data loaders) while keeping the same shell scaffolding.

If you want a hand turning this into a cron job or CI step, ask—and we’ll template it for your stack.

Related posts