Posted on
• Artificial Intelligence

Artificial Intelligence Hardware Benchmarks

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Hardware Benchmarks on Linux: A Bash-first Guide

Stop guessing which GPU or CPU is faster for your AI workloads. If you’re not benchmarking, you’re just believing marketing. This guide shows how to run reproducible, Linux-native AI hardware benchmarks from the command line, compare results across machines, and make better hardware decisions for training and inference.

You’ll learn:

  • What to measure (and why)

  • How to prepare your system for fair, apples-to-apples runs

  • How to run practical, open-source benchmarks (LLM kernels and framework inference)

  • How to capture telemetry (power, clocks, temps) alongside performance

  • Installation commands for apt, dnf, and zypper at every step

Why AI hardware benchmarks matter

  • Real workloads vary: small-batch, low-latency inference vs. large-batch offline scoring vs. training — each stresses hardware differently.

  • Cost and power are now first-class metrics. The “fastest” device may be too power-hungry or expensive for your use case.

  • Vendor claims aren’t your workloads. Standardized, reproducible tests on your models are the only reliable basis for decisions.

  • Linux is the de facto platform for AI — and Bash gives you consistent, scriptable control.

What to measure (and the traps to avoid)

Key metrics:

  • Throughput (samples/s, tokens/s)

  • Latency (p50/p95/p99)

  • Power and efficiency (W, joules per sample/token)

  • Memory behavior (peak usage, bandwidth pressure)

  • Accuracy/quality (don’t “win” by silently changing precision or numerics)

Control variables (for fairness):

  • Same model, precision, batch/sequence lengths

  • Same kernels/back-ends (e.g., cuBLAS vs. CUTLASS vs. OpenBLAS)

  • Same software stack versions and driver settings

  • Same power/perf modes and thermal conditions

Install the essentials

You’ll need compilers, CMake, Python, and a few system tools.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential git cmake pkg-config \
  python3 python3-venv python3-pip \
  libopenblas-dev wget curl jq
# perf and turbostat (kernel tools) for your running kernel:
sudo apt install -y linux-tools-common linux-tools-$(uname -r)
  • Fedora/RHEL (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake pkgconf-pkg-config \
  python3 python3-pip python3-virtualenv \
  openblas-devel wget curl jq perf kernel-tools
  • openSUSE (zypper):
sudo zypper install -y -t pattern devel_C_C++
sudo zypper install -y git cmake pkg-config \
  python3 python3-pip python3-virtualenv \
  openblas-devel wget curl jq perf kernel-tools

Optional GPU monitoring utilities:

  • Cross-vendor process/usage: nvtop

  • NVIDIA: nvidia-smi (comes with NVIDIA driver)

  • Intel iGPU: intel_gpu_top (intel-gpu-tools)

  • Debian/Ubuntu (apt):

sudo apt install -y nvtop intel-gpu-tools
# NVIDIA utilities come with the proprietary driver (e.g. ubuntu-drivers autoinstall)
  • Fedora/RHEL (dnf):
sudo dnf install -y nvtop intel-gpu-tools
  • openSUSE (zypper):
sudo zypper install -y nvtop intel-gpu-tools

Note: For NVIDIA, nvidia-smi is provided by the NVIDIA driver packages; install via your distro’s recommended method.

Step 1 — Prepare your system for fair, repeatable runs

  • Set the CPU scaling governor to “performance” (revert when done if needed):
for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
  echo performance | sudo tee "$f" >/dev/null
done
  • Optional: NVIDIA persistence and stable clocks (if supported):
sudo nvidia-smi -pm 1
# Optional, requires supported GPUs; example clock values must match your card
# sudo nvidia-smi --applications-clocks=memory_clock,graphics_clock
  • Verify idle baseline (CPU, GPU temps, clocks):
# CPU package power/temps snapshot (turbostat)
sudo turbostat --quiet --Summary --interval 5 --iterations 1

# NVIDIA quick view
nvidia-smi

# Cross-vendor GPU dashboard (interactive)
nvtop
  • System info snapshot (handy for logs):
echo "=== System ==="
uname -a
lsb_release -a 2>/dev/null || cat /etc/os-release
echo "CPU: $(lscpu | grep 'Model name' | sed 's/Model name:\s*//')"
echo "Mem: $(free -h | awk '/Mem:/ {print $2}')"
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader 2>/dev/null || true

Step 2 — Run a fast LLM kernel benchmark (llama.cpp)

Why: It’s a lightweight way to stress CPU/GPU math and memory paths used by modern LLMs without bringing in full framework overhead.

  • Build llama.cpp with OpenBLAS (CPU). GPU backends are also available if you have CUDA or others configured.
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -S . -B build -DGGML_OPENBLAS=ON
cmake --build build -j$(nproc)
  • Run the built-in kernel benchmark (no model file needed):
./build/bin/llama-bench --batch 512 --n-prompt 128 --n-gen 256 --threads $(nproc)
  • Compare across machines by keeping --batch, --n-prompt, --n-gen, and --threads the same. For GPU tests, rebuild with the appropriate backend flags (e.g., -DGGML_CUDA=ON if you have CUDA installed) and re-run llama-bench.

Tip: Track tokens/s and kernel timings. Higher is better for throughput; also watch CPU/GPU frequency and temps to ensure you’re not throttling.

Step 3 — Framework-level inference benchmark (ONNX Runtime)

Why: Framework-level runs reflect realistic operator fusion, memory planners, and backends your apps will use.

  • Create a Python virtual environment and install ONNX Runtime (CPU). For NVIDIA GPUs, you can use onnxruntime-gpu if your CUDA setup is working.

  • Debian/Ubuntu (apt):

python3 -m venv ~/venvs/ort
source ~/venvs/ort/bin/activate
pip install --upgrade pip
pip install onnxruntime onnx onnxruntime-tools
# For NVIDIA GPUs instead: pip install onnxruntime-gpu
  • Fedora/RHEL (dnf):
python3 -m venv ~/venvs/ort
source ~/venvs/ort/bin/activate
pip install --upgrade pip
pip install onnxruntime onnx onnxruntime-tools
# For NVIDIA GPUs instead: pip install onnxruntime-gpu
  • openSUSE (zypper):
python3 -m venv ~/venvs/ort
source ~/venvs/ort/bin/activate
pip install --upgrade pip
pip install onnxruntime onnx onnxruntime-tools
# For NVIDIA GPUs instead: pip install onnxruntime-gpu
  • Download a reference model (SqueezeNet):
mkdir -p ~/models && cd ~/models
wget -O squeezenet.onnx https://github.com/onnx/models/raw/main/vision/classification/squeezenet/model/squeezenet1.1-7.onnx
  • Minimal latency/throughput test script (CPU by default):
cat > ~/models/ort_bench.py << 'EOF'
import os, time, numpy as np, onnxruntime as ort

model = "squeezenet.onnx"
sess_opts = ort.SessionOptions()
providers = ort.get_available_providers()
sess = ort.InferenceSession(model, sess_options=sess_opts, providers=providers)

# Fake input: NCHW float32, SqueezeNet expects 224x224
def mk_input(batch=1):
    x = np.random.rand(batch, 3, 224, 224).astype(np.float32)
    return {sess.get_inputs()[0].name: x}

def run(n_warm=10, n_runs=50, batch=1):
    _ = [sess.run(None, mk_input(batch)) for _ in range(n_warm)]
    t0 = time.perf_counter()
    for _ in range(n_runs):
        sess.run(None, mk_input(batch))
    t1 = time.perf_counter()
    total = t1 - t0
    print(f"provider={providers}, batch={batch}, runs={n_runs}, "
          f"avg_latency_ms={(total/n_runs)*1000:.2f}, throughput_sps={(batch*n_runs)/total:.2f}")

for b in [1, 8, 32]:
    run(batch=b)
EOF
python3 ~/models/ort_bench.py

If using onnxruntime-gpu, verify the CUDA provider is in providers output. Keep the same batches across machines for fair comparison.

Step 4 — Capture telemetry alongside performance

Numbers without context mislead. Log power, clocks, and temps while the benchmark runs.

  • NVIDIA power/usage log while running llama.cpp (press Ctrl+C to stop after the bench completes):
nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,clocks.sm,clocks.mem,temperature.gpu,power.draw --format=csv -lms 500 | tee nvidia_telemetry.csv
  • CPU package power snapshot during ONNX run:
sudo turbostat --quiet --show PkgWatt,Bzy_MHz,IRQ --interval 1 --iterations 30 | tee turbostat.log
  • System-wide perf counters for 30s:
sudo perf stat -a -- sleep 30 2>&1 | tee perf_stat.txt

Automate telemetry + benchmark together:

cat > run_with_telemetry.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

logdir="${1:-logs_$(date +%Y%m%d_%H%M%S)}"
mkdir -p "$logdir"

# Start NVIDIA telemetry if available
if command -v nvidia-smi >/dev/null; then
  nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,clocks.sm,clocks.mem,temperature.gpu,power.draw \
    --format=csv -lms 500 > "$logdir/nvidia.csv" &
  nvgpu_pid=$!
fi

# Start turbostat
if command -v turbostat >/dev/null; then
  sudo turbostat --quiet --interval 1 > "$logdir/turbostat.log" &
  turbostat_pid=$!
fi

# Example benchmark (llama.cpp bench)
./build/bin/llama-bench --batch 512 --n-prompt 128 --n-gen 256 --threads $(nproc) | tee "$logdir/bench.txt"

# Cleanup
[[ -n "${nvgpu_pid:-}" ]] && kill $nvgpu_pid 2>/dev/null || true
[[ -n "${turbostat_pid:-}" ]] && sudo kill $turbostat_pid 2>/dev/null || true

echo "Logs saved to: $logdir"
EOF
chmod +x run_with_telemetry.sh
./run_with_telemetry.sh

Step 5 — Sanity-check and compare

  • Confirm you’re not thermally throttling:
grep -i throttl logs*/turbostat.log | head || true

  • Normalize by batch/sequence length, precision, and threads/backends.

  • Record software versions (framework, drivers, kernels) in your logs.

For repeatability, put your exact commands, environment variables, and git SHAs into a README alongside your logs.

Real-world scenarios

  • Low-latency API inference: Focus on p95 latency with batch=1 in ONNX Runtime. Watch GPU clocks (avoid downclocking between requests) and NUMA locality for CPU-bound stages.

  • Offline scoring: Test larger batches (e.g., 32/128). Throughput per watt is often the key KPI.

  • LLM serving: Use llama-bench to size CPU vs. GPU needs and to validate the effect of backend flags (OpenBLAS vs. CUDA kernels). If you deploy quantized models, benchmark with the same quantization.

Where standardized suites fit

MLPerf is the industry standard for rigorous, rules-based benchmarking. Use it as a north star to understand fair-comparison methodology. For day-to-day engineering, the targeted tests above get you quick, reproducible signals on your actual hardware and workloads.

Conclusion and next steps

You now have:

  • A reproducible kernel-level LLM benchmark (llama.cpp)

  • A framework-level inference benchmark (ONNX Runtime)

  • System preparation and telemetry practices that make results trustworthy

Your next step: 1) Run both benchmarks on your current and candidate machines. 2) Log everything (commands, versions, telemetry). 3) Decide using the metrics that matter to you: p95 latency, throughput per watt, or cost per QPS.

If you want a follow-up post with containerized MLPerf-style runs (Podman/Docker) and GPU backend comparisons (CUDA vs. ROCm vs. CPU), let me know which stacks you use most.

Related posts