Posted on
• Artificial Intelligence

Artificial Intelligence Inference Performance Tuning

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Inference Performance Tuning on Linux (Bash-first Guide)

You’ve trained a great model—now it needs to respond in milliseconds, not seconds. The fastest wins: lower latency, higher throughput, and lower cloud bills. The good news: a lot of speed comes from system- and runtime-level tuning you can do today with a terminal and a few environment variables.

This guide shows why inference tuning matters, what moves the needle on Linux, and how to do it—step by step—with actionable Bash and Python snippets. You’ll see quick wins from precision, threading, NUMA, batching, and graph optimization using common, open-source tooling.

Why this matters

  • Inference dominates AI costs once models hit production. Tuning often yields 2–10x throughput improvements with identical accuracy (or minimal losses with quantization).

  • Linux offers low-level levers (CPU governor, NUMA affinity, huge pages) that frameworks won’t set for you.

  • Modern runtimes (ONNX Runtime, OpenVINO, PyTorch) have powerful knobs—often off by default—that squeeze your hardware.

Prerequisites: system setup

Install core tools and libraries. Pick the commands for your distro.

  • Ubuntu/Debian (apt):
sudo apt-get update
sudo apt-get install -y build-essential cmake git python3 python3-venv python3-pip \
    numactl hwloc msr-tools linux-tools-common linux-tools-generic linux-tools-$(uname -r)
  • Fedora/RHEL/CentOS (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git python3 python3-pip numactl hwloc msr-tools kernel-tools
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y -t pattern devel_C_C++
sudo zypper install -y cmake git python3 python3-pip numactl hwloc msr-tools cpupower

Create and activate a Python virtual environment, then install runtimes:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

# CPU-focused stack
pip install onnx onnxruntime openvino

# Optional GPU support (NVIDIA): ensure proper drivers/CUDA, then
pip install onnxruntime-gpu

# For examples
pip install torch torchvision transformers

Notes:

  • For NVIDIA GPUs with PyTorch + CUDA, install the proper wheel per your CUDA version (see PyTorch docs). CPU-only PyTorch is fine for CPU tuning.

  • Vendor drivers and SDKs (e.g., CUDA, Intel oneAPI, ROCm) can provide further gains, but this guide stays on commonly available packages.

The core playbook (3–5 high-impact actions)

1) Pick the right runtime and precision

  • Use ONNX Runtime (ORT) or OpenVINO for optimized CPU inference. For many models, these outperform default framework eager mode.

  • Lower precision (INT8/FP16) often gives the biggest win per minute invested.

Export to ONNX and enable graph optimizations:

python - <<'PY'
import onnxruntime as ort

sess_opts = ort.SessionOptions()
sess_opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_opts.intra_op_num_threads = 8  # tune per CPU
sess_opts.inter_op_num_threads = 1

providers = ["CPUExecutionProvider"]  # or ["CUDAExecutionProvider"] if GPU-enabled
sess = ort.InferenceSession("model.onnx", sess_opts, providers=providers)
print("Providers:", sess.get_providers())
PY

Quick dynamic quantization to INT8 (weights) with ONNX Runtime:

python - <<'PY'
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic("model.onnx", "model.int8.onnx", weight_type=QuantType.QInt8)
print("Wrote: model.int8.onnx")
PY

OpenVINO CPU runtime (loads ONNX, compiles for CPU):

python - <<'PY'
from openvino.runtime import Core
ie = Core()
model = ie.read_model("model.onnx")
compiled = ie.compile_model(model, "CPU")  # auto-tuning inside OpenVINO
print("OpenVINO target:", compiled.properties)
PY

Tip: Start with ORT int8 quantization; if accuracy holds and latencies drop, consider more advanced static quantization or OpenVINO’s post-training optimization for further gains.

2) Threading, affinity, and CPU topology (NUMA)

Match your threading to your cores, pin the process/threads, and keep memory near compute.

  • Inspect topology:
lscpu
numactl --hardware
  • Pin to a NUMA node and its memory:
# Bind to node 0 CPU cores and memory on node 0
numactl --cpunodebind=0 --membind=0 python infer.py
  • Tune OpenMP and BLAS threads:
export OMP_NUM_THREADS=8          # set to physical cores per socket for your workload
export MKL_NUM_THREADS=$OMP_NUM_THREADS
export KMP_AFFINITY=granularity=fine,compact,1,0
  • Prefer one inter-op thread and many intra-op threads for single-model, single-process CPU inference:
# In ORT, set inter/intra as shown above; in PyTorch:
python - <<'PY'
import torch
torch.set_num_threads(8)
torch.set_num_interop_threads(1)
print("PyTorch threads set.")
PY
  • Lock CPU frequency to performance governor (measure power vs perf tradeoffs):
sudo modprobe msr
# cpupower path varies by distro; installed above
sudo cpupower frequency-set -g performance

Rule of thumb:

  • One inference worker per NUMA node.

  • Bind workers with numactl; avoid cross-node memory traffic.

  • Reserve some cores for I/O if you’re saturating all cores.

3) Batch smartly and fix the input pipeline

  • Micro-batching increases throughput; pick the smallest batch that stays within your latency SLO.

  • Keep the pipeline fed: decode/preprocess on separate threads/cores.

  • For CPU-only serving, batch sizes of 4–32 often give big wins without huge tail latency.

Simple micro-batcher pattern (toy example):

python - <<'PY'
import queue, threading, time
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("model.onnx")
input_name = sess.get_inputs()[0].name

q = queue.Queue()
def worker():
    while True:
        reqs = [q.get()]
        try:
            while len(reqs) < 8:  # batch size
                reqs.append(q.get_nowait())
        except queue.Empty:
            pass
        batch = np.concatenate(reqs, axis=0)
        _ = sess.run(None, {input_name: batch})
        for _ in reqs: q.task_done()

threading.Thread(target=worker, daemon=True).start()

# Simulate incoming single requests
for _ in range(100):
    x = np.random.randn(1,3,224,224).astype("float32")
    q.put(x)
q.join()
print("Done.")
PY

Also verify that preprocessing isn’t your bottleneck:

  • Use fast image libraries (e.g., torchvision transforms or OpenCV with multi-threaded decode).

  • Profile both compute and I/O sides.

4) Enable graph and kernel fusions

  • ONNX Runtime: ORT_ENABLE_ALL fuses common patterns, improving cache locality and reducing intermediate allocations.

  • OpenVINO: compiled graphs perform aggressive fusions per target device.

  • Prefer contiguous tensors and shapes that hit vectorization (e.g., multiples of 8/16 for AVX/AVX-512).

For ORT, you already set:

sess_opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

For PyTorch eager mode (CPU), you can still gain by:

  • Switching to TorchScript/torch.compile (when supported) and

  • Using channels-last on CNNs:

python - <<'PY'
import torch, torchvision
m = torchvision.models.resnet50(weights=None).eval()
x = torch.randn(1,3,224,224)
m = m.to(memory_format=torch.channels_last)
x = x.to(memory_format=torch.channels_last)
with torch.no_grad():
    for _ in range(10): m(x)  # warmup
    import time
    t0=time.time()
    for _ in range(100): m(x)
    print("Avg ms:", (time.time()-t0)/100*1000)
PY

5) Measure, don’t guess

Use perf and timing harnesses to identify CPU stalls, memory bottlenecks, and suboptimal threads.

  • Quick, repeatable timing loop:
python - <<'PY'
import time, numpy as np, onnxruntime as ort
sess = ort.InferenceSession("model.onnx")
inp = sess.get_inputs()[0]; name = inp.name
x = np.random.randn(8,*inp.shape[1:]).astype("float32")  # batch 8

# Warmup
for _ in range(20): sess.run(None, {name:x})

# Measure
runs=100
t0=time.time()
for _ in range(runs): sess.run(None, {name:x})
dt=time.time()-t0
print(f"Throughput: {runs/dt:.1f} it/s, Latency (ms): {dt/runs*1000:.2f}")
PY
  • System-level counters:
# Pin to cores 0-7 and collect performance counters
sudo perf stat -d -r 3 -- taskset -c 0-7 python infer.py
  • NUMA locality check:
numastat -p $(pgrep -f infer.py)

Iterate:

  • Adjust intra_op threads to match physical cores on the bound NUMA node (e.g., 8–32).

  • Try batch sizes 1, 2, 4, 8, 16… and pick the best for your SLO.

  • Compare FP32 vs INT8 (and FP16 if hardware supports).

Real-world mini lab: ResNet50 ONNX on CPU

Download a model, quantize, and compare:

wget -O resnet50.onnx https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v2-7.onnx

python - <<'PY'
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic("resnet50.onnx","resnet50.int8.onnx",weight_type=QuantType.QInt8)
print("Quantized -> resnet50.int8.onnx")
PY

Benchmark both with ORT:

python - <<'PY'
import time, numpy as np, onnxruntime as ort

def bench(path):
    so = ort.SessionOptions()
    so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    so.intra_op_num_threads = 8
    so.inter_op_num_threads = 1
    s = ort.InferenceSession(path, so)
    name = s.get_inputs()[0].name
    x = np.random.randn(8,3,224,224).astype("float32")  # batch 8
    for _ in range(20): s.run(None, {name:x})
    t0=time.time()
    runs=100
    for _ in range(runs): s.run(None, {name:x})
    dt=time.time()-t0
    return runs/dt, dt/runs*1000

for m in ["resnet50.onnx","resnet50.int8.onnx"]:
    thr, lat = bench(m)
    print(m, "Throughput(it/s):", round(thr,1), "Avg latency(ms):", round(lat,2))
PY

Typical outcome on a modern multi-core CPU:

  • INT8 improves throughput and reduces latency versus FP32, often 1.5–3x faster depending on CPU vector ISA and memory bandwidth.

Advanced system toggles (test, don’t assume)

  • Transparent Huge Pages (THP) vs explicit huge pages:
# Test on/off; effects vary by workload
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
# or
echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
  • Static huge pages (if your runtime benefits and you can allocate at boot):
echo 'vm.nr_hugepages=512' | sudo tee /etc/sysctl.d/99-hugepages.conf
sudo sysctl -p /etc/sysctl.d/99-hugepages.conf
  • Isolate service cores (systemd/cgroups) to reduce jitter in high-SLO services.

Summary and next steps (CTA)

The fastest path to real gains: 1) Move to an optimized runtime (ONNX Runtime or OpenVINO). 2) Drop precision (INT8) if accuracy allows. 3) Set threads and pin to a NUMA node. 4) Batch smartly and fix your input pipeline. 5) Measure, iterate, and automate.

Your next step:

  • Pick one production model.

  • Apply steps 1–5 this week.

  • Record baseline vs tuned metrics (p50/p95 latency, throughput, CPU utilization).

  • If you need multi-model serving and dynamic batching at scale, evaluate NVIDIA Triton Inference Server or OpenVINO Model Server next.

If you hit a ceiling, collect perf stats and share a minimal repro—it’s often one knob away from another 20–30%.

Related posts