Artificial Intelligence CPU Optimisation: 5 Bash-Friendly Wins That Deliver

You don’t always need a GPU to ship fast, reliable AI. In fact, most production inference runs on CPUs—at the edge, in containers, and in CI/CD pipelines. The problem is that default settings leave a lot of performance on the table: threads aren’t pinned, memory crosses NUMA nodes, governors throttle frequency, and math libraries run suboptimally. The value? With a handful of Bash commands and sane defaults, you can often get 1.5–4x better CPU inference throughput or lower tail latency—without changing your model.

This guide explains why CPU optimisation is worth your time, then gives you 5 actionable, Linux-ready steps you can paste into your terminal. All tools include apt, dnf, and zypper install instructions.

Why optimising AI on CPUs is valid (and often essential)

CPUs are everywhere. Think production microservices, dev laptops, CI runners, and edge nodes where GPUs aren’t available or economical.
Modern CPUs have serious AI horsepower: AVX2/AVX-512, BF16, AMX (newer Intel), and wide memory bandwidth—if you feed them properly.
Many workloads are memory-bound. Fixing NUMA locality, thread pinning, and huge pages can unlock big wins without code changes.
CPU runtimes (e.g., oneDNN-backed PyTorch/ONNX Runtime) plus quantisation (INT8, BF16) deliver large gains while preserving acceptable accuracy.

Quick preflight: capture a baseline

You can’t optimise what you don’t measure.

Install sysstat for lightweight CPU metrics:

apt:

sudo apt update && sudo apt install -y sysstat

dnf:
```
sudo dnf install -y sysstat
```
zypper:
```
sudo zypper install -y sysstat
```

Baseline loop (replace python run_infer.py with your entry point):

/usr/bin/time -f '%E elapsed, %M KB maxRSS' \
  bash -c 'python run_infer.py >/dev/null'
mpstat -P ALL 1 5

1) Pin threads and set sane thread counts

Default schedulers can bounce threads across cores, increasing cache misses and jitter. Pinning ensures locality and predictable performance.

Tools you’ll need:

util-linux (for taskset; usually preinstalled)

apt:

sudo apt update && sudo apt install -y util-linux

dnf:

sudo dnf install -y util-linux

zypper:

sudo zypper install -y util-linux

numactl (we’ll use it later too)

apt:

sudo apt update && sudo apt install -y numactl

dnf:

sudo dnf install -y numactl

zypper:

sudo zypper install -y numactl

Discover your topology:

lscpu | egrep 'Model name|Socket|Core|Thread|NUMA'
numactl -H

Pin your workload to a contiguous core range and set OpenMP threads:

# Example: Use cores 0-15 (adjust to your CPU)
export OMP_NUM_THREADS=16
export OMP_PROC_BIND=close             # bind near threads
export GOMP_CPU_AFFINITY="0-15"        # GNU OpenMP
export KMP_AFFINITY=granularity=fine,compact,1,0  # Intel OpenMP if present

taskset -c 0-15 python run_infer.py

Tip:

Start with OMP_NUM_THREADS set to the number of physical cores you’re using (not hyperthreads).
For mixed Python + NumPy/BLAS, set both OMP_NUM_THREADS and the library’s own thread var (see Step 3).

2) Get NUMA right: keep compute and memory together

On dual-socket or multi-NUMA systems, straddling nodes adds latency and reduces bandwidth. Bind CPUs and memory to the same node.

Inspect nodes:

numactl -H

Bind to a single node:

# Run entirely on NUMA node 0
numactl --cpunodebind=0 --membind=0 python run_infer.py

If your dataset doesn’t fit one node cleanly, try interleaving memory:

numactl --interleave=all python run_infer.py

Check allocation after launch (optional):

pid=$(pgrep -n python)
numastat -p $pid

Rule of thumb:

Throughput jobs: one process per NUMA node, pinned to that node’s cores and memory.
Low-latency services: keep per-request threads local to a node; avoid cross-node contention.

3) Use the right math backend and control its threads

Even when you don’t explicitly “use OpenMP,” many frameworks do under the hood. Misconfigured threads can lead to oversubscription and slowdowns.

Install OpenBLAS (common, fast default for linear algebra):

apt:

sudo apt update && sudo apt install -y libopenblas-dev

dnf:

sudo dnf install -y openblas openblas-devel

zypper:

sudo zypper install -y openblas openblas-devel

Recommended environment variables:

# Start conservative; tune later
export OMP_NUM_THREADS=16
export OPENBLAS_NUM_THREADS=16
export MKL_NUM_THREADS=16         # if MKL is present (e.g., via some Python wheels)
export NUMEXPR_MAX_THREADS=16     # if you use numexpr

Avoid thread oversubscription:

If your app has its own thread pool (e.g., web worker threads), reduce OMP_NUM_THREADS accordingly.
Consider inter_op_num_threads=1 and tune intra_op threads (see ONNX Runtime example below).

4) System tuning: frequency governors and IRQ balancing

Throttling and interrupt storms can cripple performance or cause jitter. Use distro-native tools to select a performance profile and balance interrupts.

Install and enable tuned:

apt:

sudo apt update && sudo apt install -y tuned
sudo systemctl enable --now tuned

dnf:

sudo dnf install -y tuned
sudo systemctl enable --now tuned

zypper:

sudo zypper install -y tuned
sudo systemctl enable --now tuned

Pick a profile:

# Throughput-oriented (good for batch inference)
sudo tuned-adm profile throughput-performance

# Or pick low-latency when you care about tail percentiles
sudo tuned-adm profile latency-performance

Install and enable irqbalance:

apt:

sudo apt update && sudo apt install -y irqbalance
sudo systemctl enable --now irqbalance

dnf:

sudo dnf install -y irqbalance
sudo systemctl enable --now irqbalance

zypper:

sudo zypper install -y irqbalance
sudo systemctl enable --now irqbalance

Optional checks:

# See current governor on core 0
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "No cpufreq (server CPU or governor not exposed)"

5) Quantise for the CPU: INT8/BF16 with ONNX Runtime

Quantisation frequently delivers the biggest single win for CPU inference. Accurate INT8 or BF16 models can drastically reduce compute and memory bandwidth.

Install CPU-friendly runtime via pip (example with ONNX Runtime):

python3 -m pip install --upgrade onnx onnxruntime

Dynamic INT8 weight-only quantisation (quick and safe for many models):

python3 - <<'PY'
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(
    model_input="model.onnx",
    model_output="model-int8.onnx",
    optimize_model=True,
    weight_type=QuantType.QInt8
)
print("Wrote model-int8.onnx")
PY

Run the quantised model with tuned thread options:

python3 - <<'PY'
import onnxruntime as ort, numpy as np, os

# Tune threads here (align with Steps 1–3)
intra = int(os.environ.get("OMP_NUM_THREADS", "16"))
so = ort.SessionOptions()
so.intra_op_num_threads = intra
so.inter_op_num_threads = 1
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

sess = ort.InferenceSession("model-int8.onnx", sess_options=so, providers=["CPUExecutionProvider"])

# Dummy input; replace shapes/dtypes for your model
x = np.random.rand(1,3,224,224).astype(np.float32)
y = sess.run(None, {"input": x})
print([a.shape for a in y])
PY

Notes:

On some CPUs, BF16 (bfloat16) is a free win if your backend supports it; frameworks using oneDNN may auto-enable it when hardware allows.
For best results, combine quantisation with proper thread pinning and NUMA binding.

Measure, iterate, and lock in gains

A small harness helps you find the sweet spot quickly:

for t in 1 2 4 8 16 32; do
  export OMP_NUM_THREADS=$t OPENBLAS_NUM_THREADS=$t
  echo "=== Threads=$t ==="
  /usr/bin/time -f '%E elapsed, %M KB maxRSS' \
    taskset -c 0-$((t-1)) \
    python run_infer.py >/dev/null
done

Watch CPU utilisation and steal time:

mpstat -P ALL 1
pidstat -t -p $(pgrep -n python) 1

If you use NUMA:

numastat -p $(pgrep -n python)

What to expect (typical, workload-dependent):

Thread pinning and NUMA locality: often 1.2–2x throughput improvement vs. defaults, with reduced jitter.
Quantisation (INT8): often 1.5–4x speedups on CPU-compatible models.
Governor/IRQ tuning: smoother tails and a few to double-digit percentage improvements, especially under load.

Real-world pattern you can reuse

One process per NUMA node.
Pin to that node’s cores, bind memory to the same node.
Set OMP_NUM_THREADS to the number of physical cores used by the process.
Use quantised models when acceptable; set intra_op_num_threads high, inter_op low.
Use tuned throughput-performance for batch throughput; latency-performance for P99-sensitive services.
Measure, then bake the best settings into your service unit or container entrypoint.

Example service wrapper (adjust core list and node):

#!/usr/bin/env bash
set -euo pipefail

CORES="0-15"
NODE=0
THREADS=16

export OMP_NUM_THREADS=$THREADS
export OPENBLAS_NUM_THREADS=$THREADS
export OMP_PROC_BIND=close
export GOMP_CPU_AFFINITY="${CORES}"

exec numactl --cpunodebind=${NODE} --membind=${NODE} \
  taskset -c ${CORES} \
  python run_infer.py

Conclusion and next steps (CTA)

Your CPU is better at AI than you think—if you meet it halfway. Start with: 1) Pin threads and set OMP threads.
2) Bind compute and memory to the same NUMA node.
3) Use an optimised math backend and avoid oversubscription.
4) Pick a tuned system profile and enable irqbalance.
5) Quantise where it makes sense.

Next step: take 30 minutes to apply the five steps to one model in your stack. Save the best settings in a small wrapper script or systemd unit so they’re repeatable in CI and production. Then iterate with your real traffic patterns to lock in performance and stability.

Artificial Intelligence CPU Optimisation: 5 Bash-Friendly Wins That Deliver

Why optimising AI on CPUs is valid (and often essential)

Quick preflight: capture a baseline

1) Pin threads and set sane thread counts

2) Get NUMA right: keep compute and memory together

3) Use the right math backend and control its threads

4) System tuning: frequency governors and IRQ balancing

5) Quantise for the CPU: INT8/BF16 with ONNX Runtime

Measure, iterate, and lock in gains

Real-world pattern you can reuse

Conclusion and next steps (CTA)

Related posts