Posted on
• Artificial Intelligence

Artificial Intelligence Linux Performance Optimisation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence on Linux: Practical Performance Optimisation with Bash

AI models are getting bigger, inference endpoints are busier, and training runs are more expensive. Yet many Linux systems run with conservative defaults that leave performance on the table: power-saving CPU governors, suboptimal thread placement, and memory/layout decisions that work fine for desktops but choke hot AI loops.

This guide shows how to align Linux with AI-heavy workloads—safely and repeatably—using Bash-friendly steps. You’ll learn what to tune, why it works, and how to verify it. Everything here is distribution-friendly with install commands for apt, dnf, and zypper.

Why this matters

  • AI workloads are often memory- and compute-bound with high locality. CPU frequency, NUMA layout, and thread affinity can make or break throughput.

  • Libraries (OpenBLAS, OpenMP) spin up threads automatically; without guidance they may oversubscribe cores or fight the scheduler.

  • Linux defaults target fairness and battery life. When you want low latency or maximum tokens/sec, you need to tell the kernel and runtime what “fast” means.

Install the tools you’ll need

The following utilities help you pin, profile, and tune your system. Pick what you need; you can install all of them in under a minute.

  • numactl (NUMA control and binding)

    • apt:
    sudo apt update
sudo apt install numactl
    • dnf:
    sudo dnf install numactl
    • zypper:
    sudo zypper install numactl

  • hwloc (topology awareness: lstopo, hwloc-ls)

    • apt:
    sudo apt update
sudo apt install hwloc || sudo apt install hwloc-nox
    • dnf:
    sudo dnf install hwloc
    • zypper:
    sudo zypper install hwloc

  • tuned (system-wide tuning profiles)

    • apt:
    sudo apt update
sudo apt install tuned
    • dnf:
    sudo dnf install tuned
    • zypper:
    sudo zypper install tuned

  • cpupower (CPU governor/frequency control)

    • apt (Ubuntu/Debian):
    sudo apt update
sudo apt install linux-tools-common
sudo apt install linux-tools-$(uname -r)
# cpupower will now be available as /usr/lib/linux-tools-*/cpupower or on PATH
    • dnf (Fedora/RHEL):
    sudo dnf install kernel-tools
    • zypper (openSUSE):
    sudo zypper install cpupower

  • perf (kernel profiling)

    • apt:
    sudo apt update
sudo apt install linux-tools-common
sudo apt install linux-tools-$(uname -r)
# provides the `perf` binary for your running kernel
    • dnf:
    sudo dnf install perf
    • zypper:
    sudo zypper install perf

  • powertop (optional: quick power/perf auto-tuning)

    • apt:
    sudo apt update
sudo apt install powertop
    • dnf:
    sudo dnf install powertop
    • zypper:
    sudo zypper install powertop

  • irqbalance (prevent hot CPUs from drowning in interrupts)

    • apt:
    sudo apt update
sudo apt install irqbalance
    • dnf:
    sudo dnf install irqbalance
    • zypper:
    sudo zypper install irqbalance

  • OpenBLAS (fast CPU math for many AI/ML libs; install runtime)

    • apt:
    sudo apt update
sudo apt install libopenblas0-pthread
    • dnf:
    sudo dnf install openblas
    • zypper:
    sudo zypper install openblas

Note: If you use conda/pip for AI frameworks, that’s fine—these system tools still help the OS and CPU behave better.

1) Profile first: find the bottleneck

Before changing anything, measure.

  • Quick wall-clock:

    /usr/bin/time -v python your_infer_or_train.py

  • Sample CPU hotspots:

    sudo perf stat -d -d -d -- python your_infer_or_train.py
sudo perf record -F 499 -- python your_infer_or_train.py
sudo perf report

  • Topology and NUMA view:

    lscpu --extended
numactl -H
lstopo --no-graphics

What to look for:

  • Low IPC and high cache-misses: memory locality or bandwidth issues.

  • High context switches or migrations: poor thread pinning/affinity.

  • CPU at low frequencies: governor/power profile issues.

  • Underutilized sockets: NUMA imbalance.

Real-world example:

  • Many CPU-only inference workloads are dominated by GEMM/BLAS calls and elementwise kernels. Proper thread count and affinity often produce a noticeable speedup without code changes.

2) Right-size and pin threads

AI libraries often default to “as many threads as possible,” which can cause oversubscription and cross-socket thrash.

  • Set thread counts for OpenMP/BLAS in your shell:

    export OMP_PROC_BIND=close
export OMP_PLACES=cores
export OMP_NUM_THREADS=$(nproc)               # try full machine, then tune
export OPENBLAS_NUM_THREADS=$OMP_NUM_THREADS
export MKL_NUM_THREADS=$OMP_NUM_THREADS       # respected by MKL if present
export NUMEXPR_NUM_THREADS=$OMP_NUM_THREADS

    Tip: Start with the number of physical cores (not threads). If Hyper-Threading/SMT is on, total logical CPUs = 2x physical. Try both and measure.

  • Pin a process to one socket to improve locality:

    # Bind to NUMA node 0 (CPUs and memory)
numactl --cpunodebind=0 --membind=0 python your_infer_or_train.py

    For minimal disruption, pin just the main hot process, especially for latency-sensitive inference.

  • Pin at the core level for repeated runs:

    taskset -c 0-15 python your_infer_or_train.py

  • Inside Python (PyTorch example):

    import torch, os
torch.set_num_threads(int(os.getenv("OMP_NUM_THREADS","8")))
torch.set_num_interop_threads(1)  # often helps inference latency

Measure again with perf/time and adjust. Many models like OMP_NUM_THREADS≈physical cores per socket; interop threads of 1–2; and a single-socket bind for consistent latency.

3) Tell the CPU to go fast: governors and tuned profiles

Desktop defaults favor energy savings. Throughput and low-latency work benefits from “performance” governors and tuned profiles.

  • Enable tuned and pick a high-performance profile:

    sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance
tuned-adm active

    Revert later with:

    sudo tuned-adm profile balanced

  • Set the CPU governor to performance:

    • With cpupower:
    # Fedora/openSUSE
sudo cpupower frequency-info
sudo cpupower frequency-set -g performance

    On Ubuntu, the binary may be under linux-tools path:

    sudo /usr/lib/linux-tools-*/cpupower frequency-set -g performance
    • Or via sysfs (portable):
    for cpu in /sys/devices/system/cpu/cpu[0-9]*; do
  echo performance | sudo tee $cpu/cpufreq/scaling_governor
done

  • Optional: powertop one-shot auto-tune (test impact yourself):

    sudo powertop --auto-tune

    Note: This tweaks device power settings. Keep or revert based on your measurements.

  • Make sure irqbalance is running to spread interrupts:

    sudo systemctl enable --now irqbalance

4) Optimise memory locality and pages

Memory bandwidth and locality dominate many AI loops. Keep memory close to the CPUs doing the work and reduce TLB pressure.

  • Inspect NUMA:

    numactl -H

  • Bind compute and memory together:

    numactl --cpunodebind=0 --membind=0 python your_infer_or_train.py

  • Transparent Huge Pages (THP) policy:

    • For many AI workloads, madvise is a safe middle ground:
    echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
    • To request some explicit HugePages (example: 2K pages of 2 MB = 4 GB):
    echo 2048 | sudo tee /proc/sys/vm/nr_hugepages

    Then run the workload and ensure /proc/meminfo shows HugePages in use. Test impact; not all models benefit.

  • Keep caches hot: Avoid cross-socket migration by using a single NUMA node for the main worker, especially for latency-sensitive inference services.

5) Use optimised math libraries and clean I/O paths

  • Ensure a fast BLAS backend is present (OpenBLAS runtime):

    • apt:
    sudo apt update
sudo apt install libopenblas0-pthread
    • dnf:
    sudo dnf install openblas
    • zypper:
    sudo zypper install openblas

  • Control BLAS threading so it cooperates with OpenMP:

    export OPENBLAS_NUM_THREADS=${OPENBLAS_NUM_THREADS:-$OMP_NUM_THREADS}

  • Reduce storage stalls: if datasets fit in RAM, stage hot data to tmpfs during experiments:

    sudo mkdir -p /mnt/ram
sudo mount -t tmpfs -o size=16G tmpfs /mnt/ram
rsync -a /path/to/dataset/ /mnt/ram/

    Warning: tmpfs consumes system memory; size it conservatively.

  • For streaming/preprocessing pipelines, profile your dataloader. Many “slow models” are actually I/O bound. Increase prefetch/worker counts in your framework and keep CPU cores reserved for preprocessing threads if needed.

Putting it together: a quick-start session script

Use this non-persistent script to prep a node for a benchmarking session. Adjust values to your machine.

#!/usr/bin/env bash
set -euo pipefail

# 1) High-performance tuned profile
sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance || true

# 2) Performance governor
if command -v cpupower >/dev/null 2>&1; then
  sudo cpupower frequency-set -g performance || true
else
  for cpu in /sys/devices/system/cpu/cpu[0-9]*; do
    echo performance | sudo tee $cpu/cpufreq/scaling_governor >/dev/null || true
  done
fi

# 3) NUMA info
echo "NUMA topology:"
numactl -H || true

# 4) Sensible threading defaults (edit to taste)
PHYS_CORES=$(lscpu | awk '/Core\(s\) per socket:/ {c=$4} /Socket\(s\):/ {s=$2} END{print c*s}')
export OMP_PROC_BIND=close
export OMP_PLACES=cores
export OMP_NUM_THREADS=${OMP_NUM_THREADS:-$PHYS_CORES}
export OPENBLAS_NUM_THREADS=${OPENBLAS_NUM_THREADS:-$OMP_NUM_THREADS}
export MKL_NUM_THREADS=${MKL_NUM_THREADS:-$OMP_NUM_THREADS}
export NUMEXPR_NUM_THREADS=${NUMEXPR_NUM_THREADS:-$OMP_NUM_THREADS}

echo "Using OMP_NUM_THREADS=$OMP_NUM_THREADS"

# 5) Optional: THP policy
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled >/dev/null || true

echo "Ready. Run your job with:"
echo "  numactl --cpunodebind=0 --membind=0 <your command>"

Run it:

bash ./ai-optimize.sh
numactl --cpunodebind=0 --membind=0 python your_infer_or_train.py

Verify improvements with:

/usr/bin/time -v numactl --cpunodebind=0 --membind=0 python your_infer_or_train.py
sudo perf stat -d -d -d -- numactl --cpunodebind=0 --membind=0 python your_infer_or_train.py

Common troubleshooting tips

  • If cpupower isn’t found on Ubuntu, ensure you installed both linux-tools-common and the linux-tools package matching your current kernel: sudo apt install linux-tools-$(uname -r).

  • If performance regresses, reduce thread counts (try physical cores only) or drop interop threads to 1 in your framework.

  • If latency is spiky, avoid cross-socket runs: bind to a single NUMA node and ensure dataloaders/preprocessors aren’t contending with the main threads.

  • If the system thrashes memory, undo tmpfs mounts and HugePages:

    sudo umount /mnt/ram
echo 0 | sudo tee /proc/sys/vm/nr_hugepages

Conclusion and next steps

Small, surgical changes to CPU governors, thread placement, and memory policy often unlock significant gains for AI workloads—without touching your model code. Your next step:

1) Install the tooling.
2) Profile your current baseline.
3) Apply one section at a time (threads, governor, NUMA), and re-measure.
4) Persist only what clearly helps on your hardware and model.

If you want a deeper dive tailored to your stack (PyTorch/TensorFlow/LLM serving), share your CPU model, NUMA layout (numactl -H), and a short perf stat of your workload—we’ll map out a targeted tuning plan.

Related posts