Artificial Intelligence Linux Performance Tuning: A Bash-First Guide

If your AI training job takes hours, shaving 10–30% off the runtime isn’t just nice—it’s the difference between shipping this week or next. The surprising truth: plenty of AI workloads leave performance on the table because the Linux defaults aren’t tuned for high-throughput, long-running, memory- and I/O-heavy jobs. With a few targeted, reversible system tweaks and observability tools, you can unlock serious speedups—often without touching your model code.

This post shows you exactly how to do that from the command line, with practical steps you can apply today. You’ll get the “why” and the “how,” plus real commands, package installs for apt/dnf/zypper, and a final script you can adapt.

Why this matters

AI workloads are unique:

• They’re bursty (data ingest) and sustained (matrix math) at the same time.

• NUMA, CPU governor, memory policy, and I/O scheduler choices can make or break throughput.

• GPU utilization hinges on the host: data pipelines, CPU affinity, memory allocation, and storage latency all gate your accelerator.

You don’t need to be a kernel hacker to get wins—you need a baseline, a few knobs, and discipline to test changes one-by-one.

---

1) Establish a clean baseline and observability

Before changing settings, measure. You want to know if you’re CPU-, memory-, I/O-, or GPU-bound, and where time goes.

Install common tools:

• htop (per-core usage), sysstat (iostat/mpstat), perf, numactl, hwloc (lstopo), tuned, irqbalance, nvtop (GPU), sysbench (quick CPU/memory tests)

apt:

sudo apt update
sudo apt install -y htop sysstat numactl hwloc tuned irqbalance nvtop sysbench
# perf on Debian/Ubuntu:
sudo apt install -y linux-tools-common
sudo apt install -y linux-tools-$(uname -r) || sudo apt install -y linux-tools-generic

dnf:

sudo dnf install -y htop sysstat numactl hwloc tuned irqbalance nvtop sysbench perf

zypper:

sudo zypper install -y htop sysstat numactl hwloc tuned irqbalance nvtop sysbench perf

Useful baseline commands:

# CPU and runqueue pressure
mpstat -P ALL 1
vmstat 1

# I/O service time and queue depth
iostat -xt 1

# NUMA memory locality
numastat -m
numactl --hardware

# Top-down overview
htop

# NVIDIA GPU utilization (requires NVIDIA driver and nvidia-smi)
nvidia-smi dmon -s pucm -d 1  # power/util/clock/mem; Ctrl+C to stop

Note: nvidia-smi comes with the NVIDIA driver. Install driver via your distro’s tooling:

• apt (Ubuntu):

  sudo ubuntu-drivers autoinstall
  # or specific series:
  sudo apt install -y nvidia-driver-535  # replace 535 with a supported version on your system
  

• dnf (RHEL/Fedora with NVIDIA repo enabled):

  sudo dnf install -y akmod-nvidia
  

• zypper (openSUSE with NVIDIA repo enabled):

  sudo zypper install -y nvidia-driver-G06  # or appropriate series
  

Real-world example:

• If iostat -xt 1 shows high r_await/w_await and aqu-sz on your dataset disk, your GPU may idle. Fixing I/O scheduler/readahead (Section 4) can lift GPU util by 10–40%.

---

2) CPU frequency governor and NUMA placement

AI data pipelines, tokenization, and augmentation are CPU-heavy. Ensure cores run fast and memory locality is respected.

Install cpupower:

• apt:

  # One of these provides cpupower depending on distro version:
  sudo apt install -y linux-cpupower || sudo apt install -y linux-tools-common linux-tools-$(uname -r)
  

• dnf:

  sudo dnf install -y kernel-tools
  

• zypper:

  sudo zypper install -y cpupower
  

Option A: Set performance governor (temporary, until reboot):

sudo cpupower frequency-set -g performance

Option B: Use tuned (persistable profile):

sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance
# Later: sudo tuned-adm profile balanced

Pin your process to a NUMA node and bind memory close to it:

# Pin to NUMA node 0, allocate memory from node 0
numactl --cpunodebind=0 --membind=0 python train.py

For threaded math libraries (MKL/OpenMP), set sane thread counts and affinity:

# Use physical cores, not hyperthreads; adjust as needed
export OMP_NUM_THREADS=16
export MKL_NUM_THREADS=16
export KMP_AFFINITY=granularity=fine,compact,1,0
export OPENBLAS_NUM_THREADS=16

numactl --cpunodebind=0 --membind=0 python train.py

Quick demo (see difference with/without governor pinning):

# CPU micro-benchmark
sysbench cpu --threads=$(nproc) --cpu-max-prime=40000 run

---

3) Memory: Huge pages, THP policy, and faster allocators

Large tensor allocations benefit from fewer page faults and better TLB behavior.

Transparent Huge Pages (THP) to madvise:

# Check current
cat /sys/kernel/mm/transparent_hugepage/enabled
# Set to madvise (recommended for many AI runtimes)
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Pre-allocate some 2M hugepages (adjust count):

echo 4096 | sudo tee /proc/sys/vm/nr_hugepages
# Persist across reboots:
echo "vm.nr_hugepages=4096" | sudo tee /etc/sysctl.d/99-hugepages.conf
sudo sysctl --system

Reduce swap aggressiveness:

echo "vm.swappiness=10" | sudo tee /etc/sysctl.d/99-ai-tuning.conf
sudo sysctl --system

Use a faster memory allocator (often improves Python + PyTorch/TensorFlow dataloaders):

Install jemalloc:

• apt:

  sudo apt install -y libjemalloc2
  

• dnf:

  sudo dnf install -y jemalloc
  

• zypper:

  sudo zypper install -y jemalloc
  

Install tcmalloc (gperftools):

• apt:

  sudo apt install -y google-perftools libtcmalloc-minimal4
  

• dnf:

  sudo dnf install -y gperftools-libs
  

• zypper:

  sudo zypper install -y gperftools
  

Try jemalloc first:

# Find the library path (adjust based on your distro):
ldconfig -p | grep jemalloc

# Example use with Python
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 \
MALLOC_CONF=background_thread:true,dirty_decay_ms:500,muzzy_decay_ms:500 \
python train.py

If jemalloc regresses, try tcmalloc:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 python train.py

Measure before/after with wall-clock and RSS to confirm gains.

---

4) Storage pipeline and I/O scheduler tuning

Your GPUs only move as fast as data arrives.

Pick a good I/O scheduler (NVMe: none or mq-deadline; SATA SSD: mq-deadline):

# See device name (e.g., nvme0n1, sda)
lsblk

# Check current scheduler
cat /sys/block/nvme0n1/queue/scheduler

# Temporarily set scheduler (until reboot)
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
# or
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

Increase readahead for large sequential loads:

# 8 MB readahead (for big batches, TFRecords, LM datasets, etc.)
sudo blockdev --setra 16384 /dev/nvme0n1

Persist these via udev rule:

# /etc/udev/rules.d/60-io-tuning.rules
ACTION=="add|change", KERNEL=="nvme0n1", ATTR{queue/scheduler}="none", RUN+="/sbin/blockdev --setra 16384 /dev/nvme0n1"
sudo udevadm control --reload && sudo udevadm trigger

Use tmpfs for small, hot datasets to avoid disk latency:

# Mount 32G RAM disk (adjust size)
sudo mkdir -p /mnt/ramdisk
sudo mount -t tmpfs -o size=32G,mode=1777 tmpfs /mnt/ramdisk
# Copy hot shards here during training

Real-world example:

• Switching an NVMe’s scheduler to none and setting 8 MB readahead reduced per-epoch data wait by ~20% in an image classification pipeline with heavy sequential reads.

---

5) GPU utilization quick wins (NVIDIA-focused)

If you use NVIDIA GPUs, a few host-level tweaks improve stability and utilization.

Enable persistence mode (keeps driver context alive):

sudo nvidia-smi -pm 1

For data center GPUs, optionally set application clocks (consult your SKU’s safe ranges first):

# Example for A100 (adjust for your GPU):
sudo nvidia-smi -ac 1215,1410

Pin CPU workers feeding the GPU:

# Reserve a NUMA node for dataloaders; keep model compute on another if possible
numactl --cpunodebind=0 --membind=0 python -m my_dataloader &
numactl --cpunodebind=1 --membind=1 python train_gpu.py

Watch for starvation:

nvidia-smi dmon -s pucm -d 1
iostat -xt 1
mpstat -P ALL 1

If GPU util is low while CPU I/O wait is high, revisit Section 4. If CPU is pegged at 100% on a few cores, revisit Section 2 and thread counts.

Note: AMD ROCm users should similarly watch rocm-smi and ensure driver/runtime are correctly installed from AMD’s repositories; the same host-level CPU/memory/I/O guidance applies.

---

Put it together: a minimal, reversible tuning script

Start conservative; uncomment gradually and measure.

#!/usr/bin/env bash
set -euo pipefail

# 0) Safety: require root for system changes
if [[ $EUID -ne 0 ]]; then
  echo "Please run as root (sudo)"; exit 1
fi

# 1) Baseline snapshot
echo "=== Baseline ==="
lscpu | sed -n '1,12p' || true
numactl --hardware || true
lsblk -o NAME,ROTA,SIZE,MODEL | grep -E 'sd|nvme' || true
cat /sys/kernel/mm/transparent_hugepage/enabled || true

# 2) CPU governor to performance
if command -v cpupower >/dev/null 2>&1; then
  cpupower frequency-set -g performance || true
fi

# 3) tuned throughput profile (optional)
if systemctl list-unit-files | grep -q tuned.service; then
  systemctl enable --now tuned || true
  tuned-adm profile throughput-performance || true
fi

# 4) THP to madvise
if [[ -w /sys/kernel/mm/transparent_hugepage/enabled ]]; then
  echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
fi

# 5) Low swappiness
sysctl -w vm.swappiness=10

# 6) Example: NVMe scheduler + readahead (adjust device)
DEV=/dev/nvme0n1
if [[ -e $DEV ]]; then
  echo none > /sys/block/$(basename $DEV)/queue/scheduler || true
  blockdev --setra 16384 $DEV || true
fi

echo "Tuning applied. Reboot to revert or restore previous configs."

Run:

sudo bash ai-tune.sh

---

Conclusion and next steps (CTA)

Most AI performance “mysteries” are explainable: CPU frequency scaling, NUMA misses, allocator overhead, and I/O queues are visible—and fixable—from Bash. Start by measuring, then apply the targeted tweaks above. Validate each change with your real workload; keep what helps, drop what doesn’t.

Your next steps: 1) Install the observability stack and take a 10-minute baseline. 2) Apply Sections 2–4 incrementally, re-benchmarking after each. 3) Bake the wins into a small startup script or systemd profile for your training nodes. 4) Document your final settings in your repo so teammates get the same speedups.

If you want a follow-up, ask for a distro-specific, persistent configuration pack (sysctl.d, udev rules, tuned profile, and systemd unit examples) tailored to your hardware.