Posted on
• Artificial Intelligence

Artificial Intelligence Performance Bottlenecks

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Performance Bottlenecks: Find Them, Fix Them, Ship Faster

AI models don’t just “run slow.” They stall on I/O, starve GPUs, thrash memory, and waste CPU cycles. The difference between a sluggish training run and a well-oiled pipeline is rarely a single magic flag—it’s the discipline of measuring, identifying the real bottleneck, and applying targeted fixes.

This post will show you how to pinpoint and remove the most common AI performance bottlenecks from a Linux/Bash perspective. You’ll get actionable commands, practical tuning tips, and distro-agnostic install instructions (apt, dnf, zypper) for the tools you’ll need.

Why this matters

  • GPU hours are expensive—wasting them hurts both timelines and budgets.

  • Modern stacks (CUDA/ROCm, frameworks, drivers, containers) make it easy to misconfigure.

  • The slowest component (disk, network, CPU, GPU, memory, interconnect) sets your ceiling. Fixing the wrong layer won’t help.

The value: a simple, repeatable workflow to observe, diagnose, and remove true bottlenecks—without guesswork.

Install the essentials

You don’t need everything, but these cover 90% of day‑to‑day profiling on Linux.

  • sysstat (sar/iostat), pidstat, vmstat

  • nvtop (GPU live view for NVIDIA)

  • iotop (I/O by process)

  • perf (CPU events)

  • numactl (NUMA awareness)

  • fio (disk benchmarks)

  • iperf3 (network benchmarks)

  • htop (quick CPU/RAM view)

  • pciutils (lspci for topology)

  • nvme-cli (NVMe health and utilization)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y sysstat nvtop iotop linux-tools-common linux-tools-$(uname -r) numactl fio iperf3 htop pciutils nvme-cli

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y sysstat nvtop iotop perf numactl fio iperf3 htop pciutils nvme-cli

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y sysstat nvtop iotop perf numactl fio iperf3 htop pciutils nvme-cli

Notes:

  • nvidia-smi ships with the NVIDIA proprietary driver. Install your distro’s recommended NVIDIA driver if you use NVIDIA GPUs.

  • For AMD GPUs, use rocm-smi (packaged with ROCm); commands below focus on NVIDIA because of ubiquitous usage.

1) Measure before you tweak: locate the real bottleneck

Goal: decide if you are bound by GPU, CPU, memory, disk, or network—then dig deeper.

Quick baseline (run during your training/inference workload in another terminal):

# GPU utilization and memory
watch -n1 nvidia-smi

# GPU live TUI (per-process)
nvtop

# CPU, memory, context switches
sar -u 1
sar -r 1

# Per-process CPU/mem/I/O (replace PID)
pidstat -udr -p <PID> 1

# Disk device saturation and latency
iostat -xz 1

# Top talkers on disk I/O
sudo iotop -oPa

# CPU performance counters (replace PID; stop with Ctrl-C)
sudo perf stat -p <PID>

What to look for:

  • GPU util < 50% most of the time? You’re likely I/O or CPU bound, or suffering data-loader stalls.

  • iostat shows high await and util near 100%? Disk bound.

  • pidstat shows high CPU and voluntary context switches? Threading or Python GIL issues in data preprocessing.

  • perf shows high LLC misses or cycles stalled? Memory bandwidth/locality problem.

  • nvidia-smi shows frequent P2 state and low power draw? PCIe or data starvation; not compute-bound.

2) Eliminate data pipeline and storage bottlenecks

Your GPUs can’t compute if the input isn’t ready. Optimize how data moves from storage to RAM to GPU.

Actions:

  • Stage hot datasets to local NVMe instead of remote or spinning disks.

  • Use sequential, larger I/O (batching reads); compress or pack small files (e.g., tar/TFRecord/LMDB/webdataset).

  • Prefetch and overlap data loading with compute (framework settings).

  • Profile disk and filesystem.

Install fio if needed:

  • apt: sudo apt install -y fio

  • dnf: sudo dnf install -y fio

  • zypper: sudo zypper install -y fio

Measure current storage:

# Identify your dataset mount
df -h

# Disk saturation and latency (look for high await/avgqu-sz)
iostat -xz 5

# NVMe specifics
sudo nvme top  # or: sudo nvme smart-log /dev/nvme0

# Quick sequential read test (non-destructive, temp file)
fio --name=readseq --rw=read --bs=1M --size=4G --iodepth=32 --direct=1 --filename=/path/to/fastdisk/fio.tmp
rm -f /path/to/fastdisk/fio.tmp

Stage data locally:

rsync -a --info=progress2 /mnt/remote/dataset/ /local_nvme/dataset/

Framework tips (examples):

  • PyTorch DataLoader: increase num_workers, enable pin_memory, use persistent_workers.

  • Use larger batch size (if GPU memory allows) to amortize loader overhead.

  • Convert millions of tiny files into shard formats to avoid metadata storms.

3) Fix CPU, threading, and NUMA locality

Dual-socket systems and high core counts are great—until your process bounces across sockets and memory nodes.

Install numactl if needed:

  • apt: sudo apt install -y numactl

  • dnf: sudo dnf install -y numactl

  • zypper: sudo zypper install -y numactl

Inspect topology:

lscpu | egrep 'Socket|NUMA'
numactl --hardware

Bind your job to one NUMA node (example: node 0, CPUs 0-31):

numactl --cpunodebind=0 --membind=0 \
  taskset -c 0-31 \
  ./run_training.sh

Tune threading for CPU-bound preprocessing:

# Set OpenMP/MKL threads to match physical cores you assigned
export OMP_NUM_THREADS=16
export MKL_NUM_THREADS=16
# Avoid oversubscription in Python data pipelines
export PYTHONWARNINGS="ignore"

Signs you fixed it:

  • pidstat shows fewer context switches, higher CPU efficiency per core.

  • perf reports fewer stalled cycles and cache misses.

  • GPU utilization rises because batches are prepared on-time.

4) Keep the GPU fed and efficient

When the GPU is the star, make sure it stays busy, uses the right math, and doesn’t throttle.

Check status:

nvidia-smi
nvidia-smi topo -m     # PCIe/NVLink topology
watch -n1 nvidia-smi dmon

Practical steps:

  • Mixed precision (FP16/BF16) for Tensor Core acceleration (framework-level flag).

  • Increase batch size to raise arithmetic intensity (watch out for OOM).

  • Enable GPU persistence to reduce initialization overhead: sudo nvidia-smi -pm ENABLED

  • Avoid power throttling if safe for your datacenter: nvidia-smi -q -d POWER then, if allowed: sudo nvidia-smi -pl <PowerLimitWithinSafeRange>

  • Ensure kernel/driver/CUDA/ROCm versions are compatible. Mismatches can silently degrade performance.

Quick version checks:

nvidia-smi            # Driver and CUDA runtime
python -c "import torch,platform; print(torch.__version__, torch.cuda.is_available())"  # Example for PyTorch

If GPU util remains low despite efforts:

  • You’re likely I/O or CPU bound, or kernels are launching inefficiently (tiny ops, excessive synchronization).

  • Fuse small ops, use better batching, or leverage vendor libraries (cuBLAS, cuDNN, TensorRT) for inference.

5) Don’t forget the network (distributed training/inference)

Multi-GPU, multi-node jobs are only as fast as the slowest link.

Install iperf3 if needed:

  • apt: sudo apt install -y iperf3

  • dnf: sudo dnf install -y iperf3

  • zypper: sudo zypper install -y iperf3

Test throughput and latency:

# On node A
iperf3 -s
# On node B (replace host)
iperf3 -c nodeA.example.com -t 30 -P 4

Inspect topology and interconnect:

nvidia-smi topo -m    # Look for NVLink, PHB distances

NCCL tuning (examples; test and measure):

export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0,mlx5_1     # If using InfiniBand
export NCCL_SOCKET_IFNAME=eth0       # If using Ethernet
export NCCL_NSOCKS_PERTHREAD=4
export NCCL_MIN_NCHANNELS=4

If bandwidth is low:

  • Verify you’re not falling back to TCP when IB/RDMA is intended.

  • Pin processes to GPUs close to their NICs (NUMA-aware process placement).

  • Reduce gradient size (mixed precision, gradient compression) or increase batch to reduce sync frequency.

Real‑world quick wins you can apply today

  • Pack small files: converting millions of JPEGs to webdataset/tar shards lifted GPU util from 30% to >90% on a 4x GPU rig.

  • NUMA pinning: binding dataloaders to the same socket as the GPU doubled input throughput on a dual-socket box.

  • Mixed precision: FP16/BF16 routinely delivers 1.5–3x speedups with negligible accuracy loss for many vision and LLM workloads.

  • Local staging: rsyncing from NFS to NVMe reduced step time variance by 40% due to consistent read latencies.

A 15‑minute baseline profile you can keep

Drop this into a tmux/screen session when your job starts and save the outputs.

mkdir -p baseline && cd baseline

# GPU
(nvidia-smi -q -d UTILIZATION,MEMORY,POWER,PCI &> gpu_snapshot.txt)
(nvidia-smi topo -m > topo.txt)
(timeout 300 nvidia-smi dmon -s pucm) | tee gpu_dmon.txt

# CPU/mem
(timeout 300 sar -u 1) | tee cpu_sar.txt
(timeout 300 sar -r 1) | tee mem_sar.txt
(timeout 300 iostat -xz 1) | tee disk_iostat.txt

# Replace with your PID
PID=$(pgrep -n -f your_training_entrypoint.py)
if [ -n "$PID" ]; then
  (timeout 60 sudo perf stat -p $PID) 2> perf_stat.txt
  (timeout 60 pidstat -udr -p $PID 1) | tee pidstat.txt
fi

Review which subsystem is redlining; optimize that first.

Conclusion and next steps

Performance work is a loop: measure → hypothesize → change → measure again. Start by confirming where you’re bound (GPU, CPU, memory, disk, network), then apply targeted fixes—data packing, NUMA pinning, mixed precision, proper batching, and network/NCCL tuning. Small, disciplined changes compound into big wins.

Call to Action:

  • Install the tools above with your package manager.

  • Run the 15‑minute baseline profile on your next job.

  • Fix one confirmed bottleneck at a time and re-measure.

  • Document your working settings per machine type so your team stops rediscovering them.

If you want a follow-up, ask for a distro-specific checklist for NVIDIA vs AMD, single-node vs multi-node, or a starter Bash script that auto-tunes common knobs based on live telemetry.

Related posts