Posted on
• Artificial Intelligence

Power Optimisation for Artificial Intelligence Servers

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Power Optimisation for Artificial Intelligence Servers (with Bash you can trust)

AI servers eat watts for breakfast. Between multi-socket CPUs and data-center GPUs, a single node can draw more than a household. That’s money, heat, and carbon. The good news: with a few practical, scriptable Linux/Bash tweaks, you can lower power draw and improve energy per training/inference job—often with negligible impact on throughput.

This guide explains why power optimisation matters, then walks you through actionable steps you can automate on your fleet. All examples are Linux-first, Bash-friendly, and include apt/dnf/zypper install instructions where needed.

Why this matters (and why now)

  • Cost and capacity: Power-hungry nodes limit rack density and balloon electricity and cooling bills.

  • Performance stability: Throttling from thermal/power limits tanks throughput at the worst moments.

  • Sustainability and compliance: Energy per model/epoch is increasingly a KPI; regulators and customers are watching.

Before you start: Install the essentials

We’ll use common, vendor-neutral tools and optional GPU utilities. Run the appropriate commands for your distro.

  • powertop (measure and suggest tunings)

  • tuned (system-wide power/perf profiles)

  • cpupower or cpufrequtils (CPU frequency/governor controls)

  • numactl (NUMA awareness and pinning)

  • powercap-utils (optional, RAPL power capping on Intel)

  • NVIDIA users: nvidia-smi (comes with the proprietary driver)

Install system tools:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y powertop tuned numactl powercap-utils ethtool
# cpupower lives in linux-tools; try generic first, then kernel-specific:
sudo apt install -y linux-tools-common linux-tools-generic || sudo apt install -y linux-tools-$(uname -r) cpufrequtils

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y powertop tuned numactl powercap-utils ethtool kernel-tools
# cpupower is part of kernel-tools

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y powertop tuned numactl powercap-utils ethtool cpupower

Enable tuned (all distros):

sudo systemctl enable --now tuned

NVIDIA driver (to get nvidia-smi):

  • Ubuntu/Debian (apt):
sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi
  • Fedora (dnf) via RPM Fusion:
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                    https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia
sudo reboot
nvidia-smi
  • openSUSE (zypper) using NVIDIA repo (example for Tumbleweed; replace if on Leap):
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed NVIDIA
sudo zypper install -y nvidia-computeG06
sudo reboot
nvidia-smi

Note: Package names and repos vary by GPU generation and distro release. Use your distro’s official NVIDIA guide if the above differs.

Step 1 — Measure a baseline like an SRE

You can’t optimise what you don’t measure. Capture idle and load numbers for power, clocks, and util.

  • CPU/system view:
sudo powertop --time=15s
sudo powertop --html=/tmp/power.html
xdg-open /tmp/power.html 2>/dev/null || echo "Report at /tmp/power.html"
  • NVIDIA GPUs:
nvidia-smi
nvidia-smi dmon -s pucvmet -o DT
# p: power, u: SM util, c: clocks, v: ECC, m: mem util, e: encoder, t: temperature

Tips:

  • Record idle vs training/inference loads.

  • If you have a smart PDU/UPS, log wall power too; it captures PSU and fan overheads.

Step 2 — Tune CPU governors, EPP, and turbo

Right-sizing CPU clocks can cut tens of watts with minimal hit to GPU-bound training or inference.

  • Inspect and set the CPU governor:
sudo cpupower frequency-info | sed -n '1,120p'
sudo cpupower frequency-set -g powersave     # often best for GPU-bound jobs
# For CPU-bound preprocessing, try performance or schedutil and compare energy/throughput.
  • Prefer energy-aware performance using EPP (Intel/AMD pstate):
# Show current EPP value (one CPU as example)
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference 2>/dev/null || echo "EPP not available"

# Set all CPUs to balance_performance (good default), or powersave on inference nodes:
for p in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do
  [ -f "$p" ] && echo balance_performance | sudo tee "$p" >/dev/null
done
  • Consider disabling turbo for predictable thermals (verify for your CPU):
# Intel (older intel_pstate)
[ -f /sys/devices/system/cpu/intel_pstate/no_turbo ] && echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

# Generic (some AMD/Intel)
[ -f /sys/devices/system/cpu/cpufreq/boost ] && echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost

When to use which:

  • Training heavy on GPU: powersave governor + balance_performance EPP is often ideal.

  • CPU-heavy data loading: consider performance governor during that phase only (wrap your launcher).

Example wrapper:

#!/usr/bin/env bash
set -euo pipefail
sudo cpupower frequency-set -g performance
python train.py "$@"
sudo cpupower frequency-set -g powersave

Step 3 — Cap GPU power and stabilise clocks

Most AI workloads are not perfectly power-linear. Modest GPU caps (80–90% of TDP) often reduce watts significantly with small throughput loss. Always test on your model.

  • Enable persistence and view limits:
sudo nvidia-smi -pm 1
nvidia-smi -q -d POWER | sed -n '1,120p'
  • Set a power limit (per GPU or all GPUs):
# Example: cap to 260W per GPU (adjust for your model)
for i in $(nvidia-smi --query-gpu=index --format=csv,noheader); do
  sudo nvidia-smi -i "$i" -pl 260
done
  • Optionally lock clocks for consistency (adjust values to what your GPU supports):
# Lock graphics clocks to a range (query supported with: nvidia-smi -q -d SUPPORTED_CLOCKS)
sudo nvidia-smi -lgc 1200,1500

# Lock memory clocks (example; values must match your GPU's supported list)
sudo nvidia-smi -lmc 877,877

# Reset when done:
sudo nvidia-smi -rgc
sudo nvidia-smi -rmc
  • Monitor under load:
nvidia-smi dmon -s pucvmet -o DT

Practical guidance:

  • Start at 90% of TDP, observe throughput (images/s, tokens/s) and energy per job, then step down by 10–20W.

  • For latency-critical inference, prefer consistent clocks over aggressive caps.

Step 4 — Pin workloads to the right NUMA node

Cross-socket memory traffic and IRQ migrations cost both performance and power. Keep your process near its GPU.

  • Find GPU PCI bus and NUMA node:
GPU_BUS=$(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader | sed -n '1p' | sed 's/^0000://')
echo "GPU0 PCI bus: $GPU_BUS"
NODE=$(cat /sys/bus/pci/devices/0000:$GPU_BUS/numa_node)
echo "GPU0 NUMA node: $NODE"
  • Run your job with CPU and memory bound to that node:
numactl --cpunodebind="$NODE" --membind="$NODE" python train.py
  • You can also pin with taskset if you know core lists:
taskset -c 0-15 python infer.py

This simple locality tweak often reduces CPU package power and shaves milliseconds off hot paths.

Step 5 — Make it stick: tuned profiles and powertop auto-tune

Lock in safe defaults and let SREs flip profiles per node role.

  • Try tuned profiles:
sudo tuned-adm profile balanced
# Other options to evaluate:
# sudo tuned-adm profile throughput-performance
# sudo tuned-adm profile latency-performance
# sudo tuned-adm profile powersave
  • Apply powertop recommendations at boot (be selective in production):
sudo tee /etc/systemd/system/powertop.service >/dev/null <<'EOF'
[Unit]
Description=Powertop tunings
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/powertop --auto-tune

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now powertop.service

Caution: powertop --auto-tune can put some NICs/storage into aggressive power states. Validate with your NIC/storage vendor guidance and your latency SLOs.

Optional — CPU package power capping (Intel RAPL)

If your facility enforces rack power ceilings, RAPL can keep CPUs under a steady envelope.

  • Discover domains:
ls /sys/class/powercap/intel-rapl:0/
  • Set a short-term limit (value in microwatts):
# Example: 125W package limit
echo 125000000 | sudo tee /sys/class/powercap/intel-rapl:0/constraint_0_power_limit_uw
  • Verify:
cat /sys/class/powercap/intel-rapl:0/constraint_0_power_limit_uw

Note: Names and constraints vary by CPU generation; test for throughput impact on data pipelines.

What “good” looks like

  • GPU-bound training: single-digit throughput loss for double-digit power savings is common when capping GPUs to 80–90% TDP—especially if data loading isn’t CPU-throttled.

  • Inference: tighter clocks and NUMA pinning often improve latency stability while lowering average watts.

  • Fleet view: the biggest wins come from repeatable profiles and automation, not hero-tuning per node.

Common pitfalls

  • Over-tuning powertop: can hurt NIC latency/throughput. Exclude critical devices if needed.

  • Forgetting persistence: GPU caps/clocks reset on reboot or driver reload; codify them in systemd or your node init.

  • Ignoring thermals: bad airflow negates tuning; watch inlet/outlet temps and fan curves.

Quick “put it together” script

Example: cap GPUs, set CPU for GPU-bound workloads, bind to the nearest NUMA node, run job, then restore.

#!/usr/bin/env bash
set -euo pipefail

# 1) CPU toward energy-efficient for GPU-bound job
sudo cpupower frequency-set -g powersave || true
for p in /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference; do
  [ -f "$p" ] && echo balance_performance | sudo tee "$p" >/dev/null
done

# 2) GPU caps (adjust per your hardware)
sudo nvidia-smi -pm 1 || true
for i in $(nvidia-smi --query-gpu=index --format=csv,noheader); do
  sudo nvidia-smi -i "$i" -pl 260 || true
done

# 3) Bind to the NUMA node nearest GPU0
GPU_BUS=$(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader | sed -n '1p' | sed 's/^0000://')
NODE=$(cat /sys/bus/pci/devices/0000:$GPU_BUS/numa_node)
echo "Running on NUMA node $NODE"

# 4) Launch workload
exec numactl --cpunodebind="$NODE" --membind="$NODE" "$@"

Run:

chmod +x run_energy_efficient.sh
./run_energy_efficient.sh python train.py --config config.yaml

Conclusion and next steps

Every watt you save without hurting SLA/throughput makes your AI platform cheaper, cooler, and greener. Start with measurement, right-size CPU and GPU power, keep workloads local to their GPUs, and lock in sane defaults with tuned and systemd. Then:

  • Automate: bake these steps into your provisioning (cloud-init/Ansible/Kubernetes DaemonSets).

  • Track: monitor energy per job/model as a first-class metric.

  • Iterate: each model/dataset behaves differently; keep simple toggles to A/B caps and governors.

If you’d like, I can help you turn these snippets into a hardened bootstrap script for your exact CPU/GPU mix and distro.

Related posts