Posted on
• Artificial Intelligence

Multi-GPU Artificial Intelligence Setups

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Multi‑GPU Artificial Intelligence Setups on Linux: From Zero to Scaling Hero

If you’ve ever watched your single GPU crawl through a training job and thought “there has to be a faster way,” you’re right. Multi‑GPU setups can slash training times, let you scale models and batch sizes, and future‑proof your AI workstation or lab cluster. The catch: getting multiple GPUs to play nicely on Linux requires a bit of hardware awareness, the right drivers, and sane defaults for your framework.

This guide gives you a practical, Bash‑first walkthrough to stand up a reliable multi‑GPU environment on Linux. You’ll learn how to verify your hardware and topology, install the right drivers and tools, configure PyTorch for data‑parallel training, and validate performance—using commands you can paste straight into your terminal.

Why multi‑GPU is worth it

  • Time-to-result: Parallelizing across 2–8 GPUs can cut training time by multiples when your data pipeline and communication are configured correctly.

  • Larger models and batches: Aggregate VRAM lets you train larger models or use larger batch sizes for better utilization and throughput.

  • Cost efficiency: On-prem hardware amortized over many experiments can beat cloud costs for heavy users.

  • Reproducibility and control: Full control over drivers, kernels, libraries, and network lets you tune for stability and performance.

Prerequisites: Check your hardware and OS

1) Verify GPUs and lanes

lspci | egrep -i 'vga|3d'

  • Prefer x16 slots on PCIe Gen4/Gen5.

  • If you have NVLink bridges, seat them correctly and check your board manual.

2) Confirm Linux kernel and distribution are up to date

  • Keep a recent LTS kernel for driver stability.
uname -r

3) Ensure adequate power and cooling

  • Multi‑GPU rigs pull serious wattage and emit heat. Undervolting/underclocking may be necessary for dense builds.

Step 1: Install baseline developer tools

These packages make building, debugging, and running Python ML stacks smoother.

  • Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y build-essential python3 python3-venv python3-pip git tmux pciutils openssh-server
  • Fedora/RHEL (dnf)
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y python3 python3-venv python3-pip git tmux pciutils openssh-server
  • openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y python3 python3-venv python3-pip git tmux pciutils openssh

Optional but handy:

sudo -H pip3 install --upgrade pip wheel setuptools

Step 2: Install and validate GPU drivers

NVIDIA is the most common path for multi‑GPU deep learning, thanks to NCCL and strong framework support. If you’re on AMD/ROCm, see the ROCm note further below.

NVIDIA driver installation

  • Ubuntu (apt)
sudo apt update
ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot

Tip: If you prefer a specific version, replace the autoinstall step with:

sudo apt install -y nvidia-driver-535   # replace 535 with the recommended version for your GPU
sudo reboot
  • Fedora (dnf) via RPM Fusion
sudo dnf install -y \
  https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
  • openSUSE (zypper) For Tumbleweed:
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed NVIDIA
sudo zypper refresh
sudo zypper install -y nvidia-driver-G06
sudo reboot

For Leap (example 15.5; adjust if needed):

sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/15.5 NVIDIA
sudo zypper refresh
sudo zypper install -y nvidia-driver-G06
sudo reboot

If unsure which meta‑package you need:

zypper se -s nvidia-driver

After reboot, validate:

nvidia-smi

Enable persistence and check topology:

sudo nvidia-smi -pm 1
nvidia-smi topo -m

You want high‑bandwidth links (NVLink/PIX/PHB) where possible.

AMD ROCm note

Multi‑GPU with ROCm (RCCL) works but requires specific kernel/driver versions and supported GPUs. Because installation steps differ by distro and ROCm version, follow AMD’s official ROCm repository instructions for your OS, then verify:

/opt/rocm/bin/rocminfo
/opt/rocm/bin/rocm-smi

For PyTorch ROCm wheels, see the environment setup below.

Step 3: Create a Python environment with GPU‑enabled frameworks

Use virtual environments to isolate dependencies.

Create and activate:

python3 -m venv ~/venvs/ai
source ~/venvs/ai/bin/activate
pip install --upgrade pip

Install PyTorch with GPU support.

  • NVIDIA CUDA wheels (example: CUDA 12.4 index)
pip install --index-url https://download.pytorch.org/whl/cu124 torch torchvision torchaudio
  • AMD ROCm wheels (example: ROCm 6.0 index)
pip install --index-url https://download.pytorch.org/whl/rocm6.0 torch torchvision torchaudio
  • CPU fallback
pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision torchaudio

Verify device visibility:

python - << 'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
print("GPU count:", torch.cuda.device_count())
if torch.cuda.is_available():
    for i in range(torch.cuda.device_count()):
        print(i, torch.cuda.get_device_name(i))
PY

Tip: You don’t need to install the full CUDA toolkit when using prebuilt wheels; you do need a compatible NVIDIA driver.

Step 4: Configure multi‑GPU communication

NVIDIA’s NCCL (and AMD’s RCCL) optimize GPU‑to‑GPU communication. You often get these libraries bundled with framework wheels, but runtime environment matters.

  • Inspect network and choose the right interface for multi‑node later:
ip -o -4 addr show | awk '{print $2,$4}'
  • Useful environment variables for NCCL single/multi‑node:
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1            # set to 0 if you have InfiniBand/RDMA configured
export NCCL_SOCKET_IFNAME=eth0      # change to your primary NIC (e.g., enp175s0, bond0)
export CUDA_DEVICE_MAX_CONNECTIONS=1
  • Keep the GPUs “awake”:
sudo nvidia-smi -pm 1
  • Optional: enable CUDA MPS to improve multi‑process sharing on single GPU
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-mps
mkdir -p $CUDA_MPS_PIPE_DIRECTORY $CUDA_MPS_LOG_DIRECTORY
nvidia-cuda-mps-control -d

Step 5: Run a real multi‑GPU training job (PyTorch DDP)

Here’s a minimal DistributedDataParallel (DDP) training skeleton you can extend.

  • train_ddp.py
#!/usr/bin/env python3
import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch import nn, optim
from torch.utils.data import DataLoader, DistributedSampler
from torchvision import datasets, transforms, models

def setup(rank, world_size):
    dist.init_process_group(backend="nccl", init_method="env://",
                            world_size=world_size, rank=rank)
    torch.cuda.set_device(rank)

def cleanup():
    dist.destroy_process_group()

def train(rank, world_size, epochs=2, batch_size=128):
    setup(rank, world_size)

    transform = transforms.Compose([transforms.ToTensor()])
    dataset = datasets.CIFAR10(root="./data", train=True, download=True, transform=transform)
    sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank, shuffle=True)
    loader = DataLoader(dataset, batch_size=batch_size, sampler=sampler, num_workers=4, pin_memory=True)

    model = models.resnet18().cuda(rank)
    model = DDP(model, device_ids=[rank])

    criterion = nn.CrossEntropyLoss().cuda(rank)
    optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

    scaler = torch.cuda.amp.GradScaler()
    model.train()
    for epoch in range(epochs):
        sampler.set_epoch(epoch)
        for step, (x, y) in enumerate(loader):
            x = x.cuda(rank, non_blocking=True)
            y = y.cuda(rank, non_blocking=True)
            optimizer.zero_grad(set_to_none=True)
            with torch.cuda.amp.autocast():
                loss = criterion(model(x), y)
            scaler.scale(loss).backward()
            scaler.step(optimizer)
            scaler.update()
            if rank == 0 and step % 100 == 0:
                print(f"Epoch {epoch} Step {step} Loss {loss.item():.4f}")
    cleanup()

def main():
    world_size = torch.cuda.device_count()
    assert world_size > 1, "Need at least 2 GPUs"
    mp.spawn(train, args=(world_size, 2, 128), nprocs=world_size, join=True)

if __name__ == "__main__":
    main()

Launch across all visible GPUs:

source ~/venvs/ai/bin/activate
export NCCL_DEBUG=INFO
torchrun --standalone --nproc-per-node=$(nvidia-smi -L | wc -l) train_ddp.py

Monitor utilization in another terminal:

watch -n1 nvidia-smi
# or
nvidia-smi dmon -s pucvmet

Multi‑node quick start (two nodes)

On all nodes:

pip install --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Pick a master node IP and port, then:

  • Node 0
export MASTER_ADDR=10.0.0.10
export MASTER_PORT=29500
torchrun --nnodes=2 --nproc-per-node=4 --node_rank=0 --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train_ddp.py
  • Node 1
export MASTER_ADDR=10.0.0.10
export MASTER_PORT=29500
torchrun --nnodes=2 --nproc-per-node=4 --node_rank=1 --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train_ddp.py

Ensure passwordless SSH is set up if you script remote launches. Install SSH server if missing:

  • apt
sudo apt install -y openssh-server
  • dnf
sudo dnf install -y openssh-server
  • zypper
sudo zypper install -y openssh

Performance tips that actually matter

  • Batch size and gradient accumulation: Saturate GPUs without OOM. If you can’t fit, lower per‑GPU batch size and accumulate gradients.

  • Mixed precision: Use autocast + GradScaler (shown above). It’s free speed on modern GPUs.

  • Data pipeline: Use multiple workers, pin memory, and fast local storage (NVMe). Watch CPU/io wait with pidstat and iostat.

  • Topology awareness: Prefer GPUs connected by NVLink or same PCIe root complex. Check nvidia-smi topo -m.

  • Environment hygiene: Keep drivers, CUDA runtime (from wheels), and frameworks version‑aligned. One change at a time.

Troubleshooting quick hits

  • NCCL hangs or timeouts
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=0
export NCCL_SOCKET_IFNAME=eth0   # set correctly
  • Only one GPU shows up
nvidia-smi
lspci | egrep -i 'vga|3d'
dmesg | egrep -i 'nvrm|nvidia'

Reseat GPUs, check power, BIOS Above 4G decoding, and ReBAR settings.

  • PyTorch can’t find CUDA

  • Ensure your torch wheel matches the CUDA runtime it ships with (e.g., cu124), and your NVIDIA driver is new enough for that CUDA version.

Optional: Containers with GPU access

If you prefer containers, install Docker and the NVIDIA Container Toolkit, then:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Containers are great for isolating dependencies across teams and CI.

Conclusion and next steps

Multi‑GPU on Linux isn’t magic—just a sequence: validate hardware, install stable drivers, pick the right framework wheels, use DDP/strategy APIs, and monitor. Start by getting a 2‑GPU DDP job to run cleanly, then scale up workers and nodes. Your next step:

  • Set up your Python venv and run the DDP example on two GPUs today.

  • Add telemetry (nvidia-smi, logs) and tweak batch size, mixed precision, and data workers.

  • Once stable, move to 4–8 GPUs or multi‑node, and record your throughput gains.

If you want a follow‑up post with a fully containerized, reproducible multi‑node setup and automated benchmarks, say the word—and we’ll script it end‑to‑end.

Related posts