Posted on
• Artificial Intelligence

Artificial Intelligence GPU Passthrough

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence GPU Passthrough on Linux: Near‑Native Performance for Isolated Workloads

Ever wished you could keep your AI stack isolated in a VM but still train at bare‑metal speeds? GPU passthrough makes that possible. With the right hardware and a few careful steps, you can dedicate a physical NVIDIA or AMD GPU directly to a KVM/QEMU virtual machine and run CUDA/ROCm with negligible overhead—perfect for trying different driver versions, CUDA toolkits, or even running multiple segregated environments on a single box.

This guide explains what GPU passthrough is, why it’s valuable for AI work, and how to configure it on a modern Linux host using VFIO, libvirt, and virt‑manager. You’ll get distro‑specific installation commands (apt, dnf, zypper), actionable steps, and a realistic workflow to get you training models inside a VM.

Why GPU Passthrough for AI?

  • Performance: VFIO passthrough provides near‑native GPU performance to a VM. Your PyTorch/TF jobs can train as if they were on bare metal.

  • Isolation: Keep conflicting CUDA versions, drivers, and toolchains in separate VMs. Roll back or snapshot without affecting your host.

  • Reproducibility: Pin a VM image with specific drivers and libraries, then move or clone it.

  • Security/Multitenancy: Give different users or projects dedicated GPUs without sharing a host OS.

What You’ll Need

  • CPU and motherboard with IOMMU support:

    • Intel VT‑d or AMD‑V/AMD‑Vi enabled in BIOS/UEFI.

  • A discrete GPU to pass through (NVIDIA or AMD).

  • Ideally a second GPU or iGPU to drive the host display; otherwise you’ll SSH into a headless host.

  • A Linux host (Debian/Ubuntu, Fedora/RHEL, openSUSE) with KVM support.

Step 1: Verify Hardware and IOMMU Support

1) Check CPU virtualization features:

egrep -o 'vmx|svm' /proc/cpuinfo | sort -u
  • vmx = Intel, svm = AMD.

2) Confirm IOMMU is enabled after boot:

dmesg | egrep -i 'iommu|dmar'

If you don’t see IOMMU enabled, proceed to Step 2 to enable it.

3) Identify your GPU and audio function (most GPUs have an HDMI audio device on a separate function):

lspci -nn | egrep -i 'vga|3d|audio'

4) Inspect IOMMU groups (your GPU and its audio function should be in their own group for safe passthrough):

for g in /sys/kernel/iommu_groups/*; do
  echo "IOMMU Group ${g##*/}:"
  find "$g/devices" -maxdepth 1 -type l -printf '  %f -> %l\n'
done

Step 2: Enable IOMMU in the Kernel

Edit your GRUB kernel cmdline to turn on IOMMU. For Intel:

  • intel_iommu=on iommu=pt

For AMD:

  • amd_iommu=on iommu=pt

1) Edit:

sudo editor /etc/default/grub

Add to GRUB_CMDLINE_LINUX or GRUB_CMDLINE_LINUX_DEFAULT, e.g.:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"

(or amd_iommu=on on AMD)

2) Update GRUB and reboot.

  • Debian/Ubuntu:
sudo update-grub
sudo reboot
  • Fedora/RHEL (UEFI):
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
sudo reboot
  • openSUSE (UEFI):
sudo grub2-mkconfig -o /boot/efi/EFI/opensuse/grub.cfg
sudo reboot

After reboot, rerun:

dmesg | egrep -i 'iommu|dmar'

You should see IOMMU enabled.

Step 3: Install the Virtualization Stack (KVM/QEMU, libvirt, virt‑manager, OVMF)

Install packages, enable libvirtd, and add your user to the right groups.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager ovmf
sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt,kvm $USER
newgrp libvirt
  • Fedora/RHEL (dnf):
sudo dnf install -y @virtualization virt-manager virt-install edk2-ovmf
sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt,kvm $USER
newgrp libvirt
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y qemu-kvm libvirt-daemon libvirt-tools virt-manager virt-install ovmf
sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt,kvm $USER
newgrp libvirt

Tip: Ensure your CPU supports virtualization in the host kernel:

egrep -c '(vmx|svm)' /proc/cpuinfo
lsmod | egrep 'kvm_|kvm'

Step 4: Bind the GPU to VFIO

You’ll instruct the host to reserve the GPU for the VM by binding it to vfio-pci early in boot.

1) Get device IDs for the GPU and its audio function:

lspci -nn | egrep -i 'vga|3d|audio'
# Example output snippet:
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2204]
# 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aef]

Note the IDs (e.g., 10de:2204 and 10de:1aef).

2) Create a VFIO config:

echo "options vfio-pci ids=10de:2204,10de:1aef disable_vga=1" | sudo tee /etc/modprobe.d/vfio.conf

3) Blacklist host GPU drivers (if the host is not using this GPU for display). For NVIDIA:

printf "blacklist nouveau\nblacklist nvidia\n" | sudo tee /etc/modprobe.d/blacklist-gpu.conf

For AMD GPU:

printf "blacklist amdgpu\nblacklist radeon\n" | sudo tee /etc/modprobe.d/blacklist-gpu.conf

Note: If your host relies on the same GPU for display, do NOT blacklist the driver; instead, use early VFIO binding with a second GPU or plan for headless host access (SSH).

4) Regenerate initramfs and reboot.

  • Debian/Ubuntu:
sudo update-initramfs -u
sudo reboot
  • Fedora/RHEL (dracut):
sudo dracut -f
sudo reboot
  • openSUSE (dracut):
sudo dracut -f
sudo reboot

5) Verify the GPU is using vfio-pci:

lspci -k -s 01:00.0
# Kernel driver in use: vfio-pci

Step 5: Create a UEFI VM and Attach the GPU

You can use virt‑manager for a straightforward GUI setup.

  • Launch virt‑manager and create a new VM.

  • Firmware: Choose UEFI (OVMF).

  • Chipset/Machine type: Use Q35 with PCIe.

  • CPU: Pass through host-model or choose a fixed topology. Consider CPU pinning for latency‑sensitive workloads.

  • Memory: Use large pages (hugepages) for performance if desired.

  • Storage: VirtIO SCSI or VirtIO block with writeback caching for speed.

  • Before finishing, go to Add Hardware → PCI Host Device, and add:

    • The GPU (e.g., 01:00.0)
    • Its Audio function (e.g., 01:00.1)
    • Check “Multifunction” if necessary and ensure both are on the same PCIe root port.

  • Optional: If your GPU needs a vBIOS ROM to initialize in a VM (common on consumer NVIDIA cards without a secondary GPU), supply a dumped vBIOS file in the device options.

Install a Linux guest (e.g., Ubuntu, Fedora, openSUSE) for your AI workloads.

Step 6: Install GPU Drivers and AI Stack Inside the Guest

Below are examples for NVIDIA guests; AMD ROCm is also viable but varies by distro and GPU generation.

A) NVIDIA drivers and CUDA inside the guest

  • Ubuntu/Debian (apt):
sudo apt update
# Option 1: Recommended tested driver
sudo ubuntu-drivers autoinstall

# Option 2: Specific driver + CUDA toolkit (from Ubuntu repo)
sudo apt install -y nvidia-driver-535 nvidia-cuda-toolkit

# Verify
nvidia-smi
nvcc --version
  • Fedora/RHEL (dnf, via RPM Fusion):
# Enable RPM Fusion repos (adjust releasever if needed)
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                    https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

# Install NVIDIA driver + CUDA userspace
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda

# Build kernel module (may take a moment), then reboot
sudo reboot

# Verify
nvidia-smi
  • openSUSE (zypper, NVIDIA repo):
# Add NVIDIA repo (example for openSUSE Leap 15.5; adjust for your version)
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/15.5 NVIDIA
sudo zypper refresh
sudo zypper install -y nvidia-driver-G06

# For CUDA toolkit (official NVIDIA CUDA repo provides newest versions):
# See: https://developer.nvidia.com/cuda-downloads to add the matching repo
# Example (adjust for your distro/version) then:
# sudo zypper install -y cuda-toolkit

# Verify
nvidia-smi

B) Install your AI frameworks (guest)

# Example: Miniconda + PyTorch (CUDA)
cd /tmp
curl -fsSLO https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
conda create -y -n ai python=3.11
conda activate ai
# PyTorch (example; will auto-pick CUDA build when available)
pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
python - <<'PY'
import torch
print("CUDA:", torch.cuda.is_available(), "Device:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else None)
PY

Note on Secure Boot: If your guest uses Secure Boot, unsigned kernel modules may fail to load. Either disable Secure Boot in the guest firmware or enroll a MOK/sign modules accordingly.

Real‑World Example: Single‑Host, Multi‑Stack Development

  • Host runs stable, minimal drivers and services.

  • VM #1 (Ubuntu 22.04): CUDA 12.1 + PyTorch nightly for experimentation.

  • VM #2 (Fedora): CUDA 11.8 + TensorRT for production inference.

  • Each VM has snapshots, so you can roll back driver experiments in seconds without touching the host. With VFIO passthrough, both VMs can take turns using the exact same physical RTX GPU with near‑native speed.

Troubleshooting Tips

  • GPU still grabbed by host: Ensure vfio-pci binding happens early. Re‑check /etc/modprobe.d files, initramfs rebuild, and blacklists. Confirm with lspci -k.

  • IOMMU grouping issues: Some consumer boards group multiple devices. If your GPU is in a group with other critical devices, consider moving it to a different slot, updating BIOS, or using the ACS override patch (advanced; security trade‑offs).

  • Stuck at boot or black screen: Remove blacklists temporarily (boot with modprobe.blacklist= overrides), confirm you’re not blacklisting the GPU that drives your host display.

  • Virt‑manager device options: Use OVMF (UEFI), Q35, PCIe root ports, and add both GPU and its audio function. Some GPUs need a clean vBIOS ROM dump.

  • Performance tuning: CPU pinning, hugepages, VirtIO SCSI single I/O thread, host‑model CPU in libvirt.

Conclusion and Next Steps

GPU passthrough lets you maintain clean, reproducible AI environments while keeping the raw performance you need for training and inference. You’ve seen how to:

  • Enable IOMMU and set up VFIO

  • Install KVM/QEMU, libvirt, and virt‑manager

  • Bind a physical GPU to a VM

  • Install NVIDIA drivers/CUDA inside the guest and verify with nvidia-smi

Your next step: Try passing through your GPU to a fresh Linux guest and run a short training job. Once you’re comfortable, create a “golden” VM image for your preferred AI stack and snapshot it before big experiments. If you need multi‑tenant GPU access without full VMs, explore NVIDIA vGPU/mdev (where supported) or containers with the NVIDIA Container Toolkit as a complementary approach.

Questions or stuck on a specific device? Drop your IOMMU group layout and lspci -nn output in the comments—we can help you fine‑tune your setup.

Related posts