Posted on
• Artificial Intelligence

GPU Passthrough for Artificial Intelligence Labs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

GPU Passthrough for Artificial Intelligence Labs: Near-Bare-Metal AI in VMs

Ever wished you could hand your RTX/AMD GPU directly to a virtual machine and train models at almost native speed—without juggling dual-boot or risking global package conflicts? GPU passthrough makes that real. With KVM, VFIO, and a bit of kernel tuning, you can carve out isolated, reproducible AI sandboxes that deliver 95–100% of bare-metal performance for many workloads.

This guide explains why GPU passthrough is worth it, and walks you through a practical, distro-agnostic setup. You’ll get step-by-step instructions and commands for apt, dnf, and zypper, plus tips for validation and troubleshooting.

Why GPU Passthrough for AI Labs?

  • Isolation and reproducibility: Students and researchers get clean, snapshot-able VMs with pinned driver/toolkit versions.

  • Multi-tenant efficiency: Share a single GPU host across many labs, safely partitioned per VM.

  • Performance: PCIe passthrough bypasses emulation—latency is low, bandwidth is high, overhead is often in the low single digits.

  • Flexibility: Run Ubuntu, Fedora, or openSUSE guests with their own CUDA/ROCm stacks, CI pipelines, and security policies.

What you’ll need

  • CPU/board with IOMMU (Intel VT-d or AMD-Vi/SVM) and BIOS options for:

    • SVM/AMD-V or VT-x/VT-d
    • Above 4G decoding (recommended)
    • Resizable BAR (optional)

  • A GPU in its own IOMMU group (or a board that supports good PCIe isolation)

  • Ideally a second GPU or onboard graphics for the host display (headless host is fine)

  • A recent Linux host (Ubuntu/Debian, Fedora/RHEL, openSUSE) with KVM support

Actionable Plan (4 Steps)

1) Prepare the host: Virtualization stack + IOMMU

Install KVM/libvirt/OVMF and essentials.

  • Ubuntu/Debian (apt):
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients virt-manager ovmf pciutils swtpm swtpm-tools bridge-utils dnsmasq numactl
sudo usermod -aG libvirt,kvm $USER
sudo systemctl enable --now libvirtd
  • Fedora/RHEL (dnf):
sudo dnf groupinstall -y "Virtualization"
sudo dnf install -y virt-manager edk2-ovmf pciutils swtpm swtpm-tools numactl
sudo usermod -aG libvirt,kvm $USER
sudo systemctl enable --now libvirtd
  • openSUSE (zypper):
sudo zypper install -y qemu-kvm libvirt virt-manager ovmf pciutils swtpm swtpm-tools numactl
sudo usermod -aG libvirt,kvm $USER
sudo systemctl enable --now libvirtd

Enable IOMMU in your kernel command line.

  • Determine your vendor:
lscpu | grep -i vendor
  • Edit /etc/default/grub and append:
    • Intel: intel_iommu=on iommu=pt
    • AMD: amd_iommu=on iommu=pt

Example line:

GRUB_CMDLINE_LINUX="... amd_iommu=on iommu=pt"
  • Update GRUB and reboot:
    • Ubuntu/Debian:
sudo update-grub
  • Fedora/RHEL (BIOS):
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  • Fedora/RHEL (UEFI, Fedora path example):
sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
  • openSUSE:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  • Reboot:
sudo reboot

Verify IOMMU is active:

dmesg | grep -e IOMMU -e DMAR

List IOMMU groups:

find /sys/kernel/iommu_groups -type l | sort

2) Bind the GPU to VFIO (passthrough driver)

Identify your GPU and its paired functions (e.g., GPU + HDMI audio share the same slot, different function).

lspci -nnk | grep -A3 -E "VGA|3D|Audio"

Note the vendor:device IDs in brackets, for example 10de:1eb8 (GPU) and 10de:10f0 (audio). Bind them to vfio-pci.

  • Create a VFIO modprobe config:
sudo nano /etc/modprobe.d/vfio.conf

Add (replace IDs with yours):

options vfio-pci ids=10de:1eb8,10de:10f0 disable_vga=1
  • Blacklist host GPU drivers for the passthrough device only (do NOT blacklist the driver for the GPU driving your host display):
sudo nano /etc/modprobe.d/blacklist-gpu.conf

Add as appropriate for your device:

blacklist nouveau
blacklist nvidia
# or for AMD passthrough device:
# blacklist amdgpu
# blacklist radeon
  • Regenerate initramfs and reboot:
    • Ubuntu/Debian:
sudo update-initramfs -u
  • Fedora/RHEL/openSUSE:
sudo dracut -f
  • Reboot:
sudo reboot

Confirm it’s bound to vfio-pci:

lspci -nnk -d 10de:1eb8
# Should show: Kernel driver in use: vfio-pci

Tip: If the device won’t bind (still using nvidia/amdgpu), ensure no display managers or services are using it and that you captured all functions (function 0 and 1).

3) Create the VM and attach the GPU

Use OVMF (UEFI) and Q35 machine type. You can use virt-manager (GUI) or CLI.

  • virt-install (example; adjust paths and sizes):

Ubuntu/Debian OVMF path:

virt-install \
  --name ai-guest \
  --memory 32768 --memorybacking hugepages=on \
  --vcpus 8 \
  --cpu host-passthrough,topology.sockets=1,cores=4,threads=2 \
  --disk size=200,bus=virtio \
  --cdrom /var/lib/libvirt/boot/ubuntu-22.04.iso \
  --os-variant ubuntu22.04 \
  --graphics none \
  --network network=default,model=virtio \
  --boot loader=/usr/share/OVMF/OVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/OVMF/OVMF_VARS.fd \
  --machine q35

Fedora/RHEL OVMF path:

--boot loader=/usr/share/edk2/ovmf/OVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/edk2/ovmf/OVMF_VARS.fd

openSUSE OVMF path (example):

--boot loader=/usr/share/qemu/ovmf-x86_64-code.bin,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/qemu/ovmf-x86_64-vars.bin

Attach the GPU (and its audio/function). With virt-manager: Add Hardware -> PCI Host Device -> select both functions. Or via XML:

virsh edit ai-guest

Add inside the domain:

<memoryBacking>
  <hugepages/>
</memoryBacking>

<devices>
  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x2d' slot='0x00' function='0x0'/>
    </source>
  </hostdev>
  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x2d' slot='0x00' function='0x1'/>
    </source>
  </hostdev>
</devices>

Use your actual domain/bus/slot/function values from lspci -nn.

Optional performance tuning:

  • CPU pinning (example pin vCPUs to physical cores 2–9):
<cputune>
  <vcpupin vcpu='0' cpuset='2'/>
  <vcpupin vcpu='1' cpuset='3'/>
  <vcpupin vcpu='2' cpuset='4'/>
  <vcpupin vcpu='3' cpuset='5'/>
  <vcpupin vcpu='4' cpuset='6'/>
  <vcpupin vcpu='5' cpuset='7'/>
  <vcpupin vcpu='6' cpuset='8'/>
  <vcpupin vcpu='7' cpuset='9'/>
</cputune>
  • Reserve hugepages on the host (example: 16 GB of 2 MB pages):
echo 8192 | sudo tee /proc/sys/vm/nr_hugepages

Persist via /etc/sysctl.d/99-hugepages.conf:

vm.nr_hugepages = 8192

4) Install GPU drivers inside the guest and validate

Boot the VM, install your OS, then install NVIDIA (example) or AMD drivers. Below are distro-native approaches for NVIDIA.

  • Ubuntu/Debian guest (apt):
sudo apt update
sudo apt install -y build-essential dkms linux-headers-$(uname -r)
sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi
  • Fedora guest (dnf):
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
nvidia-smi
  • openSUSE guest (zypper) – add NVIDIA repo, then install:
sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed NVIDIA
# For Leap, replace 'tumbleweed' with your Leap version path from NVIDIA docs
sudo zypper refresh
sudo zypper install -y nvidia-driver-G06
sudo reboot
nvidia-smi

Install PyTorch with CUDA and run a quick check:

python3 -m pip install --upgrade pip
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
python3 - <<'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
print("Device:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "N/A")
PY

Optional quick training smoke test:

python3 - <<'PY'
import torch, torch.nn as nn
x = torch.randn(1024, 2048, device='cuda')
w = torch.randn(2048, 1024, device='cuda')
for _ in range(5):
    y = x @ w
    y = nn.functional.relu(y)
torch.cuda.synchronize()
print("OK")
PY

Real-World Notes and Troubleshooting

  • Check VFIO and IOMMU logs:
dmesg | grep -e VFIO -e IOMMU -e DMAR
journalctl -b | grep -i vfio

  • IOMMU groups not isolated: Some motherboards group multiple PCIe slots together. If your GPU shares a group with other devices you cannot pass through safely, try different slots or update BIOS. As a last resort, the ACS override kernel parameter exists (pcie_acs_override=downstream,multifunction), but it reduces isolation and can be unstable—use only if you understand the risks.

  • Attach both functions: Many GPUs have an audio function; pass both or the guest may fail to init drivers.

  • Above 4G decoding: If the VM refuses to initialize the GPU or you see BAR-mapping errors, enable “Above 4G Decoding” in BIOS.

  • Headless host: If you bind your only display GPU to VFIO, you’ll lose local console graphics. Use SSH or IPMI/serial for management, or keep a separate host GPU.

  • Performance tips:

    • Use host-passthrough CPU in libvirt for best instruction set exposure.
    • Keep VM vCPUs/hugepages NUMA-local to the GPU’s PCIe root complex for low latency.
    • Prefer virtio for disk/network; consider a bridged network for lab environments.

Example: A Small AI Lab on One Host

  • Host: 64-core EPYC, 256 GB RAM, 2×RTX 4090

  • Two VMs: Each gets 1 GPU, 16 vCPUs, 64 GB RAM with hugepages

  • Students SSH into their VM, run conda or pip, train models. Snapshots keep environments clean; passthrough keeps training fast and consistent.

Conclusion and Next Steps (CTA)

You now have a repeatable path to put real GPUs inside virtual machines for AI work—without giving up performance. Next:

  • Automate with Ansible: template your libvirt XML and host tuning.

  • Add quotas and scheduling: integrate with Slurm or orchestration that boots VMs on demand.

  • Explore advanced sharing: NVIDIA MIG (A100/Hopper) or SR-IOV-capable GPUs for finer-grained multi-tenancy.

If you’d like a follow-up post with a complete Ansible playbook and libvirt hooks to hot-plug GPUs per user or per CI job, let me know what distro and GPU you’re targeting.

Related posts