Posted on
• Artificial Intelligence

Getting Started with Artificial Intelligence GPUs on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Getting Started with Artificial Intelligence GPUs on Linux

AI workloads love GPUs—but getting a Linux box from zero to “GPU-accelerated and training” can feel like a maze of drivers, toolkits, and version pinning. This guide gets you to a working setup quickly and safely, with vendor-specific steps, verification commands, and real-world examples. You’ll also get distro-accurate install snippets for apt (Debian/Ubuntu), dnf (Fedora/RHEL/CentOS Stream), and zypper (openSUSE).

Why this matters:

  • GPUs can deliver 10–100x speedups for training and inference compared to CPUs.

  • The “stack” is layered: kernel + driver + compute toolkit (CUDA/ROCm) + framework (PyTorch/TensorFlow).

  • Correctly matching versions keeps you out of dependency hell and in the fast lane.

1) Know your GPU and prep the system

Identify your GPU and install basic build tools and headers (often required for drivers).

Detect your GPU:

lspci | grep -E "NVIDIA|AMD|Intel"

Install pciutils (if lspci is missing):

  • apt:

    sudo apt update && sudo apt install -y pciutils

  • dnf:

    sudo dnf install -y pciutils

  • zypper:

    sudo zypper install -y pciutils

Install build tools and kernel headers:

  • apt (Debian/Ubuntu):

    sudo apt update
sudo apt install -y build-essential dkms linux-headers-$(uname -r)

  • dnf (Fedora/RHEL/CentOS Stream):

    sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y kernel-devel kernel-headers

  • zypper (openSUSE):

    sudo zypper install -y gcc make kernel-default-devel dkms

Tip: If Secure Boot is enabled, driver modules may not load unless signed. Consider temporarily disabling Secure Boot or signing modules yourself.

2) Install your GPU driver and compute toolkit

Pick the path for your hardware. After driver install, reboot.

A) NVIDIA: Driver + CUDA

Ubuntu/Debian (simplest: distro driver):

sudo apt update
# Ubuntu: auto-selects recommended proprietary driver
sudo ubuntu-drivers autoinstall
sudo reboot

Optional (CUDA Toolkit from Ubuntu repos; versions may lag):

sudo apt update
sudo apt install -y nvidia-cuda-toolkit

Fedora (via RPM Fusion):

# Enable RPM Fusion
sudo dnf install -y \
  https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
# NVIDIA driver + CUDA libraries from RPM Fusion
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo systemctl reboot

Optional (CUDA Toolkit from NVIDIA repo):

sudo dnf config-manager --add-repo \
  https://developer.download.nvidia.com/compute/cuda/repos/fedora$(rpm -E %fedora)/x86_64/cuda-fedora$(rpm -E %fedora).repo
sudo dnf install -y cuda-toolkit

openSUSE (official NVIDIA repo):

# Add NVIDIA repo (Leap example; adjust for your release—Leap/Tumbleweed)
. /etc/os-release
if [ "$NAME" = "openSUSE Leap" ]; then
  sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/$VERSION_ID/ nvidia
else
  sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed/ nvidia
fi
sudo zypper refresh
# Install compute-capable driver (G06 series supports most modern GPUs)
sudo zypper install -y nvidia-computeG06
sudo reboot

Optional (CUDA Toolkit from NVIDIA repo):

# For Leap 15.x; adjust 'opensuse15' as needed
sudo zypper addrepo --refresh \
  https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/ cuda
sudo zypper refresh
sudo zypper install -y cuda-toolkit

Verify NVIDIA:

nvidia-smi

B) AMD: AMDGPU + ROCm

ROCm is officially supported on specific Ubuntu LTS and RHEL/CentOS/Fedora versions with recent GPUs (RDNA/CDNA). Use AMD’s amdgpu-install.

Ubuntu (22.04/24.04 LTS; example uses ROCm 6.x):

VER=6.1
wget https://repo.radeon.com/amdgpu-install/${VER}/ubuntu/jammy/amdgpu-install_${VER}-1_all.deb
sudo apt install -y ./amdgpu-install_${VER}-1_all.deb
# Install kernel driver + ROCm user-space
sudo amdgpu-install --usecase=rocm,dkms --no-32 --accept-eula
sudo reboot

Fedora/RHEL (example for RHEL 9/Fedora; adapt VER and distro path per AMD docs):

VER=6.1
# RHEL 9 example:
sudo dnf install -y https://repo.radeon.com/amdgpu-install/${VER}/rhel/9/amdgpu-install-${VER}-1.el9.noarch.rpm
# Fedora example (replace 39 with your release):
# sudo dnf install -y https://repo.radeon.com/amdgpu-install/${VER}/fedora/39/amdgpu-install-${VER}-1.fc39.noarch.rpm

sudo amdgpu-install --usecase=rocm,dkms --no-32 --accept-eula
sudo reboot

Note for openSUSE: ROCm is not officially supported on openSUSE at the time of writing. If you’re on openSUSE, prefer a supported distro for ROCm or use containers on a supported host.

Verify AMD:

/opt/rocm/bin/rocminfo | head
/opt/rocm/bin/rocm-smi

3) Set up a Python environment and install frameworks

Create a clean virtual environment and install GPU-enabled builds. First ensure Python + venv:

  • apt:

    sudo apt install -y python3 python3-pip python3-venv

  • dnf:

    sudo dnf install -y python3 python3-pip python3-virtualenv

  • zypper:

    sudo zypper install -y python3 python3-pip python3-virtualenv

Create and activate venv:

python3 -m venv ~/.venvs/ai
source ~/.venvs/ai/bin/activate
pip install --upgrade pip

Install PyTorch (pick one):

  • NVIDIA CUDA 12.1 wheels:

    pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio

  • AMD ROCm 6.0 wheels (ensure ROCm 6.0+ installed):

    pip install --index-url https://download.pytorch.org/whl/rocm6.0 torch torchvision

Install TensorFlow (CUDA 12.x-supported builds):

pip install "tensorflow[and-cuda]==2.15.*"

Quick GPU test in Python:

python - << 'PY'
import torch, sys
print("PyTorch CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("Device:", torch.cuda.get_device_name(0))
try:
    import tensorflow as tf
    gpus = tf.config.list_physical_devices("GPU")
    print("TensorFlow GPUs:", gpus)
except Exception as e:
    print("TensorFlow check error:", e, file=sys.stderr)
PY

4) Verify with CLI and run a real workload

NVIDIA:

nvidia-smi

AMD:

/opt/rocm/bin/rocminfo | grep -E "Name:|gfx"

Run a tiny inference (PyTorch + ResNet18):

python - << 'PY'
import torch
from torchvision import models
m = models.resnet18(weights="IMAGENET1K_V1").eval()
x = torch.randn(1,3,224,224)
if torch.cuda.is_available():
    m = m.cuda(); x = x.cuda()
with torch.no_grad():
    y = m(x)
print("Output shape:", y.shape, "| On CUDA:", x.is_cuda)
PY

If that prints a 1×1000 tensor and CUDA=True on a GPU box, you’re good.

5) Optional: Containers for reproducibility (Docker/Podman + GPU runtime)

Install Docker Engine:

  • apt:

    sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(. /etc/os-release; echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker

  • dnf (Fedora example):

    sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker

  • zypper:

    sudo zypper install -y docker
sudo systemctl enable --now docker

NVIDIA Container Toolkit:

  • apt:

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

  • dnf (RPM-based):

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
sudo rpm --import https://nvidia.github.io/libnvidia-container/gpgkey
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

  • zypper (SUSE/openSUSE):

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
sudo rpm --import https://nvidia.github.io/libnvidia-container/gpgkey
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU in container (NVIDIA):

docker run --rm --gpus all nvcr.io/nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

AMD ROCm containers (host must have AMDGPU + ROCm installed):

docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  rocm/pytorch:rocm6.0.0 python -c "import torch; print(torch.cuda.is_available()); print(torch.version.hip)"

Podman users: replace docker with podman, and on NVIDIA use --hooks-dir=/usr/share/containers/oci/hooks.d if your distro integrates nvidia-ctk via OCI hooks.

Common pitfalls and quick fixes

  • Nouveau vs NVIDIA: Blacklist nouveau when using proprietary NVIDIA drivers if conflicts arise.

  • Kernel updates: Rebuild or reinstall drivers after major kernel updates (akmods/DKMS usually handle this).

  • Mixed toolkit versions: Let pip wheels provide the right cuDNN/CUDA runtime (PyTorch/TensorFlow ship compatible runtimes). Avoid system-wide conflicting CUDA libs in LD_LIBRARY_PATH.

  • Secure Boot: Unsigned kernel modules won’t load; disable or sign them.

Conclusion and Next Steps

You’ve assembled the full GPU AI stack on Linux: drivers, compute toolkits, frameworks, and containers—plus verification and a real inference test. Your next moves:

  • Benchmark your setup with your actual model.

  • Containerize your workflow for portability.

  • Keep drivers/toolkits aligned with your framework’s recommended versions.

If you want a follow-up, ask for a “production-ready GPU workstation checklist” or a “multi-GPU training starter pack” and I’ll tailor it to your distro and hardware.

Related posts