Posted on
• Artificial Intelligence

Artificial Intelligence GPU Homelab Guide

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence GPU Homelab Guide (Linux + Bash)

Want to run large language models, image generation, or fine-tune small models at home—privately and without cloud bills? With a single GPU and Linux, you can build a local AI lab that’s fast, flexible, and fun to tinker with.

In this guide you’ll:

  • Understand the value and trade-offs of a GPU homelab

  • Set up your Linux box with drivers, containers, and frameworks

  • Run real workloads (LLMs, diffusion) locally

  • Optimize, monitor, and automate with Bash

No fluff—just working commands and a pragmatic path.

Why an AI GPU homelab?

  • Privacy by default: Your data never leaves the box.

  • Predictable cost: One-time hardware, minimal ongoing expense.

  • Low latency: On-device inference is instant for many tasks.

  • Skills and control: Learn drivers, containers, compilers, and performance tuning.

The caveat: you own the plumbing (drivers, compatibility, thermals). This guide keeps it practical and reproducible.

1) Plan the hardware (what actually matters)

  • VRAM first:

    • 8–12 GB = lightweight LLMs (e.g., 7B quantized), SD 1.5 at modest batch.
    • 16–24 GB = Llama 3 8B/70B (quantized), SDXL, LoRA finetuning on small models.
    • 48+ GB = larger context or partial training.

  • Bandwidth and lanes: PCIe x16 preferred (x8 works), avoid bifurcation bottlenecks.

  • Power and cooling: Size PSU with 30–40% headroom. Maintain case airflow; keep dust out.

  • System RAM + storage: 32–64 GB RAM is comfortable; NVMe SSD for model caches and datasets.

  • Vendor reality check:

    • NVIDIA: Widest framework support, mature tooling, many community images.
    • AMD: ROCm support has improved; great value if your card is supported.
    • Intel: Best for iGPU experiments; discrete support improving.

Tip: Stick to an LTS-like distro and avoid constantly chasing latest kernels if you want stable DKMS/driver builds.

2) Base OS prep: dev tools and essentials

Install core tools, Python, and build utilities. Use your package manager:

  • Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y build-essential git curl wget python3 python3-venv python3-pip cmake pciutils lsb-release tmux htop nvtop lm-sensors
  • Fedora/RHEL (dnf)
sudo dnf -y groupinstall "Development Tools"
sudo dnf -y install git curl wget python3 python3-virtualenv python3-pip cmake pciutils redhat-lsb-core tmux htop nvtop lm_sensors
  • openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y git curl wget python3 python3-pip cmake pciutils lsb-release tmux htop nvtop sensors

Quick checks:

lspci | grep -E "NVIDIA|AMD|Intel"
python3 -V
pip3 --version

Note: If you use secure boot, proprietary drivers may require signing or disabling secure boot.

3) GPU drivers and toolkits (pick your path)

Keep it boring and reliable: prefer distro-supported paths first.

  • NVIDIA

    • Ubuntu (recommended):
    sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi
    • Fedora (RPM Fusion):
    sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                    https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf -y install akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
nvidia-smi
    • openSUSE:
    • Add NVIDIA repo via YaST or: sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/$releasever/ nvidia sudo zypper refresh sudo zypper install -y nvidia-glG06 nvidia-computeG06 sudo reboot nvidia-smi Package names can vary by release; if in doubt, use YaST Software to select the proprietary driver.

    Optional CUDA Toolkit (compilers, nvcc):

    • Fedora (dnf):
    sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/fedora$(rpm -E %fedora)/x86_64/cuda-fedora$(rpm -E %fedora).repo
sudo dnf -y install cuda-toolkit
    • Ubuntu/openSUSE: Prefer NVIDIA’s official instructions for your exact release to install cuda-toolkit. If you don’t need to compile, you can skip the full toolkit.

  • AMD

    • Open-source amdgpu drivers ship with the kernel. For ROCm (GPU compute), use AMD’s amdgpu-install for your distro/version (varies). Start here: https://rocm.docs.amd.com
    • Quick verification (after ROCm):
    rocm-smi

  • Intel

    • For iGPU (Arc/Xe): use distro Mesa/oneAPI where appropriate. See: https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html

Pro tip: Don’t mix random PPAs/repos. Get the driver working first, then add toolkits.

4) Containers for GPU workloads (Docker and Podman)

Containers make AI stacks portable and clean.

Install Docker:

  • Debian/Ubuntu (apt)
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(. /etc/os-release; echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker
  • Fedora/RHEL (dnf)
sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
  • openSUSE (zypper)
sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

NVIDIA in containers (nvidia-container-toolkit):

  • Debian/Ubuntu (apt)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
  • Fedora/RHEL (dnf)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /etc/pki/rpm-gpg/RPM-GPG-KEY-nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
  • openSUSE (zypper)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /etc/pki/rpm-gpg/RPM-GPG-KEY-nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

AMD ROCm in containers:

  • Many images expect access to /dev/kfd and /dev/dri:
podman run --rm -it --device=/dev/kfd --device=/dev/dri --group-add keep-groups --security-opt=label=disable rocm/pytorch:latest rocm-smi
# or with Docker (rootful):
sudo docker run --rm -it --device=/dev/kfd --device=/dev/dri rocm/pytorch:latest rocm-smi

Podman (optional):

  • Debian/Ubuntu (apt): sudo apt update && sudo apt install -y podman

  • Fedora/RHEL (dnf): sudo dnf -y install podman

  • openSUSE (zypper): sudo zypper install -y podman

Tip: For NVIDIA + Podman, configure the OCI hook via nvidia-ctk and use the hooks directory. For most users, Docker is simpler.

5) Real workloads you can run today

A) One-command LLMs with Ollama (NVIDIA/AMD/CPU)

curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama run llama3
  • Switch models easily (mistral, phi3, neural-chat). Ollama detects CUDA/ROCm if present.

B) Llama.cpp from source (lean, fast, great for scripting)

  • NVIDIA:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)" LLAMA_CUBLAS=1
  • AMD (ROCm):
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)" LLAMA_HIPBLAS=1
  • Run a quantized GGUF model:
./main -m ./models/your-model.Q4_K_M.gguf -p "Write a Bash one-liner to list GPUs" -ngl 99

C) Stable Diffusion via Docker (NVIDIA example)

docker run --rm -it --gpus all -p 7860:7860 -v $PWD/sd:/data ghcr.io/automatic1111/stable-diffusion-webui:latest
# Open http://localhost:7860

AMD path: use a ROCm-enabled image or build a venv with PyTorch ROCm wheels.

D) PyTorch in a venv (choose your wheel)

  • Create venv:
python3 -m venv ~/venvs/ai && source ~/venvs/ai/bin/activate
pip install --upgrade pip
  • NVIDIA wheels (example CUDA 12.1):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
  • AMD ROCm wheels (pick your ROCm series, e.g., 6.0/6.1; see PyTorch site for exact index):
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.0
  • CPU fallback:
pip install torch torchvision
  • Quick test:
python - << 'PY'
import torch
print("CUDA:", torch.cuda.is_available(), "Device:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")
PY

6) Performance, monitoring, and reliability

Enable persistence and set power limits (NVIDIA):

sudo nvidia-smi -pm 1
nvidia-smi --query-gpu=name,clocks.sm,clocks.mem,power.draw,power.limit,temperature.gpu --format=csv
# Set a safe power limit (check allowed range first):
nvidia-smi -q -d POWER | grep -E "Limit|Draw"
sudo nvidia-smi -pl 250  # example

AMD quick stats:

rocm-smi
rocm-smi --showvoltage --showmeminfo vram

Install sensors and monitor:

  • Debian/Ubuntu (apt): sudo apt install -y lm-sensors nvtop && sudo sensors-detect

  • Fedora/RHEL (dnf): sudo dnf -y install lm_sensors nvtop && sudo sensors-detect

  • openSUSE (zypper): sudo zypper install -y sensors nvtop && sudo sensors-detect

Watch GPU live:

watch -n 1 nvidia-smi
# or for AMD:
watch -n 1 rocm-smi
# cross-vendor process/VRAM TUI:
nvtop

Pin stable drivers and avoid surprise upgrades:

  • Debian/Ubuntu:
sudo apt-mark hold nvidia-driver-* cuda-*
  • Fedora/openSUSE: consider excluding driver packages in dnf.conf/zypper locks if you need strict stability.

Back up model caches (they get big fast):

du -sh ~/.cache/* 2>/dev/null | sort -h

Example Bash: set a safe power profile at boot (NVIDIA)

sudo tee /usr/local/sbin/gpu-power.sh >/dev/null << 'SH'
#!/usr/bin/env bash
set -euo pipefail
# Enable persistence and set power limit if supported
if command -v nvidia-smi >/dev/null; then
  sudo nvidia-smi -pm 1 || true
  # Set to 90% of max if possible
  max=$(nvidia-smi -q -d POWER | awk '/Max Power Limit/ {print int($5)}' | head -1)
  if [[ -n "${max:-}" && "$max" -gt 0 ]]; then
    target=$(( (max * 90) / 100 ))
    sudo nvidia-smi -pl "$target" || true
  fi
fi
SH
sudo chmod +x /usr/local/sbin/gpu-power.sh

Create a systemd service that runs it at boot if desired.

Troubleshooting quick hits

  • Driver DKMS failing after kernel update? Reboot once more or reinstall the driver; keep headers installed.

  • Wayland + headless tweaks: for some legacy tools (nvidia-settings), use X11 or CLI-only nvidia-smi/rocm-smi.

  • Container can’t see GPU?

    • NVIDIA: confirm nvidia-smi works on host, then docker run --gpus all ....
    • AMD: pass --device=/dev/kfd --device=/dev/dri.

  • Running out of VRAM? Quantize models (GGUF), reduce batch/steps, or offload some layers to CPU if the framework allows.

Conclusion and next steps (CTA)

You now have the building blocks: a stable Linux base, drivers, containers, and working AI examples. Next:

  • Pick one workload to own (e.g., Ollama with Llama 3 for chat, or SDXL for images).

  • Capture your setup as Bash scripts and a README for repeatability.

  • Add observability (nvtop, rocm-smi, Prometheus/Grafana) and backups for models/datasets.

  • Iterate: tune power/thermals and try different containers/framework builds.

If this helped, script your own “homelab bootstrap” and share it. Your future self (and the community) will thank you.

Related posts