Posted on
• Artificial Intelligence

Building a Linux Artificial Intelligence Workstation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Building a Linux Artificial Intelligence Workstation (The Practical, Bash-First Guide)

You’ve got a powerful Linux box (or you’re about to build one) and you want it to crush model training, fine‑tuning, and inference. But between drivers, CUDA/ROCm stacks, Python envs, and container runtimes, it’s easy to waste hours on yak‑shaving. This guide is your shortcut: the why and the how, with copy‑pasteable commands for apt, dnf, and zypper.

What you’ll get:

  • A clear plan to make your Linux workstation AI‑ready

  • Minimal, distro‑specific install commands

  • Real‑world examples that validate your setup

Why “AI‑Ready” Matters

  • Performance: Correct GPU drivers and math libraries can be the difference between 1× and 10× throughput.

  • Reproducibility: Using virtual environments and containers prevents “works on my machine” bugs.

  • Longevity: A clean, modular stack lets you swap GPUs, update frameworks, and scale up without starting over.

Think in pillars:

  • Compute and memory (GPU VRAM, CPU cores, RAM, NVMe)

  • Software stack (drivers + CUDA/ROCm + BLAS + compilers)

  • Environments (Python venv/Conda, containers)

  • Workloads (PyTorch/TensorFlow/JAX, inference engines)

Step 1 — Choose Compatible, Balanced Hardware

  • GPU:

    • NVIDIA: Broadest framework support via CUDA; great for LLMs and diffusers. Aim for 12–24 GB VRAM minimum; 48–80 GB for big models or multi‑GPU.
    • AMD: ROCm 6.x has grown fast; check card support first. Great value if your workloads and frameworks (e.g., PyTorch ROCm) align.

  • CPU: 8–16 cores (or more) with AVX2/AVX‑512 helps CPU inference and data preprocessing.

  • RAM: 32–64 GB is a practical floor; match or exceed total GPU VRAM if you batch large datasets.

  • Storage: 1–2 TB NVMe (Gen4) for datasets and checkpoints; leave headroom for Docker images.

  • PSU/Cooling: Stable power and airflow = fewer mysterious crashes under load.

Pro tip: If you’ll push VRAM limits, add swap or zram so processes fail slower (or recover).

Step 2 — Base OS Setup and Dev Tools

Install compilers, Python, and common build deps.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y build-essential python3 python3-venv python3-pip git curl wget cmake ninja-build pkg-config \
                   libopenblas-dev libssl-dev libffi-dev

Fedora/RHEL/CentOS (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y python3 python3-venv python3-pip git curl wget cmake ninja-build pkgconfig \
                   openblas-devel openssl-devel libffi-devel

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh
sudo zypper install -y -t pattern devel_C_C++
sudo zypper install -y python3 python3-pip python3-venv git curl wget cmake ninja pkg-config \
                      openblas-devel libopenssl-devel libffi-devel gcc gcc-c++ make

Optional (recommended) swapfile for memory headroom:

sudo fallocate -l 64G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapon -a

Step 3 — Install a GPU Stack (NVIDIA CUDA or AMD ROCm)

Pick exactly one path below and follow it carefully. Reboot after driver installation.

Path A: NVIDIA (CUDA)

Quick check: after install, nvidia-smi should show your GPU.

Debian/Ubuntu (apt) — distro packages (simple, not always latest):

sudo apt update
# Ubuntu users can auto-select a recommended driver:
# sudo ubuntu-drivers autoinstall
sudo apt install -y nvidia-driver nvidia-cuda-toolkit
sudo reboot

Fedora (dnf) — via RPM Fusion:

sudo dnf install -y \
  https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot

openSUSE (zypper) — CUDA repo (installs driver + toolkit):

sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
sudo zypper refresh
sudo zypper install -y cuda
sudo reboot

Verify:

nvidia-smi
nvcc --version

Path B: AMD (ROCm)

Important: Check GPU support in the ROCm release notes for your distro/GPU model.

Ubuntu (apt) — example for ROCm 6.x:

echo "deb [arch=amd64 signed-by=/usr/share/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.1 jammy main" | \
  sudo tee /etc/apt/sources.list.d/rocm.list
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/rocm.gpg
sudo apt update
sudo apt install -y rocm
sudo usermod -a -G render,video $USER
sudo reboot

RHEL9/CentOS Stream 9/Fedora‑like (dnf):

sudo dnf config-manager --add-repo https://repo.radeon.com/rocm/rhel9/6.1/main
sudo rpm --import https://repo.radeon.com/rocm/rocm.gpg.key
sudo dnf install -y rocm
sudo usermod -a -G render,video $USER
sudo reboot

SLES/openSUSE (zypper) — SLES example:

sudo zypper ar -f https://repo.radeon.com/rocm/sles/6.1/main rocm
sudo rpm --import https://repo.radeon.com/rocm/rocm.gpg.key
sudo zypper install -y rocm
sudo usermod -a -G render,video $USER
sudo reboot

Verify:

rocminfo | head -n 50
hipcc --version

Step 4 — Python Environments and AI Frameworks

Use a per‑project venv so global packages don’t collide.

Create a venv:

python3 -m venv ~/venvs/ai
source ~/venvs/ai/bin/activate
python -m pip install -U pip wheel setuptools

PyTorch (choose one; CPU, NVIDIA CUDA, or AMD ROCm):

  • CPU‑only:
pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision
  • NVIDIA CUDA 12.1:
pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision
python - <<'PY'
import torch; print("CUDA available:", torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "")
PY
  • AMD ROCm 6.0/6.1 (check your ROCm minor version):
pip install --index-url https://download.pytorch.org/whl/rocm6.0 torch torchvision
# or replace rocm6.0 with rocm6.1 depending on your stack
python - <<'PY'
import torch; print("HIP/ROCm available:", torch.version.hip is not None, torch.cuda.is_available())
PY

TensorFlow (CPU or CUDA‑enabled wheels):

pip install "tensorflow[and-cuda]==2.16.*"
python - <<'PY'
import tensorflow as tf; print(tf.config.list_physical_devices())
PY

JAX (CUDA 12 builds):

pip install -U "jax[cuda12]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python - <<'PY'
import jax; import jax.numpy as jnp
x = jnp.ones((1024, 1024)); print("Device:", jax.default_backend()); print(x.device())
PY

BLAS sanity check (useful even on CPU):

python - <<'PY'
import numpy as np
a = np.random.randn(4096, 4096).astype(np.float32)
b = a @ a.T
print(b.shape, b.dtype)
PY

Step 5 — Containers for Reproducible AI (Docker or Podman)

Docker

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y docker.io
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker
docker run --rm hello-world

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y docker
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker
docker run --rm hello-world

openSUSE (zypper):

sudo zypper install -y docker
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker
docker run --rm hello-world

NVIDIA GPU into containers (NVIDIA Container Toolkit):

Debian/Ubuntu (apt):

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu22.04 nvidia-smi

Fedora/RHEL/CentOS (dnf):

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /etc/pki/rpm-gpg/NVIDIA-Container-Toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.3.1-base-ubi9 nvidia-smi

openSUSE (zypper):

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
sudo rpm --import https://nvidia.github.io/libnvidia-container/gpgkey
sudo curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
  -o /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.3.1-base-opensuse15 nvidia-smi

Podman (alternative to Docker):

apt:

sudo apt install -y podman

dnf:

sudo dnf install -y podman

zypper:

sudo zypper install -y podman

Note: GPU passthrough with Podman may require additional hooks/config; Docker + NVIDIA toolkit is simpler for most users.

Real‑World Validation: Two Quick Wins

1) Fast local LLM inference with llama.cpp

  • Build with your acceleration and run a prompt.

NVIDIA:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)" LLAMA_CUBLAS=1
./main -ngl 20 -m ./models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -p "Explain Bash pipes simply."

AMD ROCm:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)" LLAMA_HIPBLAS=1
./main -ngl 20 -m ./models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -p "Explain Bash pipes simply."

CPU:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)"
./main -m ./models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -p "Explain Bash pipes simply."

2) Serve an LLM via vLLM in a container (NVIDIA GPUs)

docker run --rm --gpus all -p 8000:8000 --shm-size 16g \
  -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
  vllm/vllm-openai:latest --model meta-llama/Llama-3-8B-Instruct
# Test:
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3-8B-Instruct","messages":[{"role":"user","content":"Hello from Linux!"}]}'

Troubleshooting Cheatsheet

  • Driver not loaded:

    • NVIDIA: nvidia-smi should work; check lsmod | grep nvidia, dmesg | grep -i nvidia
    • AMD: rocminfo should list agents; check groups includes video and render

  • Wrong toolkit version:

    • Align CUDA/ROCm with your wheels (PyTorch/TensorFlow/JAX). Use their official wheels/index URLs.

  • Out of memory:

    • Reduce batch size, enable swap/zram, use quantized models, or offload layers.

Conclusion (Your Next Step)

You now have a blueprint to turn Linux into an AI powerhouse: 1) Balance hardware for your workloads 2) Install dev tools 3) Choose and install a GPU stack (CUDA or ROCm) 4) Build isolated Python envs and install frameworks 5) Use containers for reproducibility 6) Validate with a real model run

Call to action: Pick your GPU path (CUDA or ROCm), run the install commands above, and verify with the quick validation examples. If you want a tailored, distro‑specific script that automates everything for your exact GPU and framework, tell me your GPU model and Linux version—I’ll generate it.

Related posts