rocm

Hands-on guide to enable fast, private local LLMs with Ollama on Linux: install via apt/dnf/zypper, confirm NVIDIA or AMD/ROCm drivers (nvidia-smi/rocminfo), pull and run models, watch GPU activity, tune gpu_layers and num_ctx to fit VRAM, optionally run in Docker (NVIDIA --gpus all, AMD /dev/kfd,/dev/dri), and apply env vars and troubleshooting to avoid OOM, CPU fallback, and container GPU access issues.

ROCm turns supported AMD GPUs into fast, cost‑effective AI workhorses on Linux, with an open stack, CUDA‑like HIP, and optimized libs (rocBLAS, hipBLASLt, MIOpen, RCCL) plus official PyTorch wheels. The guide covers prereqs, apt/dnf/zypper installs, env/permissions, validation (rocminfo, rocm‑smi, HIP samples), running PyTorch and llama.cpp or containers, and tips for support, tuning, and multi‑GPU.

Guide to choosing and installing NVIDIA CUDA or AMD ROCm for AI on Linux: NVIDIA wins for widest framework support, mature multi‑GPU and tooling; AMD offers strong price/VRAM and open stack if your workload is supported. Covers hardware/VRAM needs, drivers vs toolkits, using wheels/containers, sanity checks, distro‑specific installs, ROCm nuances, and quickstarts—then benchmark your real model.

A concise, distro-specific guide to set up GPU-accelerated AI on Linux: identify hardware; install and match drivers/toolkits (NVIDIA CUDA, AMD ROCm) via apt/dnf/zypper; verify with nvidia-smi/rocminfo; create Python venvs and install PyTorch/TensorFlow; run a test inference; enable Docker/Podman GPU containers; and avoid pitfalls like Secure Boot, nouveau conflicts, kernel updates, and mismatched versions for a stable, reproducible stack.