Posted on
• Artificial Intelligence

Running Ollama with GPU Acceleration

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Running Ollama with GPU Acceleration on Linux

Want local LLMs that feel instant? The secret is GPU acceleration. If you’ve tried running large models on CPU-only and watched tokens trickle out, switching to GPU can deliver a dramatic, often order-of-magnitude speedup—without sending your data to the cloud. In this guide, you’ll set up Ollama on Linux with NVIDIA or AMD GPUs, verify that acceleration is working, and tune performance for your hardware.

Why this matters

  • Privacy and control: Keep your prompts and data local.

  • Latency and cost: GPUs deliver fast, predictable performance without per-token bills.

  • Simplicity: Ollama wraps model download, serving, and execution in a clean CLI and a simple local API.

  • Portability: Works on common Linux distros and supports NVIDIA (CUDA/cuBLAS) and AMD (ROCm/HIP).

What you’ll need

  • A Linux distro with one of:

    • NVIDIA GPU + proprietary driver installed (CUDA toolkit not required).
    • AMD GPU + ROCm userspace/runtime installed (on supported distros/kernels).

  • Shell access with sudo.

  • A few GB of disk for models and enough VRAM for the model you choose.

Tip: If you’re not sure your GPU stack is working yet, don’t worry—below you’ll verify it in one command.

Install Ollama (apt, dnf, zypper)

The official one-liner sets up the correct repository on your distro and installs Ollama via your package manager:

curl -fsSL https://ollama.com/install.sh | sh

After running the one-liner, you can manage Ollama with your native package manager:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install ollama

  • Fedora/RHEL/CentOS (dnf):

    sudo dnf install ollama

  • openSUSE (zypper):

    sudo zypper refresh
sudo zypper install ollama

Start and check the service:

sudo systemctl enable --now ollama
systemctl status ollama

By default, Ollama listens on 127.0.0.1:11434 and will automatically use your GPU if supported drivers/runtimes are available.

Step 1: Verify your GPU drivers

  • NVIDIA:

    nvidia-smi

    You should see a table of GPU processes and driver version. If the command is missing or fails, install the proprietary driver from your distro’s docs/tools and reboot.

  • AMD (ROCm):

    rocminfo | head -n 40

    or (depending on your install path):

    /opt/rocm/bin/rocminfo | head -n 40

    You should see your GPU(s) listed. If not, install ROCm appropriate for your distro/kernel per AMD’s docs and reboot.

Optional tools:

  • OpenCL info:
    • apt: sudo apt install clinfo && clinfo | head -n 20
    • dnf: sudo dnf install clinfo && clinfo | head -n 20
    • zypper: sudo zypper install clinfo && clinfo | head -n 20

Step 2: Pull a model and run it on the GPU

Try a lightweight but capable model to confirm acceleration:

ollama pull llama3:8b
ollama run llama3:8b

While it’s generating text, watch GPU usage in another terminal:

  • NVIDIA:

    watch -n1 nvidia-smi

  • AMD:

    watch -n1 /opt/rocm/bin/rocm-smi

    or:

    watch -n1 cat /sys/class/drm/card*/device/pp_dpm_sclk

If the GPU column shows activity and memory usage climbs while the model is responding, you’re accelerated. If not, see “Troubleshooting” below.

Pro tip: To force a specific GPU index on multi-GPU systems, use:

CUDA_VISIBLE_DEVICES=0 ollama run llama3:8b

Step 3: Tune GPU offload for your VRAM

Ollama can offload some or most of a model’s layers to the GPU. More layers offloaded = more VRAM used and higher throughput. You can tune this with a simple Modelfile.

Example: Create a partially offloaded variant for an 8B model:

cat > Modelfile <<'EOF'
FROM llama3:8b
# Increase context for longer prompts if desired:
PARAMETER num_ctx 4096
# Offload layers to the GPU; raise until stable without OOM:
PARAMETER gpu_layers 35
EOF

ollama create llama3-8b-gpu -f Modelfile
ollama run llama3-8b-gpu

Tips:

  • Start with a conservative gpu_layers (e.g., 20–35 for 8B) and increase gradually until you either:

    • See errors about out-of-memory, or
    • Notice swapping/fallback to CPU. Then back off slightly.

  • Larger models need more VRAM. If VRAM is limited, reduce gpu_layers or choose a smaller model (e.g., 7B variants).

Step 4: Run via Docker with GPU (optional)

Containerizing Ollama is handy for servers and reproducible setups.

  • NVIDIA: 1) Install the NVIDIA Container Toolkit (per NVIDIA’s docs). 2) Run: docker run --gpus all -d --name ollama \ -p 11434:11434 \ -v ollama:/root/.ollama \ ollama/ollama 3) Then interact: docker exec -it ollama ollama pull llama3:8b docker exec -it ollama ollama run llama3:8b

  • AMD (ROCm): Ensure the host has ROCm installed and the container has access to GPU devices:

    docker run -d --name ollama \
--device=/dev/kfd --device=/dev/dri \
--group-add video \
--security-opt seccomp=unconfined \
-p 11434:11434 \
-v ollama:/root/.ollama \
ollama/ollama

    Then:

    docker exec -it ollama ollama pull llama3:8b
docker exec -it ollama ollama run llama3:8b

Note: ROCm containerization details can vary by kernel/distro/GPU generation. If the container can’t see the GPU, first confirm ROCm is working on the host.

Step 5: Practical tips for smooth GPU runs

  • Keep the daemon lean on VRAM:

    • Limit concurrent models:
    export OLLAMA_MAX_LOADED_MODELS=1

  • Force the server to foreground for debug messages:

    OLLAMA_DEBUG=1 ollama serve

  • Constrain CPU threads if needed:

    OLLAMA_NUM_THREADS=8 ollama run llama3:8b

  • Pin to specific GPUs:

    CUDA_VISIBLE_DEVICES=1 ollama run llama3:8b

Troubleshooting

  • Ollama uses CPU, not GPU:

    • Confirm drivers/runtime:
    • NVIDIA: nvidia-smi
    • AMD: rocminfo
    • Reboot after installing drivers/ROCm.
    • Check service logs:
    journalctl -u ollama -e

  • Out-of-memory or crash when starting:

    • Lower gpu_layers in your Modelfile.
    • Reduce num_ctx or use a smaller model.
    • Ensure no other heavy GPU workloads are active.

  • Docker can’t access GPU:

    • NVIDIA: Confirm --gpus all works with a test container (e.g., nvidia/cuda:12.3.0-base + nvidia-smi).
    • AMD: Verify /dev/kfd and /dev/dri are present on host and user is in the video group:
    sudo usermod -aG video $USER
newgrp video

Real-world example flow

  • Install Ollama:

    curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

  • Verify GPU:

    nvidia-smi

  • Run a model and watch the GPU:

    ollama pull llama3:8b
watch -n1 nvidia-smi &
ollama run llama3:8b

Most users see significant speedups when the GPU is active, especially on 8B–13B models.

Conclusion and next steps

You now have a GPU-accelerated local LLM stack running on Linux with Ollama. Next:

  • Explore different models: ollama list and ollama pull <model>.

  • Tune gpu_layers and num_ctx for your VRAM and workload.

  • Script prompts or build tools against the local API at http://127.0.0.1:11434.

  • Containerize for servers and share a simple one-liner with your team.

If this saved you time, benchmark your tokens/sec before and after GPU acceleration and share your results. Happy hacking!

Related posts