Posted on
• Artificial Intelligence

Optimising Ollama Performance

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Optimising Ollama Performance on Linux: Practical Tuning for Faster Local LLMs

Local LLMs feel magical—until they don’t. If your first Ollama run felt sluggish, or GPU utilization sits idle while your CPU screams, you’re leaving performance on the table. The good news: a few targeted tweaks can unlock big gains in tokens-per-second, latency, and overall responsiveness.

This guide explains why performance tuning matters for Ollama, then gives you concrete, bash-friendly steps to squeeze more from your hardware—whether you’re on CPU-only or a CUDA/ROCm-capable GPU box.

Why tuning Ollama is worth it

  • Local models are bandwidth-bound. VRAM/RAM bandwidth, cache locality, and fast storage often matter more than raw FLOPS.

  • Quantization choices trade quality for speed and memory—pick the right one and you can double throughput while cutting VRAM use in half.

  • GPU offloading, threading, and context sizing directly impact latency and tokens/sec.

  • Keeping models warm and placing them on NVMe eliminates repeated cold-start overhead.

Quick setup and smoke test

These steps ensure you have the basics in place and can run your first prompt.

1) Install prerequisites (curl/jq)

  • Debian/Ubuntu (apt):
sudo apt update && sudo apt install -y curl jq
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl jq
  • openSUSE (zypper):
sudo zypper ref && sudo zypper in -y curl jq

2) Install Ollama (official script)

curl -fsSL https://ollama.com/install.sh | sh

3) Start the server (systemd) and test a model

sudo systemctl enable --now ollama
ollama run llama3:instruct -p "Say hello in one sentence."

If you see a response, you’re good. The first run will download and load the model; subsequent runs should be faster.

Actionable optimisations (with real commands)

1) Choose the right model and quantization for your VRAM/RAM

  • Smaller models or more aggressive quantization (e.g., Q4_K_M) are dramatically faster and lighter than higher-precision variants.

  • Rough VRAM guidance (very approximate, varies by model family and context):

    • 7B Q4 ≈ 4–6 GB VRAM
    • 7B Q5 ≈ 6–8 GB VRAM
    • 13B Q4 ≈ 8–12 GB VRAM

Check what you’ve got:

ollama show llama3:instruct | grep -i -E 'parameter|quant|size'

Pull a model (Ollama picks a sensible default quant by tag):

ollama pull llama3:instruct

Tip: If a model struggles to fit or is slow, try a smaller family (e.g., phi3, mistral, qwen) or a more aggressive quantized variant.

2) Enable and verify GPU acceleration (CUDA/ROCm)

Ollama will use your GPU automatically when compatible drivers/toolchains are installed. Verify with:

  • NVIDIA:
nvidia-smi
  • AMD ROCm:
rocminfo  # if ROCm is installed

Confirm Ollama sees the GPU:

OLLAMA_DEBUG=1 ollama run llama3:instruct -p "hi" 2>&1 | grep -i -E 'cuda|rocm|gpu'

If GPU is detected, you’ll see logs indicating CUDA/ROCm use. If not, install appropriate drivers/toolchains per your distro and GPU vendor guides.

You can also experiment with how many transformer layers to offload to the GPU for speed (more layers → more VRAM):

ollama run -o num_gpu=999 llama3:instruct -p "benchmark me"

Note: Using a high value like 999 asks Ollama to offload as many layers as possible within VRAM. If you hit OOM, reduce it (e.g., -o num_gpu=20).

3) Tune runtime options for throughput and latency

  • Set CPU threads to match your machine (helps for CPU-only or mixed CPU/GPU pipelines):
ollama run -o num_thread=$(nproc) llama3:instruct -p "Summarize this in one line."
  • Adjust context window thoughtfully. Larger num_ctx improves long prompts but increases memory and can reduce speed:
ollama run -o num_ctx=4096 llama3:instruct -p "..."
  • Keep models hot to avoid cold starts between runs:
    • Per-call:
ollama run --keepalive 1h llama3:instruct -p "Hello again"
  • Default for the server (systemd override below) with an environment variable:
sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_KEEP_ALIVE=1h"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

4) Put models on fast NVMe and control the model cache location

Model (GGUF) loads and swaps are I/O sensitive. Place Ollama’s model directory on NVMe.

  • Create a fast directory:
sudo mkdir -p /mnt/nvme/ollama
sudo chown $USER:$USER /mnt/nvme/ollama
  • Point Ollama to it via systemd:
sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_MODELS=/mnt/nvme/ollama"

Then apply:

sudo systemctl daemon-reload
sudo systemctl restart ollama
  • Move or re-pull models:
    • If you already have models, copy them to the new location (the source path depends on how you ran Ollama earlier):
# If models live under your home
rsync -av ~/.ollama/ /mnt/nvme/ollama/

# If models live under a shared path (service-managed)
sudo rsync -av /usr/share/ollama/ /mnt/nvme/ollama/

Otherwise, just ollama pull again and the new models will land on NVMe.

5) Benchmark and monitor like a pro

Measure first, tweak, then re-measure. The non-streaming HTTP API returns timing stats:

curl -s -X POST http://localhost:11434/api/generate \
  -d '{
    "model": "llama3:instruct",
    "prompt": "Write a haiku about penguins.",
    "stream": false
  }' \
| jq '.eval_count as $c | .eval_duration as $d | {tokens:$c, seconds:($d/1e9), tps:($c/($d/1e9))}'

Install helpful monitors:

  • Debian/Ubuntu (apt):
sudo apt update && sudo apt install -y htop nvtop
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y htop nvtop
  • openSUSE (zypper):
sudo zypper ref && sudo zypper in -y htop nvtop

Then:

  • htop to watch CPU threads.

  • nvtop to watch NVIDIA GPU load and VRAM usage.

  • For AMD with ROCm, use vendor tools (e.g., rocm-smi) if available.

Real-world quick recipes

  • CPU-only laptop (8 logical cores):
ollama run -o num_thread=$(nproc) -o num_ctx=2048 llama3:instruct -p "TL;DR this text: ..."

If it stutters, try a smaller model (e.g., phi3:mini), or reduce num_ctx.

  • Midrange NVIDIA GPU (8 GB VRAM):
# Verify GPU
nvidia-smi

# Pull and run a compact 7–8B instruct model
ollama pull llama3:instruct
OLLAMA_DEBUG=1 ollama run -o num_gpu=999 llama3:instruct -p "Plan a 3-day trip to Tokyo."

If you see OOM in the logs, set -o num_gpu to a lower value or pick a smaller/stronger-quantized model.

Common pitfalls to avoid

  • Oversized context window leading to OOM or slowdowns. Start with 2–4K, grow only as needed.

  • Storing models on a slow HDD—dramatically slower cold starts and swaps.

  • Forgetting to keep models warm, causing repeated loading between short runs.

  • Chasing bigger models without measuring; a well-quantized 7–8B often beats a starved 13B on consumer hardware.

Your next step (CTA)

  • Benchmark your current setup with the curl/jq snippet above.

  • Apply one change at a time: move to NVMe, enable GPU offload, tune num_thread, then num_ctx.

  • Re-run the benchmark and record tokens/sec. Iterate until you hit diminishing returns.

If you found a tuning combo that screams on your hardware, share your model, quant, and tps numbers with the community. Happy hacking!

Related posts