Posted on
• Artificial Intelligence

Ollama Performance Tuning

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Ollama Performance Tuning on Linux: Faster Tokens, Lower Latency

If you’ve ever stared at a terminal waiting for a local LLM to finish a sentence, you know the pain: great privacy and control, but sometimes sluggish performance. The good news? With a few Linux-friendly tweaks, Ollama can feel dramatically faster—often with nothing more than a couple of environment variables, a smarter model choice, and the right system settings.

This guide explains why tuning matters, shows you how to set up Ollama correctly on Linux, and walks through actionable steps that measurably improve throughput and latency.

Why tune Ollama?

  • Local LLMs are sensitive to I/O, memory bandwidth, and GPU/CPU threading. Small configuration changes can yield big wins.

  • Cold starts (loading models into memory) are expensive—especially on spinning disks or networked storage.

  • Model size and quantization directly change RAM/VRAM needs and tokens/sec.

  • Server-level concurrency and caching choices determine whether your terminal or API responds instantly or stalls.

Quick install and prerequisites

You’ll need curl and a few helper tools to follow along (jq, numactl, bc). Install them with your distro’s package manager:

  • Debian/Ubuntu (apt)

    sudo apt update
sudo apt install -y curl jq numactl bc

  • Fedora/RHEL/CentOS (dnf)

    sudo dnf install -y curl jq numactl bc

  • openSUSE (zypper)

    sudo zypper refresh
sudo zypper install -y curl jq numactl bc

Install Ollama (official script):

curl -fsSL https://ollama.com/install.sh | sh

Start/enable the service:

sudo systemctl enable --now ollama
sudo systemctl status ollama

Basic smoke test:

ollama run llama3 "Say hello from Linux."

GPU prerequisites (optional but recommended):

  • NVIDIA: Ensure the proprietary driver is installed (use your distro’s recommended method).

    • Ubuntu:
    sudo ubuntu-drivers autoinstall
sudo reboot
    • Fedora (RPM Fusion):
    sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                     https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf install -y akmod-nvidia
sudo reboot
    • openSUSE: Use NVIDIA’s official repository for your version, then:
    sudo zypper install -y nvidia-glG06
sudo reboot

  • AMD: Install ROCm per your distro/driver stack. Ensure rocm-smi works and HIP runtime is present.

Tip: After reboot, verify:

nvidia-smi  # NVIDIA
rocm-smi    # AMD

1) Put models on fast storage and keep them hot

Model load time dominates first-token latency. Moving models to NVMe and keeping active models resident cuts delays dramatically.

  • Choose a fast path and set OLLAMA_MODELS:

    sudo mkdir -p /mnt/nvme/ollama-models
sudo chown -R $USER:$USER /mnt/nvme/ollama-models

  • Create a systemd override to tune server settings:

    sudo systemctl edit ollama

    Paste:

    [Service]
Environment="OLLAMA_MODELS=/mnt/nvme/ollama-models"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_MAX_QUEUE=512"

    Apply:

    sudo systemctl daemon-reload
sudo systemctl restart ollama

Notes:

  • OLLAMA_KEEP_ALIVE tells Ollama not to immediately unload models. Increase cautiously if RAM/VRAM is limited.

  • If you already have models, move your old models directory (often ~/.ollama/models or /var/lib/ollama) into /mnt/nvme/ollama-models before restarting.

2) Pick the right quantization for your hardware

Quantization controls memory footprint and speed:

  • q4_K_M and q5_1: excellent balance for consumer GPUs/CPUs.

  • q8_0: higher quality, needs more RAM/VRAM.

  • f16: maximum quality, slowest and heaviest.

Examples:

# Pull a lighter quantized variant
ollama pull llama3:8b-q4_K_M

# Run it
ollama run llama3:8b-q4_K_M "Summarize the Linux memory model in 3 bullets."

Rule of thumb:

  • CPU-only systems: q4_K_M or q5_1 for speed.

  • 8–12 GB VRAM GPUs: 7–8B models at q4_* or q5_* feel snappy.

  • Larger VRAM (24+ GB): consider q8_0 or even fp16 for niche quality needs.

3) Offload to GPU and size GPU layers

If you have a supported GPU, offloading transformer layers provides a big speedup. The idea: push as many layers as your VRAM allows without swapping.

Create a tuned model variant via a Modelfile:

# Start from an existing base
ollama pull llama3:8b-q4_K_M

# Export its Modelfile, edit, and re-create
ollama show llama3:8b-q4_K_M --modelfile > Modelfile

# Open Modelfile in your editor and add or adjust parameters:
# Example parameters (tweak to your hardware)
# PARAMETER num_ctx 4096
# PARAMETER num_batch 512
# PARAMETER num_gpu_layers 35

Then build a custom variant:

ollama create llama3:8b-q4_K_M-gpu -f Modelfile

Test and watch VRAM in real time:

watch -n1 nvidia-smi    # NVIDIA
watch -n1 rocm-smi      # AMD

Tips:

  • Increase num_gpu_layers until VRAM usage is just below the safe limit for your workload.

  • num_batch speeds up prompt ingestion; going too high can increase memory pressure. Start at 256–512 and iterate.

  • num_ctx increases memory use; use only what you need (e.g., 4096 for chat, higher only if your prompts require it).

4) Concurrency, caching, and API behavior

If you’re serving multiple shells or apps, tune server concurrency to avoid thrashing.

  • OLLAMA_MAX_LOADED_MODELS controls how many models can stay resident. Set to 1 on tight RAM; 2–3 on roomy systems.

  • OLLAMA_MAX_QUEUE caps incoming requests waiting to start. Keep it reasonable to avoid timeouts.

  • Warm your most-used model after reboot:

    # Warm-up script (put in your login profile or a systemd timer)
ollama run llama3:8b-q4_K_M "warmup" >/dev/null 2>&1

Optional: bind the server for LAN access (secure your network before doing this):

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then:

sudo systemctl daemon-reload && sudo systemctl restart ollama

5) Measure what matters (tokens/sec and latency)

Always validate changes with a repeatable test. Here’s a simple benchmark that calls the HTTP API, disables streaming for a single JSON response, and computes tokens/sec.

Install helpers if you haven’t:

  • apt:

    sudo apt install -y jq bc

  • dnf:

    sudo dnf install -y jq bc

  • zypper:

    sudo zypper install -y jq bc

Benchmark script:

#!/usr/bin/env bash
MODEL="${1:-llama3:8b-q4_K_M}"
PROMPT="${2:-Explain virtual memory in Linux in one paragraph.}"

OUT=$(curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"${MODEL}\", \"prompt\": \"${PROMPT}\", \"stream\": false}")

TOKENS=$(echo "$OUT" | jq -r '.eval_count // 0')
DUR_NS=$(echo "$OUT" | jq -r '.eval_duration // 0')
if [ "$TOKENS" -gt 0 ] && [ "$DUR_NS" -gt 0 ]; then
  TPERSEC=$(echo "scale=2; $TOKENS / ($DUR_NS/1000000000)" | bc -l)
  echo "Model: $MODEL"
  echo "Eval tokens: $TOKENS"
  echo "Eval duration (s):" $(echo "scale=3; $DUR_NS/1000000000" | bc -l)
  echo "Tokens/sec: $TPERSEC"
else
  echo "Benchmark failed or no tokens produced."
  echo "$OUT"
fi

Usage:

chmod +x bench.sh
./bench.sh llama3:8b-q4_K_M

Run this before and after each change to see real gains.

Real-world example workflow

  • Start with a balanced quantization (q4_K_M).

  • Place models on NVMe and set OLLAMA_KEEP_ALIVE=24h.

  • Tune num_gpu_layers to nearly fill your VRAM without OOMs.

  • Set num_batch to 256–512 for faster prompt processing.

  • Keep OLLAMA_MAX_LOADED_MODELS low unless you truly need concurrent distinct models.

Expect noticeably faster first-token times (due to hot models + NVMe) and higher tokens/sec (due to GPU offload + sane batching).

Conclusion and next steps

Local LLMs don’t have to feel slow. By choosing the right quantization, keeping models hot on fast storage, sizing GPU offload, and controlling concurrency, you can turn Ollama into a responsive, reliable workhorse—right from your Bash shell.

Your next steps:

  • Implement step 1 (fast storage + keep-alive) and re-benchmark.

  • Experiment with num_gpu_layers and num_batch on your GPU.

  • Lock in a quantization that fits your RAM/VRAM with headroom.

  • Wrap your favorite model in a tuned Modelfile and re-use it everywhere.

Questions or want a deeper dive on your specific hardware? Gather your benchmark numbers (tokens/sec, VRAM, CPU/RAM usage) and iterate—small changes add up fast.

Related posts