Running Quantised Models on Linux: Faster AI from Your Terminal

What if you could run modern AI models on a modest laptop or a small home server—no 24 GB GPU required? Quantisation makes that possible. With a few Bash commands, you can shrink model memory usage by 2–4x and unlock near-real-time inference on CPUs and low-VRAM GPUs.

In this article, you’ll learn what quantisation is, why it matters, and how to run quantised Large Language Models (LLMs), speech-to-text, and ONNX models on Linux. You’ll get actionable, copy-pasteable commands using apt, dnf, and zypper, and finish with performance tips to get the most from your hardware.

Why quantisation is worth your time

• Quantisation reduces the precision of model weights (for example, from 16-bit floats to 8-bit or 4-bit integers) while preserving most of the model’s accuracy.

• The results:

  • Lower RAM/VRAM usage (run 7B-13B LLMs on typical desktops)
  • Faster inference on CPUs due to better cache/memory bandwidth usage
  • Lower cost and power consumption for edge deployments

Common formats and runtimes you’ll see:

• GGUF (successor to GGML) for LLMs via llama.cpp

• GPTQ/AWQ/NF4 for GPU-first quantisation

• INT8/INT4 quantisation for ONNX Runtime and CPU inference

• Pre-quantised Whisper models or on-the-fly quantisation via whisper.cpp

Below are three practical ways to run quantised models from your Linux shell.

---

1) Prepare your machine (build tools, Python, git-lfs)

Install the essentials (choose the command set for your distro):

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y build-essential cmake git git-lfs pkg-config libopenblas-dev \
  python3 python3-venv python3-pip ffmpeg wget curl
git lfs install

• Fedora/RHEL (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git git-lfs openblas-devel \
  python3 python3-virtualenv python3-pip ffmpeg wget curl
git lfs install

• openSUSE (zypper):

sudo zypper install -y -t pattern devel_basis
sudo zypper install -y cmake git git-lfs openblas-devel \
  python3 python3-venv python3-pip ffmpeg wget curl
git lfs install

Notes:

• ffmpeg is used later for audio (whisper.cpp). If ffmpeg isn’t in your distro’s default repo, enable the relevant multimedia repo (e.g., RPM Fusion on Fedora).

• If you prefer isolation, create a Python venv:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

---

2) Run a quantised LLM with llama.cpp (GGUF)

llama.cpp is a blazingly fast C/C++ inference engine for LLMs. It supports GGUF-quantised models that run well on CPUs and can optionally leverage GPUs.

• Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j"$(nproc)"

Optional GPU acceleration at build time (if you have CUDA or ROCm):

• CUDA (NVIDIA): LLAMA_CUBLAS=1 make -j"$(nproc)"

• ROCm (AMD): LLAMA_HIPBLAS=1 make -j"$(nproc)"

• Install Hugging Face CLI to fetch a quantised model:

python -m pip install -U "huggingface_hub[cli]"

• Download a GGUF model from Hugging Face (replace the repo/model file as desired; accept model licenses as required):

mkdir -p models/mistral-7b-instruct-gguf
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  --include "*Q4_K_M.gguf" \
  --local-dir models/mistral-7b-instruct-gguf

• Run a quick prompt on CPU:

./main \
  -m models/mistral-7b-instruct-gguf/*Q4_K_M.gguf \
  -p "Write a short haiku about Linux servers." \
  -t "$(nproc)" \
  -n 128 \
  --temp 0.7

• Or start an HTTP server (handy for apps or curl testing):

./server \
  -m models/mistral-7b-instruct-gguf/*Q4_K_M.gguf \
  --port 8080 \
  -t "$(nproc)"

Tips:

• Try different quant levels: Q4_K_M (great balance), Q5_K_M (better quality, more RAM), Q8_0 (near-FP16, more RAM).

• If built with GPU support, offload layers with -ngl 20 (example value) to speed up inference.

• Increase context length with -c 4096 if your RAM allows; adjust batch size with -b 32 for throughput.

Rough RAM guidance for LLMs (model weights only, not counting KV cache):

• 7B in Q4: ~4–5 GB

• 13B in Q4: ~8–10 GB KV cache grows with context length; keep some headroom free.

---

3) Do speech-to-text with whisper.cpp (quantised Whisper)

whisper.cpp brings OpenAI’s Whisper to CPU devices. You can quantise locally to shrink the model with minimal accuracy loss.

• Build whisper.cpp:

cd ..
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make -j"$(nproc)"

• Download a base model and quantise it (q5_1 is a good trade-off):

bash ./models/download-ggml-model.sh small
./quantize models/ggml-small.bin models/ggml-small-q5_1.bin q5_1

• Transcribe a sample WAV (repo includes example audio):

./main -m models/ggml-small-q5_1.bin -f samples/jfk.wav

• Transcribe your own audio file:

ffmpeg -i your_audio.mp3 -ar 16000 -ac 1 -c:a pcm_s16le input.wav
./main -m models/ggml-small-q5_1.bin -f input.wav

Notes:

• For faster CPUs and longer recordings, try a larger model (e.g., base or medium) and quantise it similarly.

• If you see real-time factor > 1 (slower than real-time), reduce model size or number of threads.

---

4) Quantise and run an ONNX model with ONNX Runtime (INT8)

ONNX Runtime supports dynamic (post-training) INT8 quantisation that works very well for many CPU workloads.

• Install Python deps (in or out of venv):

python -m pip install -U onnx onnxruntime onnxruntime-tools numpy

• Download a sample model (ResNet50 v2 from the ONNX Model Zoo):

wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v2-7.onnx -O resnet50.onnx

• Quantise to INT8 dynamically (per-channel):

python -m onnxruntime.tools.quantization.quantize_dynamic \
  --input resnet50.onnx \
  --output resnet50.int8.onnx \
  --per_channel

• Run a quick inference (CPUExecutionProvider):

python - <<'PY'
import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("resnet50.int8.onnx", providers=["CPUExecutionProvider"])
x = np.random.rand(1, 3, 224, 224).astype(np.float32)
out = sess.run(None, {sess.get_inputs()[0].name: x})[0]
print("Output shape:", out.shape)
PY

Notes:

• Dynamic quantisation is quick and usually safe for transformers and many CNNs.

• For maximum speed on Intel/AMD CPUs, ensure you have OpenBLAS or oneDNN underneath; distro packages above already install OpenBLAS headers and libs.

---

5) Tuning tips and real-world gotchas

• Threading matters:
  • llama.cpp: use -t "$(nproc)" as a baseline, then experiment (e.g., half your cores if you see contention).
  • Pin to a NUMA node for dual-socket servers:

numactl --cpunodebind=0 --membind=0 ./main -m model.gguf -p "hello" -t 24

• Batch sizes and context:

  • Larger -b (batch) improves throughput; too large can hurt latency.
  • Longer context (-c) increases KV cache memory; plan RAM accordingly.

• Storage and I/O:

  • Keep models on fast local SSDs; avoid network mounts for hot paths.

• Accuracy vs size:

  • Prefer Q4_K_M for general LLM use, Q5_K_M when you can spare RAM for better quality.
  • For ONNX, validate accuracy post-quantisation on a small validation set.

• Licensing:

  • Many model weights (e.g., Llama, Mistral variants) require accepting terms. Authenticate with huggingface-cli login if needed.

---

Conclusion and next steps

Quantisation turns “maybe someday” into “runs today” for AI on Linux. You’ve seen how to:

• Install the right system packages (apt, dnf, zypper)

• Run a quantised LLM with llama.cpp

• Transcribe audio with a quantised Whisper model

• Quantise and serve ONNX models with ONNX Runtime

Your next steps:

• Try a different LLM and quant level; benchmark with different threads and batch sizes.

• Wrap llama.cpp’s server behind Nginx or Caddy and wire up your apps.

• For GPUs, rebuild llama.cpp with CUDA/ROCm and experiment with -ngl offloading.

• Evaluate accuracy trade-offs on a real validation set for your workload.

Have a favorite model or a performance trick? Share your results and configs with the community—your Bash one-liner could make someone else’s hardware sing.