- Posted on
- Featured Image
Quantization made practical on Linux: a step-by-step guide to shrinking models 2-4x and running near real time on CPUs or low-VRAM GPUs. Install deps with apt/dnf/zypper, run GGUF LLMs via llama.cpp, transcribe with whisper.cpp, and INT8-quantize ONNX models. Includes copy-paste commands, GPU offload, RAM estimates, and tuning tips for threads, batch, context, storage, and accuracy trade-offs.