quantization

The article guides Linux users to run open-source LLMs locally for privacy, speed, and cost control, offering three paths: easiest with Ollama (CLI/API), more control by building llama.cpp (CPU/GPU, GGUF), and a web UI via Open WebUI + Ollama. It covers prerequisites, model/quantization choices, tuning (threads, GPU offload, context), bash automations, resource monitoring, and updates—ending with quick demos and action steps.

Hands-on, Bash-first guide to local AI on Linux: map hardware and use case; choose model families and sizes (Llama 3.1, Mistral, Qwen2, Phi-3) with the right quantization (Q4–Q6); pick a runtime (Ollama for simplicity, llama.cpp for control); install via apt/dnf/zypper; pull GGUFs; benchmark and iterate; includes RAM/VRAM rules, starter builds, and troubleshooting to get private, fast, predictable LLMs in ~15 minutes.

Practical guide to speed up Ollama on Linux: store models on NVMe and keep them hot (OLLAMA_KEEP_ALIVE, OLLAMA_MAX_LOADED_MODELS), pick sensible quantization (q4_K_M/q5_1/q8_0/f16), offload layers to GPU and tune num_gpu_layers, num_batch, num_ctx, set sane concurrency and OLLAMA_MAX_QUEUE, warm favorite models, and benchmark tokens/sec via the HTTP API. Yields snappier first tokens and higher throughput.

Practical Linux guide to speed up Ollama: explains why tuning matters and gives copy-paste steps to boost tokens/sec and cut latency - pick the right model/quant, enable CUDA/ROCm GPU offload and adjust num_gpu, set threads and context (num_thread/num_ctx), keep models warm, store GGUFs on fast NVMe via OLLAMA_MODELS, benchmark with curl+jq and monitor with htop/nvtop - plus quick CPU/GPU recipes and common pitfalls.