Posted on
• Artificial Intelligence

Choosing Local Artificial Intelligence Models

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Choosing Local Artificial Intelligence Models on Linux: A Practical Guide

Local AI is having a moment. Whether you care about privacy, cost control, or latency, running models on your own Linux machine can be a game-changer. But there’s a catch: with so many runtimes, quantizations, and model families, how do you choose the right setup without wasting weekends?

This guide gives you a practical, Bash-first way to pick and run a local model that fits your hardware and your use case—fast.

Why local AI is worth it

  • Privacy and compliance: keep data on your machine

  • Reliability: offline and edge-friendly

  • Predictable cost: no API bills, no rate limits

  • Control: tune, firewall, and sandbox however you want

If those matter to you, choosing the right model and runtime is the next step.

1) Map your constraints before choosing a model

Your hardware and tasks determine your ceiling. Start with a quick inventory.

Check CPU, RAM, and disk:

lscpu | egrep 'Model name|^CPU\(s\)'
free -h
df -h /

If you have an NVIDIA GPU:

nvidia-smi || echo "No NVIDIA GPU detected"

Decide on:

  • Task category: chat/instructions, coding, summarization, RAG, multilingual

  • Data sensitivity: private vs. public

  • License needs: commercial vs. non-commercial

  • Latency expectations: interactive chat vs. batch jobs

Rule-of-thumb RAM/VRAM needs (4-bit quantized GGUF):

  • 3–4B models: ~2–3 GB

  • 7–8B models: ~4–6 GB

  • 13B models: ~8–10 GB

  • 70B models: typically GPU farm territory or pre-sharded inference

Tip: For laptops or mini-PCs, start with a 3–8B instruction-tuned model in 4–5-bit quantization.

2) Pick a model family and size

Good general-purpose choices for local use:

  • Llama 3.1 8B Instruct: strong all-rounder

  • Mistral 7B Instruct: fast, efficient, solid reasoning for size

  • Qwen2 7B Instruct: strong multilingual capability

  • Phi-3 Mini (3.8B) Instruct: tiny footprint, decent quality for its size

If you need code-heavy tasks, consider instruct/coder variants. Always check the license if you’re using models commercially.

3) Choose a runtime: Ollama or llama.cpp (both are great)

You can’t go wrong starting with either:

  • Ollama: simplest UX; easy model pulls, works well out-of-the-box

  • llama.cpp: highly optimized for CPU (and can use GPU); granular control and benchmarks

Below are installation steps with apt, dnf, and zypper where system packages are needed.

A) System prerequisites (tools and build deps)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git build-essential cmake curl wget unzip python3 python3-venv python3-pip
# Optional BLAS acceleration for llama.cpp:
sudo apt install -y libopenblas-dev

Fedora/RHEL (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake curl wget unzip python3 python3-pip python3-virtualenv
# Optional BLAS acceleration for llama.cpp:
sudo dnf install -y openblas-devel

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git gcc-c++ make cmake curl wget unzip python3 python3-pip python3-virtualenv
# Optional BLAS acceleration for llama.cpp:
sudo zypper install -y openblas-devel

Note: GPU acceleration for llama.cpp requires CUDA (NVIDIA) or other backends and is beyond basic package manager steps. Start on CPU; add GPU offload later if needed.

B) Ollama: easiest way to try models

Install:

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a model:

ollama pull llama3.1:8b
ollama run llama3.1:8b

One-shot generation:

ollama generate llama3.1:8b "Explain what quantization means for LLMs in two sentences."

Run as a background service (systemd):

# Starts automatically after install on many systems.
# If needed:
systemctl --user enable ollama
systemctl --user start ollama

C) llama.cpp: maximum control, great performance

Clone and build:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j

If you installed OpenBLAS (optional), you can enable it:

make clean
make LLAMA_OPENBLAS=1 -j

Download a quantized GGUF model. The huggingface-cli is convenient:

python3 -m pip install --upgrade huggingface_hub
# Example: Mistral 7B Instruct in GGUF (Q4_K_M). Replace with any GGUF you prefer.
# This is just an example filename; check the model repo for exact file names.
huggingface-cli download \
  TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  --local-dir ./models

Run a prompt and print timing:

./llama-cli -m ./models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -p "Give me three bullet points on why local LLMs matter." \
  -n 128 --timing

Note on binary names: recent llama.cpp builds the llama-cli binary; older guides may refer to main.

4) Quantization: your biggest speed/size lever

Quantization compresses model weights to reduce memory and improve speed at some quality cost.

Practical picks for GGUF:

  • Q4_K_M: solid default; good balance of speed and quality

  • Q5_K_M: slightly better quality; more RAM/VRAM

  • Q6_K: higher quality; needs more memory

  • Q2_K/Q3_K: smaller footprint; more quality drop

Memory estimate (rough): 8B in Q4 can fit in ~4–6 GB; 13B in Q4 needs ~8–10 GB. If you’re throttling swap, pick a smaller model or lower bit quantization.

5) Measure, don’t guess: quick local benchmarks

You want acceptable quality and interactive latency. Run a short, repeatable test.

Ollama throughput test:

/usr/bin/time -v bash -lc '
  prompt="Summarize the benefits of local AI in 5 concise bullets."
  out=$(ollama generate llama3.1:8b "$prompt")
  echo "$out" | wc -w
'

llama.cpp timing (tokens/s, prompt+gen time):

/usr/bin/time -v ./llama-cli \
  -m ./models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -p "Explain quantization to a developer." \
  -n 128 --timing

Try the same prompt with:

  • a different model size (e.g., 3–4B vs. 7–8B)

  • a different quantization (Q4 vs. Q5)

  • a different runtime (Ollama vs. llama.cpp)

Pick the fastest setup that still gives you answers you like.

Real-world starting points

  • Low-RAM laptop (8–12 GB, CPU only): Phi-3 Mini (3.8B) or Mistral 7B in Q4; run with llama.cpp or Ollama

  • Mid-range desktop (16–32 GB, CPU or modest GPU): Llama 3.1 8B or Mistral 7B in Q4/Q5; Ollama is simplest

  • Workstation (32+ GB RAM, 12–24 GB VRAM): Llama 3.1 8B/13B in higher-bit quant; experiment with GPU offload in llama.cpp

If you’re multilingual or do code-heavy work, prefer instruct/coder and multilingual variants (e.g., Qwen2 Instruct).

Troubleshooting tips

  • Hangs or OOM: choose smaller model or lower-bit quant; close RAM-hungry apps

  • Slow generations: reduce n_predict (tokens), try Q4_K_M, or move to a 3–4B model

  • Bad output quality: try a newer instruct-tuned model or move from Q4 to Q5

  • GPU not used (llama.cpp): you likely built CPU-only; add GPU backend later once CUDA/ROCm is properly installed

Conclusion and Call To Action

The “best” local model is the one that fits your hardware and answers your prompts well enough—consistently and quickly. Here’s a 15-minute path:

1) Install a runtime

  • Ollama: curl -fsSL https://ollama.com/install.sh | sh

  • Or build llama.cpp: see commands above

2) Pull a starter model

  • Ollama: ollama pull llama3.1:8b

  • llama.cpp: download a GGUF Q4 model via huggingface-cli download ...

3) Benchmark and iterate

  • Run the timing commands above

  • Adjust size/quantization until you like both the speed and the answers

Own your AI stack. Start small, measure, and grow into exactly what you need—no cloud tokens required.

Related posts