Posted on
• Artificial Intelligence

Learning Local LLMs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Learning Local LLMs on Linux: Private, Fast, and Scriptable

Tired of shipping your code, logs, or customer data to a cloud AI? Local LLMs let you run powerful language models entirely on your Linux box—no API keys, no token limits, no vendor latency. Even better: they play nicely with Bash, so you can pipe text straight from your terminal into an LLM and back.

This post shows you why local LLMs are worth your time and exactly how to get started on Linux with two battle‑tested options (Ollama and llama.cpp), including install commands for apt, dnf, and zypper. You’ll finish with working examples you can drop into your shell scripts today.

Why local LLMs are worth it

  • Privacy by default: keep code, logs, and documents on your machine.

  • Predictable cost: no metered tokens or surprise bills.

  • Low latency: responses in milliseconds over localhost.

  • Hackable: script with Bash, embed in cron jobs, or run air‑gapped.

  • Reproducible: pin model files and configurations in your repo.

Choose your path

Two excellent, Linux‑friendly ways to run models locally:

  • Ollama (easiest): one command to install, one command to run; REST API included.

  • llama.cpp (bare‑metal): tiny native binaries, blazing CPU/GPU inference, lots of flags.

You can install both and pick the right tool per task.

1) Install the runtime

Option A: Install Ollama (recommended for most users)

Ollama bundles model management, quantized models, and a local server.

curl -fsSL https://ollama.com/install.sh | sh

Start/check the user service:

systemctl --user enable --now ollama
systemctl --user status ollama

If you don’t use systemd user services, run:

ollama serve

Dependencies (if you don’t have curl, git, etc.):

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl git
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl git
  • openSUSE (zypper):
sudo zypper install -y curl git

Option B: Build llama.cpp (maximum control and speed)

1) Install build tools and optional BLAS for faster CPU:

  • Debian/Ubuntu:
sudo apt update
sudo apt install -y git cmake build-essential libopenblas-dev
  • Fedora/RHEL/CentOS:
sudo dnf install -y git cmake gcc gcc-c++ make openblas-devel
  • openSUSE:
sudo zypper install -y git cmake gcc-c++ make openblas-devel

2) Clone and build (CPU‑only, fast and simple):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS
cmake --build build -j

3) Optional: build with your GPU for big speedups

  • NVIDIA (requires CUDA toolkit installed):
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUBLAS=ON
cmake --build build -j
  • AMD (ROCm):
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_HIPBLAS=ON
cmake --build build -j

The main CLI ends up at:

./build/bin/llama-cli

2) Get a model

Quantized models run well on consumer hardware. Start small, then scale.

With Ollama

Pull and run an 8B model (about 4–6 GB RAM used with 4‑bit quantization):

ollama pull llama3.1:8b
ollama run llama3.1:8b

Other good starters:

ollama pull mistral:7b
ollama pull phi3:mini

List installed models:

ollama list

With llama.cpp

Download a GGUF file (quantized format). Pick a 7B “Instruct” model in Q4_K_M (≈4–5 GB RAM). For example (replace URL with the model you choose on Hugging Face):

mkdir -p ~/models
cd ~/models
wget -O qwen2-7b-instruct.Q4_K_M.gguf \
  https://huggingface.co/YourChosenRepo/resolve/main/qwen2-7b-instruct.Q4_K_M.gguf

Run it:

./llama.cpp/build/bin/llama-cli \
  -m ~/models/qwen2-7b-instruct.Q4_K_M.gguf \
  -p "Explain the difference between a subshell and a child process in Bash." \
  -n 200 -t $(nproc)

Tips:

  • RAM ballpark (Q4): 7B ≈ 4–5 GB, 13B ≈ 8–10 GB. Leave headroom for your OS.

  • If you built with CUDA/ROCm, add GPU offload, e.g. -ngl 35 (tune by VRAM).

3) Talk to your model from Bash

Quick prompts

  • Ollama (interactive):
ollama run mistral:7b
  • Ollama (one‑shot prompt):
ollama run phi3:mini "Give me three ways to speed up a Bash script that parses logs."
  • llama.cpp (one‑shot, CPU threads auto‑set):
./build/bin/llama-cli -m ~/models/qwen2-7b-instruct.Q4_K_M.gguf \
  -p "Write a robust Bash for-loop that retries a failing command 3 times." \
  -n 180 -t $(nproc)

Use REST from curl (Ollama’s local API)

Install jq if you want pretty JSON:

  • apt:
sudo apt update && sudo apt install -y jq
  • dnf:
sudo dnf install -y jq
  • zypper:
sudo zypper install -y jq

Call the API:

curl -s http://localhost:11434/api/generate \
  -d '{
        "model": "mistral:7b",
        "prompt": "In 5 bullets, best practices for secure SSH on Linux."
      }' | jq -r '.response' | tr -d '\n'

4) Real‑world Bash automations

  • Summarize recent logs
journalctl -n 500 --no-pager \
| sed '1s/^/Summarize the key warnings and errors in 5 bullet points:\n/' \
| ollama run phi3:mini
  • Explain a shell one‑liner (llama.cpp)
echo 'for f in *.log; do grep -H "ERROR" "$f"; done' \
| sed '1s/^/Explain what this shell one-liner does, step by step:\n/' \
| ./build/bin/llama-cli -m ~/models/qwen2-7b-instruct.Q4_K_M.gguf -n 200 -t $(nproc)
  • Draft a conventional commit message from staged changes
git diff --staged \
| sed '1s/^/Write a concise conventional commit message (type: feat/fix/docs/etc.) for this diff:\n/' \
| ollama run llama3.1:8b
  • Translate a README to another language
sed '1s/^/Translate this Markdown to Spanish, keep code blocks intact:\n/' README.md \
| ollama run mistral:7b

5) Tune for speed and quality

  • Pick the right size:

    • 3–4B: very fast on CPUs; good for simple tasks.
    • 7–8B: good general assistant on most laptops.
    • 13B+: better reasoning; prefer GPU or lots of RAM.

  • Quantization:

    • Q4_K_M is a strong default; Q5_* is higher quality; Q8_* is near‑fp16 but heavy.

  • llama.cpp flags:

    • -t $(nproc) threads, -c 4096 context tokens, -b 512 batch size, -ngl N GPU layers.

  • Ollama tips:

    • Use smaller models for CLI glue, larger for deep reasoning.
    • Create a Modelfile to set defaults (e.g., temperature, num_ctx) per model.

Monitor:

time <your command>        # wall-clock
nvidia-smi                 # GPU usage (NVIDIA)
top/htop                   # CPU and RAM

Troubleshooting quick hits

  • Out of memory? Drop to a smaller model or lower quantization (Q4 → Q3_K_M), reduce -c context.

  • Slow or choppy output? Increase threads (-t), enable BLAS/GPU builds, reduce -n (max tokens).

  • Garbage or role confusion? Use an “Instruct”‑tuned model and include a clear instruction prefix.

Conclusion and next steps

Local LLMs unlock private, scriptable AI that fits right into your Linux workflow. Start simple: 1) Install Ollama or build llama.cpp. 2) Pull a 7–8B instruct model. 3) Wire it into one daily task (log summaries, commit drafting, or explaining Bash).

From there, explore:

  • RAG: feed your docs to the model via prompt chunks or a lightweight retriever.

  • Long contexts: try models with 8K–32K contexts for larger files.

  • Automation: wrap prompts in Makefiles, cron jobs, or git hooks.

Your next step: pick one runtime above, run a prompt that improves a task you do daily, and save the winning one‑liner to your dotfiles.

Related posts