Posted on
• Artificial Intelligence

Running Open Source LLMs Locally

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Running Open-Source LLMs Locally on Linux: From Zero to First Tokens

Want ChatGPT-style power without sending your data to the cloud? You can run open-source Large Language Models (LLMs) directly on your Linux machine—privately, offline, and under your control. In this guide, we’ll show you practical, bash-first ways to get started, from the “it just works” path to full-on source builds, plus real-world examples you can copy-paste.

Why run LLMs locally?

  • Privacy and control: Keep sensitive prompts, logs, and code on your machine.

  • Latency and reliability: Instant responses, even without an internet connection.

  • Cost and scale: No per-token cloud bills; scale across your own hardware.

  • Flexibility: Choose models, quantization levels, and runtime knobs that fit your hardware.

Hardware rule of thumb:

  • CPU-only works on most machines (8–16 GB RAM recommended).

  • For faster inference, use a GPU:

    • NVIDIA CUDA (most common)
    • AMD ROCm (Linux support improving)

  • Model size matters: 7–8B parameter models run on many desktops/laptops; larger models need more RAM/VRAM and better cooling.

Step 0: Prerequisites (apt/dnf/zypper)

Install common developer tools, Python, and OpenBLAS (helpful for CPU performance).

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl git cmake build-essential pkg-config \
  python3 python3-venv python3-pip libopenblas-dev
  • Fedora/RHEL (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y curl git cmake python3 python3-virtualenv python3-pip openblas-devel
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl git cmake gcc gcc-c++ make pkg-config \
  python3 python3-pip python3-virtualenv openblas-devel

Path 1 (Easiest): Ollama — Batteries Included

Ollama is the simplest way to run LLMs locally. It downloads, configures, and serves models with a CLI and HTTP API.

1) Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

2) Pull a model (choose one):

# Balanced, capable, 8B
ollama pull llama3.1:8b

# Smaller/faster alternatives
ollama pull mistral:instruct
ollama pull phi3:mini

3) Chat from the terminal:

echo "Write a 2-line bash tip about process management." | ollama run mistral:instruct

4) Use the local HTTP API (default: http://127.0.0.1:11434):

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Give me three hardening steps for an Ubuntu server."
}'

Tips:

  • CPU-only works out of the box. For GPU acceleration, install the proper drivers/toolkits (CUDA for NVIDIA, ROCm for AMD) and restart the ollama service.

  • Manage models with ollama list, ollama show, and remove with ollama rm.

Real-world one-liner:

# Summarize the last 300 lines of syslog
tail -n 300 /var/log/syslog | sed 's/"/\\"/g' | \
  xargs -0 printf "%s" | \
  curl -s http://127.0.0.1:11434/api/generate -d '{
    "model": "mistral:instruct",
    "prompt": "Summarize key errors and warnings from this log:\n<<<LOG>>>\n'"$(cat -)"'\n<<<END>>>"}' \
  | jq -r '.response'

Path 2 (More Control): Build llama.cpp from Source

llama.cpp is a high-performance C/C++ inference engine for GGUF models. You get fine-grained control, great CPU performance, and optional GPU acceleration.

1) Clone and build (CPU + OpenBLAS):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS ..
cmake --build . -j

2) (Optional) Build with NVIDIA CUDA or AMD ROCm

  • NVIDIA (CUDA installed):
cmake -DLLAMA_CUBLAS=ON ..
cmake --build . -j
  • AMD (ROCm installed):
cmake -DLLAMA_HIPBLAS=ON ..
cmake --build . -j

Note: Ensure you’ve installed the correct toolkit/driver from NVIDIA/AMD docs before enabling these flags.

3) Get a GGUF model Use the Hugging Face CLI to download a quantized model (Q4_K_M is a good start):

pip3 install --user -U huggingface_hub hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
mkdir -p ~/models && cd ~/models
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  mistral-7b-instruct-v0.2.Q4_K_M.gguf --local-dir .

4) Run your first prompt

cd ~/llama.cpp/build
./bin/llama-cli \
  -m ~/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -t "$(nproc)" \
  -p "In one paragraph, explain what a Linux namespace is."

5) (Optional) Offload layers to GPU

./bin/llama-cli \
  -m ~/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -t "$(nproc)" \
  -ngl 35 \
  -p "Explain cgroups like I'm a sysadmin with 5 minutes."
  • The -ngl value (number of GPU layers) depends on your VRAM; start low and increase until stable.

Path 3 (Nice UI): Open WebUI + Ollama (Docker/Podman)

Prefer a browser UI with chat history and settings? Pair Open WebUI with Ollama.

1) Install a container runtime

  • Debian/Ubuntu (apt) – Docker:
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker
  • Fedora/RHEL (dnf) – Podman:
sudo dnf install -y podman
  • openSUSE (zypper) – Docker or Podman:
# Docker
sudo zypper install -y docker
sudo systemctl enable --now docker

# Or Podman
sudo zypper install -y podman

2) Run Open WebUI attached to your local Ollama

  • Docker:
docker run -d --name openwebui --restart unless-stopped \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  ghcr.io/open-webui/open-webui:main
  • Podman:
podman run -d --name openwebui --restart unless-stopped \
  --net host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  ghcr.io/open-webui/open-webui:main

Open http://127.0.0.1:3000 and start chatting with any model you’ve pulled in Ollama.

Actionable Tips for Smooth Local LLMs

1) Choose the right model and quantization

  • Start with 7–8B models (e.g., Mistral Instruct, Llama 3.1 8B).

  • Use quantized GGUF (Q4_K_M/Q5_K_M) for a good balance of speed and quality on CPUs/GPUs with limited memory.

  • Larger models (13B/70B) need much more RAM/VRAM and stronger cooling.

2) Tune for your hardware

  • CPU threads: use -t "$(nproc)" with llama.cpp.

  • GPU offload: increase -ngl gradually while monitoring VRAM.

  • Context window: increasing context increases memory use; tune conservatively (e.g., 4k–8k tokens).

3) Automate real tasks with Bash

  • Summaries: pipe logs/docs to the model and ask for condensed bullet points.

  • Code assist: prompt for shell scripts or awk/sed transformations, then review and test.

  • Data cleaning: provide example rows and ask the model to propose normalization rules.

4) Mind the resource budget

  • Watch top, nvidia-smi, or rocm-smi while testing.

  • Use lighter models for background jobs; reserve bigger models for interactive use.

  • Kill background servers when not needed:

sudo systemctl stop ollama

5) Keep models and tools up to date

  • Update Ollama and llama.cpp regularly for performance and bug fixes.

  • Refresh models as new, better quantizations appear.

A quick, practical demo you can try now

  • With Ollama:
ollama pull mistral:instruct
echo "Give me three `ufw` hardening steps, with exact commands." | ollama run mistral:instruct
  • With llama.cpp (after you’ve built it):
./bin/llama-cli -m ~/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -t "$(nproc)" \
  -p "Write a portable bash script that finds the 10 largest files in the current directory tree."

Review outputs, test safely, and iterate.

Conclusion and Call to Action

You don’t need a data center—or the cloud—to use powerful LLMs. Start simple with Ollama, graduate to llama.cpp for fine control, and add a web UI when you want a friendlier front end. Your next steps:

1) Install prerequisites (apt/dnf/zypper) and pick a path (Ollama or llama.cpp). 2) Pull a small instruct model and run your first prompt. 3) Automate a real task from your daily workflow. 4) Experiment with quantization, threads, and GPU offload for speedups.

When you’ve got your first tokens flowing, share your setup and tips with the community—and script the next boring task out of your life.

Related posts