Running Local LLMs on Linux with Bash

Want ChatGPT‑style power without the cloud? Run large language models (LLMs) locally on your Linux box—fast, private, and scriptable with pure Bash. In this guide, you’ll go from zero to “talking to a model in your terminal,” with two practical paths:

• The easy way with Ollama (one command, auto‑managed models)

• The bare‑metal way with llama.cpp (maximum control and performance tuning)

You’ll get copy‑paste installs for apt, dnf, and zypper, actionable Bash snippets, and real‑world tips for speed and ergonomics.

Why run LLMs locally?

• Privacy and control: Your data never leaves your machine.

• Latency: Replies start streaming in milliseconds—no network roundtrips.

• Cost: Stop burning API credits for everyday prompts.

• Hackability: Bash + Unix tools = infinite automation.

• Portability: Freeze exact models, quantization, and parameters for reproducibility.

---

Prerequisites

Hardware (rough guidance; you can start smaller):

• CPU: Any modern x86_64; more cores = more tokens/s.

• RAM: 8–16 GB for 7B models, 24–32 GB for 13B+. Swap helps in a pinch.

• Optional GPU:

  • NVIDIA: Great speedups with CUDA/cuBLAS.
  • AMD: ROCm/hipBLAS.
  • Intel: SYCL (Arc/iGPU).

Base packages (choose your package manager):

# Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y git build-essential cmake curl wget unzip

# Fedora/RHEL/CentOS (dnf)
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git cmake curl wget unzip

# openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y git cmake curl wget unzip

Optional helpers used later:

# apt
sudo apt install -y tmux

# dnf
sudo dnf install -y tmux

# zypper
sudo zypper install -y tmux

Note: GPU toolkits (CUDA/ROCm) are optional and distro‑specific; install them from your vendor/distro if you plan to offload to GPU.

---

Path A (easiest): Run local LLMs with Ollama

Ollama bundles model download, quantization, and a clean CLI/server.

1) Install Ollama (the script detects apt/dnf/zypper and sets up a service):

curl -fsSL https://ollama.com/install.sh | sh

If the service doesn’t start automatically:

sudo systemctl enable --now ollama

2) Pull a model (choose one):

# Great general chat models:
ollama pull llama3:instruct
ollama pull mistral:instruct
ollama pull qwen2:7b-instruct

3) Chat from Bash:

# One-off prompt
ollama run llama3:instruct "Explain Linux namespaces in two sentences."

# Pipe input
echo "Summarize: $(uname -a)" | ollama run mistral:instruct

# Interactive shell
ollama run qwen2:7b-instruct

4) Use the local HTTP API (handy for scripts or other tools):

curl http://localhost:11434/api/generate -d '{
  "model": "llama3:instruct",
  "prompt": "Write a 10-word haiku about Bash."
}'

Tips:

• List installed models: ollama list

• Delete a model: ollama rm modelname

• Set bind address/port: export OLLAMA_HOST=0.0.0.0:11434 && systemctl --user restart ollama (or adjust service)

---

Path B (maximum control): llama.cpp from source

llama.cpp is a high‑performance C/C++ inference engine for GGUF models, with CPU and GPU backends, a CLI, and an OpenAI‑style HTTP server.

1) Clone and build:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# CPU-only (portable):
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# NVIDIA GPU (CUDA/cuBLAS) acceleration:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUBLAS=ON
cmake --build build -j

# AMD GPU (ROCm/hipBLAS):
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_HIPBLAS=ON
cmake --build build -j

# Intel (SYCL):
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_SYCL=ON
cmake --build build -j

2) Get a quantized GGUF model

• You can use any compatible GGUF. Example (Mistral 7B Instruct, Q4_K_M quantization):

mkdir -p models/mistral
cd models/mistral
wget -O mistral-7b-instruct.Q4_K_M.gguf \
  https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf
cd ../../

3) Run from the terminal

# Replace the binary name if your build outputs 'main' instead of 'llama-cli'
./build/bin/llama-cli \
  -m models/mistral/mistral-7b-instruct.Q4_K_M.gguf \
  -p "Explain cgroups v2 in 3 bullet points." \
  -t $(nproc) \
  -c 4096

GPU offload (mix CPU+GPU) for faster inference:

./build/bin/llama-cli \
  -m models/mistral/mistral-7b-instruct.Q4_K_M.gguf \
  --ngl 20 -t $(nproc) -c 4096 \
  -p "Give three Bash tips for working with JSON."

• --ngl = number of layers to offload to GPU (tune based on VRAM).

• For interactive chat, add -i and -r "User:" style roles, or simply run without -p and type.

4) Serve an OpenAI‑style API locally

./build/bin/llama-server \
  -m models/mistral/mistral-7b-instruct.Q4_K_M.gguf \
  -c 4096 --port 8080

Call it with curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role":"user","content":"Generate a Bash oneliner to count unique IPs in a log."}],
    "stream": true
  }'

Note: llama.cpp binary names may differ slightly across versions (e.g., main, llama-cli, llama-server). Run ls build/bin to confirm.

---

Bash superpowers: Script your AI

1) A tiny CLI wrapper

# Put this in ~/.bashrc or a script in your PATH
ai() {
  # Prefer Ollama if present; fallback to llama.cpp
  if command -v ollama >/dev/null 2>&1; then
    if [ -t 0 ]; then
      ollama run "${OLLAMA_MODEL:-llama3:instruct}" "$*"
    else
      ollama run "${OLLAMA_MODEL:-llama3:instruct}"
    fi
  elif [ -x "$HOME/llama.cpp/build/bin/llama-cli" ]; then
    MODEL="${LLAMA_MODEL_PATH:-$HOME/models/mistral/mistral-7b-instruct.Q4_K_M.gguf}"
    if [ -t 0 ]; then
      "$HOME/llama.cpp/build/bin/llama-cli" -m "$MODEL" -t "$(nproc)" -c 4096 -p "$*"
    else
      "$HOME/llama.cpp/build/bin/llama-cli" -m "$MODEL" -t "$(nproc)" -c 4096
    fi
  else
    echo "No local LLM backend found (install Ollama or llama.cpp)" >&2
    return 1
  fi
}

Usage:

ai "Summarize the differences between apt, dnf, and zypper."

# Pipe a file
ai "Summarize this changelog:" < CHANGELOG.md

2) An interactive REPL in pure Bash

llmsh() {
  MODEL="${OLLAMA_MODEL:-llama3:instruct}"
  echo "Local LLM REPL (Ctrl+C to exit). Using model: $MODEL"
  while true; do
    read -rp "> " prompt || break
    [ -z "$prompt" ] && continue
    ollama run "$MODEL" "$prompt"
  done
}

3) Automate tasks

• Generate commit messages:

git diff | ai "Write a concise, imperative git commit message for this diff:"

• Explain a script:

ai "Explain what this script does, step by step:" < ./backup.sh

---

Performance tuning cheatsheet

• Choose the right quantization:

  • Q4_K_M: good balance of speed + quality on CPU.
  • Q5_K_M / Q6_K: higher quality; need more RAM/VRAM.
  • Q8: near‑fp16 quality; much larger memory.

• Threads: -t $(nproc) usually best; try fewer on NUMA/thermal limits.

• Context: -c 2048–4096 is typical; larger uses more RAM and can be slower.

• GPU offload: --ngl N for llama.cpp; start small and increase until VRAM is full but not swapping.

• Prompt caching (llama.cpp): --prompt-cache cache.bin speeds up repeated contexts.

• Keep logs tidy with tmux:

tmux new -s llm './build/bin/llama-server -m model.gguf -c 4096 --port 8080 | tee server.log'

---

Real‑world examples

• Private docs Q&A:

  • Convert PDFs to text, feed chunks to the model with a retrieval script.

• On‑device coding helper:

  • Pipe snippets into ai for refactors without sending code off‑box.

• Air‑gapped environments:

  • Pre‑download models; then run fully offline on secure machines.

Model licensing reminder: Many weights are for research/non‑commercial use. Always check the model’s license before deploying.

---

Conclusion and next steps

You now have two solid paths to run local LLMs on Linux:

• Need the fastest setup and an HTTP API out of the box? Install Ollama and pull a chat model.

• Want knobs to turn and maximum control? Build llama.cpp, pick a GGUF, and script it in Bash.

Your next steps: 1) Install the tooling (apt/dnf/zypper commands above). 2) Pick a model and run your first prompt. 3) Drop the ai() function into your shell and start automating.

If this helped, try swapping models, tuning quantization, and wiring your local LLM into a script you already use daily. Happy hacking!