Posted on
• Artificial Intelligence

Running Multiple Local LLMs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Running Multiple Local LLMs on One Linux Box: A Practical, Bash‑First Guide

Ever wish you could run a coding assistant, a summarizer, and a lightweight Q&A model side by side on the same machine—without shipping your data to the cloud? With the right setup, you can serve multiple local LLMs concurrently on Linux, even on modest hardware. This article shows you how, using command-line tools and straightforward process controls you already know.

Why this matters:

  • Privacy and control: Keep prompts and data on your machine.

  • Cost and reliability: No per-token bills; works offline.

  • Speed and flexibility: Pin specific models to specific tasks; test, benchmark, and swap at will.

Below you’ll set up multiple LLM “micro‑services” using llama.cpp, give them resource boundaries so they don’t fight, and put a single front door in front of them with NGINX. There’s also a quick Ollama alternative if you want something simpler.

1) Prerequisites: Build tools, OpenBLAS, and basics

Install the tools we’ll use to build and run servers, plus optional NGINX and tmux.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  git build-essential cmake ninja-build pkg-config libopenblas-dev \
  python3 python3-venv python3-pip curl wget unzip tmux nginx

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y \
  git gcc gcc-c++ cmake make ninja-build pkgconfig openblas-devel \
  python3 python3-virtualenv python3-pip curl wget unzip tmux nginx

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  git gcc gcc-c++ cmake make ninja pkg-config openblas-devel \
  python3 python3-virtualenv python3-pip curl wget unzip tmux nginx

Optional GPU checks (only if you have a GPU and drivers installed):

# NVIDIA
nvidia-smi

# AMD ROCm
rocminfo

2) Run two LLM servers with llama.cpp (GGUF), each on its own port

We’ll build llama.cpp with OpenBLAS (CPU-friendly) and run two separate HTTP servers—one “coder”, one “summarizer”.

Build llama.cpp (CPU-first; add GPU flags later if you like):

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -S . -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build -j

Optional: Build with GPU support if your drivers/workflow are ready.

  • NVIDIA CUDA:
cmake -B build -S . -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_CUDA=ON
cmake --build build -j
  • AMD ROCm (example targets; adjust as needed):
cmake -B build -S . -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1030
cmake --build build -j

Download two example GGUF models (one tiny, one mid-size):

mkdir -p ~/models && cd ~/models

# TinyLlama (tiny, good for tests)
wget -O tinyllama-q4.gguf \
  "https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf?download=true"

# Mistral 7B Instruct (mid-size; requires more RAM/VRAM)
wget -O mistral-7b-instruct-q4.gguf \
  "https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf?download=true"

Start two servers (different ports). The server binary is typically build/bin/llama-server or build/bin/server—use what you have:

cd ~/llama.cpp
LLAMA_BIN=$(ls build/bin/*server 2>/dev/null | head -n1)

# 1) Tiny "coder" service, port 8081 (CPU threads: 8, context: 2048)
"$LLAMA_BIN" -m ~/models/tinyllama-q4.gguf \
  --host 127.0.0.1 --port 8081 -c 2048 -t 8

# 2) "summarizer" service, port 8082 (threads: 12, larger context)
# If you built with CUDA/ROCm, consider offloading some layers: --n-gpu-layers 30
"$LLAMA_BIN" -m ~/models/mistral-7b-instruct-q4.gguf \
  --host 127.0.0.1 --port 8082 -c 4096 -t 12

Quick tests via HTTP (try either the simple or OpenAI-compatible endpoint):

# Simple completion endpoint (llama.cpp server)
curl -s http://127.0.0.1:8081/completion \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "Write a one-line greeting.", "n_predict": 64 }'

# OpenAI-compatible chat endpoint (many clients already speak this)
curl -s http://127.0.0.1:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{ "model":"local-mistral", "messages":[{"role":"user","content":"Summarize why local LLMs are useful."}] }'

Tips:

  • Prefer quantized models (Q4_K_M, Q5_K_M) for lower RAM/VRAM.

  • Use smaller context sizes (-c) if memory is tight.

3) Make them play nice: CPU, GPU, and memory boundaries

Pin CPU cores and set niceness so the two servers don’t starve each other:

# Bind "coder" to cores 0-7; nicer priority (lower CPU pressure)
taskset -c 0-7 nice -n 5 "$LLAMA_BIN" -m ~/models/tinyllama-q4.gguf \
  --host 127.0.0.1 --port 8081 -c 2048 -t 8

# Bind "summarizer" to cores 8-15; slightly lower prio (higher nice)
taskset -c 8-15 nice -n 10 "$LLAMA_BIN" -m ~/models/mistral-7b-instruct-q4.gguf \
  --host 127.0.0.1 --port 8082 -c 4096 -t 8

If you have multiple GPUs, isolate them:

# NVIDIA
CUDA_VISIBLE_DEVICES=0 "$LLAMA_BIN" -m ~/models/tinyllama-q4.gguf --host 127.0.0.1 --port 8081 -c 2048 -t 8 --n-gpu-layers 20
CUDA_VISIBLE_DEVICES=1 "$LLAMA_BIN" -m ~/models/mistral-7b-instruct-q4.gguf --host 127.0.0.1 --port 8082 -c 4096 -t 8 --n-gpu-layers 35

# AMD ROCm
HIP_VISIBLE_DEVICES=0  "$LLAMA_BIN" ...
HIP_VISIBLE_DEVICES=1  "$LLAMA_BIN" ...

Use systemd-run for quick cgroup quotas (no unit files needed):

systemd-run --user -u llm-coder \
  -p CPUQuota=150% -p MemoryMax=6G -p Nice=5 \
  "$LLAMA_BIN" -m ~/models/tinyllama-q4.gguf --host 127.0.0.1 --port 8081 -c 2048 -t 8

systemd-run --user -u llm-summarizer \
  -p CPUQuota=200% -p MemoryMax=12G -p Nice=10 \
  "$LLAMA_BIN" -m ~/models/mistral-7b-instruct-q4.gguf --host 127.0.0.1 --port 8082 -c 4096 -t 8

Monitor usage in real time:

top -H -p $(pgrep -d, -f "$LLAMA_BIN")
nvidia-smi   # or rocminfo / rocm-smi

4) One front door: route by path with NGINX

Put both services behind a single URL and let clients choose a model by path.

Install/reconfirm NGINX (if you skipped earlier):

  • apt:
sudo apt update && sudo apt install -y nginx
  • dnf:
sudo dnf install -y nginx
  • zypper:
sudo zypper install -y nginx

Create a simple proxy config:

sudo tee /etc/nginx/conf.d/llm.conf >/dev/null <<'EOF'
upstream llm_coder { server 127.0.0.1:8081; }
upstream llm_summarizer { server 127.0.0.1:8082; }

server {
    listen 8000;
    server_name _;

    # /coder/* -> tiny coder model
    location /coder/ {
        proxy_pass http://llm_coder/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
    }

    # /summarizer/* -> larger summarizer model
    location /summarizer/ {
        proxy_pass http://llm_summarizer/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
    }
}
EOF

sudo nginx -t && sudo systemctl reload nginx

Test through NGINX:

curl -s http://127.0.0.1:8000/coder/completion \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "Write a tiny bash oneliner that prints the date.", "n_predict": 64 }'

curl -s http://127.0.0.1:8000/summarizer/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{ "model":"any", "messages":[{"role":"user","content":"Summarize the Unix philosophy."}] }'

5) Make it stick: systemd user services

Turn each LLM into a managed service that starts on login and restarts on failure.

Coder service:

mkdir -p ~/.config/systemd/user
tee ~/.config/systemd/user/llm-coder.service >/dev/null <<'EOF'
[Unit]
Description=Local LLM - coder (llama.cpp)

[Service]
ExecStart=/bin/bash -lc 'LLAMA_BIN=$(ls ~/llama.cpp/build/bin/*server | head -n1); taskset -c 0-7 nice -n 5 "$LLAMA_BIN" -m ~/models/tinyllama-q4.gguf --host 127.0.0.1 --port 8081 -c 2048 -t 8'
Restart=on-failure
RestartSec=2

[Install]
WantedBy=default.target
EOF

Summarizer service:

tee ~/.config/systemd/user/llm-summarizer.service >/dev/null <<'EOF'
[Unit]
Description=Local LLM - summarizer (llama.cpp)

[Service]
ExecStart=/bin/bash -lc 'LLAMA_BIN=$(ls ~/llama.cpp/build/bin/*server | head -n1); taskset -c 8-15 nice -n 10 "$LLAMA_BIN" -m ~/models/mistral-7b-instruct-q4.gguf --host 127.0.0.1 --port 8082 -c 4096 -t 8'
Restart=on-failure
RestartSec=2

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now llm-coder llm-summarizer
loginctl enable-linger "$USER"  # keep services running after you log out (optional)

Alternative: Quick multi‑model serving with Ollama

Ollama runs multiple models under one daemon and exposes a simple local API. It’s the fastest path to “two models at once” if you don’t want to compile anything.

Install (official script):

curl -fsSL https://ollama.com/install.sh | sh

Pull two models and use them concurrently:

ollama pull llama3:8b
ollama pull mistral:7b-instruct

# Start the server (usually auto-starts)
ollama serve &

# Two terminals (or background jobs), two models:
ollama run llama3:8b
ollama run mistral:7b-instruct

Programmatic calls (same server, different model names):

curl http://localhost:11434/api/generate \
  -d '{ "model": "llama3:8b", "prompt": "Write a POSIX shell one-liner to count lines in *.log" }'

curl http://localhost:11434/api/generate \
  -d '{ "model": "mistral:7b-instruct", "prompt": "Summarize the last git commit in one sentence." }'

Tip: Prefer smaller/quantized variants when running more than one model at once.

Real‑world patterns that work

  • Divide and conquer: a tiny, fast model for intent detection or routing; a mid-size model for the “heavy” reply.

  • Latency tiers: small model produces a quick draft; larger model refines when latency budget allows.

  • Dedicated GPUs: one GPU per model with CUDA_VISIBLE_DEVICES or HIP_VISIBLE_DEVICES, keeping interference low.

  • Safe defaults: cap context sizes (-c), pin CPU cores, and set MemoryMax with systemd to prevent OOM.

Conclusion and next steps

You now have a practical way to run multiple local LLMs at once, with resource boundaries and a clean HTTP front door. Your next step:

  • Pick two models that fit your hardware (one small, one mid).

  • Stand up both with llama.cpp on different ports.

  • Put NGINX in front and route by path.

  • Add systemd user services so the stack survives reboots.

From here, you can add logging, TLS, a vector database, or a simple Bash router that picks models based on prompt size. If you build something cool—or want a follow-up on GPU tuning or streaming—tell me what you’d like to see next!

Related posts