Posted on
• Artificial Intelligence

Running Multiple LLMs

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Running Multiple LLMs on Linux: A Practical, Bash-First Guide

Ever wanted to A/B test two models, dedicate one GPU to a code model and another to a chat model, or route different teams to different assistants—all on one box? Running multiple LLMs concurrently is not only possible on Linux; it’s straightforward once you have the right runtime, isolation strategy, and a repeatable way to start/stop services.

This guide explains why you might want to run multiple LLMs, then walks you through actionable setups with Bash-friendly tooling (Ollama, llama.cpp, and optional vLLM), resource isolation, routing, and monitoring. Everything here is terminal-first and includes install commands for apt, dnf, and zypper where relevant.

Why run multiple LLMs?

  • Specialization: Route code questions to a code-tuned model and general chat to a general model.

  • A/B testing: Compare latency and quality across models in real-time with identical prompts.

  • Resource utilization: Keep multi-GPU rigs fully occupied; don’t leave VRAM idle.

  • Isolation: Prevent one heavy job from starving another. Pin CPU, split GPUs, and cap memory.

  • High availability: If one process crashes or gets OOM-killed, others keep serving.

Prerequisites

Install common build, Python, and ops tools (pick your distro). You don’t need all of these for every option below, but having them available saves time later.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  git build-essential cmake curl wget \
  python3 python3-venv python3-pip \
  tmux nginx nvtop htop \
  docker.io docker-compose-plugin

Fedora/RHEL (dnf):

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y \
  git cmake curl wget \
  python3 python3-pip python3-virtualenv \
  tmux nginx nvtop htop \
  docker docker-compose-plugin
sudo systemctl enable --now docker

openSUSE (zypper):

sudo zypper install -t pattern devel_basis
sudo zypper install -y \
  git cmake curl wget \
  python3 python3-pip python3-virtualenv \
  tmux nginx nvtop htop \
  docker docker-compose
sudo systemctl enable --now docker

Notes:

  • GPU use requires a working NVIDIA driver/CUDA stack. Verify with:
nvidia-smi

Option A: Multiple LLMs the easy way with Ollama

Ollama runs models locally with a simple API. You can run multiple Ollama servers on different ports (and even bind each to different GPUs).

Install Ollama (Linux):

curl -fsSL https://ollama.com/install.sh | sh

Run two Ollama servers on different ports and model directories:

# Terminal 1: Llama 3 server on port 11434, GPU 0
export OLLAMA_HOST=127.0.0.1:11434
export OLLAMA_MODELS=$HOME/.ollama/llama3-models
export CUDA_VISIBLE_DEVICES=0
mkdir -p "$OLLAMA_MODELS"
ollama serve

# Terminal 2: Mistral server on port 11435, GPU 1
export OLLAMA_HOST=127.0.0.1:11435
export OLLAMA_MODELS=$HOME/.ollama/mistral-models
export CUDA_VISIBLE_DEVICES=1
mkdir -p "$OLLAMA_MODELS"
ollama serve

Pull and test models against each instance:

# Talk to the Llama 3 server
export OLLAMA_HOST=127.0.0.1:11434
ollama pull llama3
curl -s http://127.0.0.1:11434/api/generate -d '{"model":"llama3","prompt":"Hello!"}'

# Talk to the Mistral server
export OLLAMA_HOST=127.0.0.1:11435
ollama pull mistral
curl -s http://127.0.0.1:11435/api/generate -d '{"model":"mistral","prompt":"Hello!"}'

Tip:

  • Assign CPUs to each server to avoid contention:
# Pin first server to cores 0-15, second to 16-31
taskset -cp 0-15 $(pgrep -f "ollama serve" | sed -n '1p')
taskset -cp 16-31 $(pgrep -f "ollama serve" | sed -n '2p')

Option B: High-control, lightweight servers with llama.cpp

llama.cpp is a fast C/CUDA inference engine for GGUF models. You can run multiple HTTP servers with different ports and models.

Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
# CPU-only:
make -j
# Or GPU-accelerated (CUDA):
LLAMA_CUBLAS=1 make -j

Run two servers on different ports (adjust paths to your GGUF models):

# Server 1 on GPU 0
CUDA_VISIBLE_DEVICES=0 ./server \
  -m /models/llama3/llama3-8b-instruct.Q4_K_M.gguf \
  --port 8001 --ctx-size 4096 --threads 8

# Server 2 on GPU 1
CUDA_VISIBLE_DEVICES=1 ./server \
  -m /models/mistral/mistral-7b-instruct.Q5_K_M.gguf \
  --port 8002 --ctx-size 4096 --threads 8

Query each server:

curl -s http://127.0.0.1:8001/completion -H "Content-Type: application/json" \
  -d '{"prompt":"Hi from server 1","n_predict":64}'

curl -s http://127.0.0.1:8002/completion -H "Content-Type: application/json" \
  -d '{"prompt":"Hi from server 2","n_predict":64}'

Performance tips:

# Pin CPUs (example: 0-15 for server 1, 16-31 for server 2)
taskset -c 0-15 ./server ...
taskset -c 16-31 ./server ...

# NUMA-aware (if you have multiple sockets)
numactl --cpunodebind=0 --membind=0 ./server ...
numactl --cpunodebind=1 --membind=1 ./server ...

# Control threading
export OMP_NUM_THREADS=8

Option C (advanced): Throughput-first serving with vLLM

vLLM provides an OpenAI-compatible server with excellent throughput on GPUs.

Install with pip:

python3 -m venv ~/.venvs/vllm
source ~/.venvs/vllm/bin/activate
pip install --upgrade pip
pip install vllm

Run two vLLM servers on different ports/GPUs:

# Server 1 (GPU 0)
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --port 9001

# Server 2 (GPU 1)
CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --port 9002

Query:

curl http://127.0.0.1:9001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct","messages":[{"role":"user","content":"Hello!"}]}'

Note: You may need a Hugging Face token and sufficient VRAM.

Resource isolation essentials (CPU, RAM, GPU)

  • CPU pinning:
taskset -c 0-15 <your-llm-command>
  • Memory capping (systemd-run creates a transient scope with cgroup limits):
systemd-run --scope -p MemoryMax=24G -p CPUQuota=200% bash -lc '<your-llm-command>'
  • GPU assignment:
CUDA_VISIBLE_DEVICES=0 <your-llm-on-gpu0>
CUDA_VISIBLE_DEVICES=1 <your-llm-on-gpu1>
  • MIG (A100/H100, optional): Partition a GPU into slices; then bind each process to a slice. Refer to nvidia-smi mig -h.

Routing multiple models behind one URL with Nginx

Install Nginx if you haven’t:

  • apt:
sudo apt install -y nginx
sudo systemctl enable --now nginx
  • dnf:
sudo dnf install -y nginx
sudo systemctl enable --now nginx
  • zypper:
sudo zypper install -y nginx
sudo systemctl enable --now nginx

Example path-based routing to two backends:

# /etc/nginx/conf.d/llms.conf
upstream llama3_upstream { server 127.0.0.1:11434; }  # Ollama or llama.cpp
upstream mistral_upstream { server 127.0.0.1:11435; }

server {
  listen 80;
  server_name _;

  location /llama3/ {
    proxy_pass http://llama3_upstream/;
    proxy_set_header Host $host;
  }

  location /mistral/ {
    proxy_pass http://mistral_upstream/;
    proxy_set_header Host $host;
  }
}

Reload:

sudo nginx -t && sudo systemctl reload nginx

Now:

curl -s http://localhost/llama3/api/tags
curl -s http://localhost/mistral/api/tags

Keep processes alive: tmux, systemd, or Docker

  • tmux (quick and dirty):
tmux new -s llms
# start your servers in separate panes/windows
  • systemd template units (robust). Example for two Ollama instances:
# /etc/systemd/system/ollama@.service
[Unit]
Description=Ollama instance %i
After=network.target

[Service]
Environment=OLLAMA_HOST=127.0.0.1:%i
Environment=OLLAMA_MODELS=/var/lib/ollama/%i
Environment=CUDA_VISIBLE_DEVICES=%I
ExecStartPre=/bin/mkdir -p /var/lib/ollama/%i
ExecStart=/usr/local/bin/ollama serve
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Start instances on ports 11434 and 11435 (mapping GPU 0 and 1 respectively):

sudo systemctl daemon-reload
sudo systemctl enable --now ollama@11434.service
sudo systemctl enable --now ollama@11435.service
  • Docker Compose (portable). Example with two llama.cpp servers:
# docker-compose.yml
services:
  llama3:
    image: ghcr.io/ggerganov/llama.cpp:server-cublas
    command: ["-m","/models/llama3/llama3-8b-instruct.Q4_K_M.gguf","--port","8001","--ctx-size","4096"]
    ports: ["8001:8001"]
    volumes:
      - /models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    environment:
      - CUDA_VISIBLE_DEVICES=0

  mistral:
    image: ghcr.io/ggerganov/llama.cpp:server-cublas
    command: ["-m","/models/mistral/mistral-7b-instruct.Q5_K_M.gguf","--port","8002","--ctx-size","4096"]
    ports: ["8002:8002"]
    volumes:
      - /models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    environment:
      - CUDA_VISIBLE_DEVICES=1

Run:

docker compose up -d

Observe and tune

  • CPU/mem:
htop
  • GPU:
watch -n1 nvidia-smi
nvtop
  • Quick latency tests:
time curl -s http://127.0.0.1:8001/health
time curl -s http://127.0.0.1:8002/health

Adjust context sizes, quantization levels (GGUF Q4/Q5), threads, and GPU layer offload to fit your hardware and concurrency goals.

Real-world patterns

  • Team split: Port 11434 (Llama 3) for general chat; Port 11435 (Code LLM) for devs. Route via Nginx paths or subdomains.

  • Batch vs. interactive: Run vLLM for batch jobs on GPU 1; keep Ollama snappy on GPU 0 for interactive queries.

  • A/B testing: Mirror the same prompts to both ports; log latency and response quality; pick a winner.

  • Edge node: Run two small quantized models on CPU-only llama.cpp across distinct cores for ultra-portable demos.

Conclusion and next steps

Running multiple LLMs on one Linux machine is about three things: a simple runtime, solid isolation, and predictable startup. Start with Ollama for the quickest wins, move to llama.cpp or vLLM when you need tighter control or higher throughput, and put Nginx in front when you want one neat URL.

Your next step:

  • Pick Option A (Ollama) or B (llama.cpp) above, launch two models on different ports, and route via Nginx.

  • Then add CPU pinning and per-process memory caps to keep everything stable under load.

If you want a follow-up, ask for a ready-to-drop systemd unit pack or a Docker Compose stack targeting your exact GPU and models.

Related posts