Posted on
• Artificial Intelligence

Deploying Open Source Artificial Intelligence at Scale

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Deploying Open Source Artificial Intelligence at Scale: A Bash-Friendly Playbook

OpenAI-like APIs are magical—until the bill arrives or your data policy says “no external services.” Open source AI lets you own the stack, control costs, and run models where your data lives. The challenge: going from “demo on a laptop” to “reliable, observable, horizontally scalable service.” This post gives you a practical, bash-first path to deploy open source AI at scale on Linux, with clear installation steps for apt, dnf, and zypper.

Why this matters now

  • Open models have matured: you can fine-tune and deploy capable LLMs, vision models, and embeddings with permissive licenses.

  • Container and orchestration ecosystems are battle-tested: Podman/Docker, systemd, and Kubernetes make AI services reproducible.

  • Throughput-focused serving stacks (vLLM, text-generation-inference, llama.cpp, Triton) deliver real performance gains.

  • Cost, data residency, and privacy concerns push workloads on-prem or in your VPC.

What you’ll build

  • A standardized, containerized AI runtime (rootless Podman recommended)

  • Single-node serving with GPU (vLLM) or CPU (llama.cpp)

  • Horizontal scaling with a simple load balancer (Nginx)

  • Observability with Prometheus + Grafana (containers)

  • Reproducibility with Git LFS and offline model caching

1) Standardize your runtime

Install the basics used across examples: container engine, Python tooling, Git LFS, curl, and jq.

# Debian/Ubuntu
sudo apt update
sudo apt install -y podman python3 python3-pip python3-venv git-lfs curl jq

# Fedora/RHEL/CentOS 8+
sudo dnf install -y podman python3 python3-pip git-lfs curl jq

# openSUSE/SLE
sudo zypper refresh
sudo zypper install -y podman python3 python3-pip git-lfs curl jq

Tip:

  • On RHEL/CentOS you may need EPEL for git-lfs: sudo dnf install -y epel-release && sudo dnf install -y git-lfs

  • Rootless Podman is secure by default and integrates cleanly with systemd for auto-restarts.

Optional: NVIDIA GPU support via NVIDIA Container Toolkit.

# NVIDIA Container Toolkit repo + install

# Debian/Ubuntu
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
     | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
     | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
     | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null \
  && sudo apt update \
  && sudo apt install -y nvidia-container-toolkit

# Fedora/RHEL
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
     | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo \
  && sudo dnf clean expire-cache \
  && sudo dnf install -y nvidia-container-toolkit

# openSUSE/SLE
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
     | sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo \
  && sudo zypper refresh \
  && sudo zypper install -y nvidia-container-toolkit

Configure the runtime and restart Podman:

sudo nvidia-ctk runtime configure --runtime=crun
systemctl --user daemon-reload || true
systemctl daemon-reload || true

Note: Ensure NVIDIA drivers are installed and nvidia-smi works before containerizing.

2) Serve a model on a single node

Option A: GPU-accelerated LLM with vLLM (OpenAI-compatible API, high throughput via continuous batching).

# Pull and run vLLM (GPU). Requires a compatible NVIDIA GPU.
podman run --rm --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGINGFACE_HUB_CACHE=/root/.cache/huggingface \
  docker.io/vllm/vllm-openai:latest \
  --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --dtype float16 --max-model-len 2048

Test the endpoint:

curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","prompt":"Write a Bash one-liner to count lines in all .log files","max_tokens":128}'

Option B: CPU-friendly serving with llama.cpp (GGUF quantized models, modest RAM/CPU).

# Fetch a quantized TinyLlama model (uses Git LFS)
mkdir -p ~/models && cd ~/models
git lfs install
git clone https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF
ls TinyLlama-1.1B-Chat-v1.0-GGUF | grep '\.gguf$'

Pick a Q4_K_M file from the list above, then:

# Serve on CPU (port 8000) with llama.cpp
podman run --rm -p 8000:8080 \
  -v ~/models/TinyLlama-1.1B-Chat-v1.0-GGUF:/models:ro \
  ghcr.io/ggerganov/llama.cpp:server \
  -m /models/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf -c 2048 -ngl 0

Test:

curl -s http://localhost:8000/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Summarize rsync in one sentence.","n_predict":96}'

Notes:

  • Replace the model with one that fits your hardware and license needs.

  • For privacy or offline installs, prefetch models into the mounted cache or volume.

3) Scale out horizontally with a simple load balancer

Run multiple instances across machines or ports, then front them with Nginx.

Install Nginx:

# Debian/Ubuntu
sudo apt update && sudo apt install -y nginx

# Fedora/RHEL/CentOS 8+
sudo dnf install -y nginx

# openSUSE/SLE
sudo zypper refresh && sudo zypper install -y nginx

Example Nginx config (/etc/nginx/conf.d/llm.conf):

upstream llm_backend {
  server 10.0.0.11:8000;
  server 10.0.0.12:8000;
  server 10.0.0.13:8000;
  keepalive 64;
}

server {
  listen 80;
  location / {
    proxy_pass http://llm_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_read_timeout 300s;
    client_max_body_size 20m;
  }
}

Enable and test:

sudo nginx -t && sudo systemctl enable --now nginx
curl -s http://<nginx-host>/v1/models || true

Tips for scale:

  • Start each backend with identical settings and a local Hugging Face cache (use fast storage).

  • Pin CPU/GPU resources per instance for predictability.

  • Use keepalive and HTTP/1.1 to reuse connections for streaming tokens.

Optional: make services persistent with systemd (rootless Podman shown):

# Run once per user to allow lingering services
loginctl enable-linger "$USER"

# Example: start a named container and autogenerate a systemd unit
podman run -d --name vllm-1 --gpus all -p 8000:8000 docker.io/vllm/vllm-openai:latest --model TinyLlama/TinyLlama-1.1B-Chat-v1.0
podman generate systemd --name vllm-1 --files --new
mkdir -p ~/.config/systemd/user
mv container-vllm-1.service ~/.config/systemd/user/
systemctl --user enable --now container-vllm-1.service

4) Add observability (Prometheus + Grafana)

Expose host metrics with Node Exporter and scrape them with Prometheus. Run both as containers to avoid distro package variance.

Minimal Prometheus config:

mkdir -p ~/monitoring
cat > ~/monitoring/prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']  # add more nodes here
EOF

Run Node Exporter and Prometheus:

# Node Exporter (host metrics)
podman run -d --name node-exporter --restart=always \
  --net=host --pid=host --uts=host \
  -e TZ=$(cat /etc/timezone 2>/dev/null || echo UTC) \
  quay.io/prometheus/node-exporter:latest

# Prometheus
podman run -d --name prometheus --restart=always \
  -p 9090:9090 \
  -v ~/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro \
  docker.io/prom/prometheus:latest

Add Grafana:

podman run -d --name grafana --restart=always \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD='admin' \
  docker.io/grafana/grafana-oss:latest

Then:

  • Visit Grafana at http://localhost:3000 (admin/admin)

  • Add Prometheus data source: http://prometheus:9090 or http://localhost:9090

  • Import dashboards for Node Exporter and application metrics (or add custom endpoints in your servers to expose QPS/latency).

5) Make it reproducible: models, versions, and CI/CD

  • Use Git LFS for large files and model artifacts.
git lfs install
git lfs track "*.gguf" "*.safetensors"
git add .gitattributes
git add -A && git commit -m "Track large model artifacts"

  • Cache models offline for deterministic builds (bake into images if needed).

  • Pin container image tags and model SHAs.

  • If you fine-tune:

    • Store checkpoints and metadata (MLflow via pip install mlflow).
    • Record the exact training/inference Containerfile and environment variables.

Example: bake a model into an image for air-gapped deploys.

# Containerfile
FROM docker.io/vllm/vllm-openai:latest
ENV HUGGINGFACE_HUB_CACHE=/root/.cache/huggingface
RUN --mount=type=cache,target=/root/.cache/huggingface \
    python3 - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(repo_id="TinyLlama/TinyLlama-1.1B-Chat-v1.0", local_dir="/models", local_dir_use_symlinks=False)
PY
CMD ["--model", "/models"]

Build and run:

podman build -t vllm-tinyllama:cached .
podman run --rm --gpus all -p 8000:8000 vllm-tinyllama:cached

Real-world guidance and pitfalls

  • Pick the right serving stack:

    • vLLM for GPU LLMs and high throughput; text-generation-inference is another strong option.
    • llama.cpp for CPU-friendly, memory-efficient serving of quantized models.
    • NVIDIA Triton for multi-framework (TensorRT, PyTorch, ONNX) and ensemble pipelines.

  • Throughput over single-request latency:

    • Prefer continuous batching (vLLM) and longer context windows only when needed.
    • Quantize when quality allows (GGUF Q4/Q5, AWQ/GPTQ) to fit memory and boost speed.

  • Storage and I/O matter:

    • Put model files and HF cache on fast local NVMe.
    • Warm your cache at boot to avoid first-request stalls.

  • Security and governance:

    • Lock down Nginx with TLS and IP allowlists.
    • Scan images (Trivy/Clair) and use signed images (cosign).
    • Verify model licenses for your use case.

Where to go next

  • Start small: deploy a single-node server with vLLM or llama.cpp and put Nginx in front.

  • Add two more nodes, point Nginx at all three, and measure QPS/latency with hey or wrk.

  • Wire up Prometheus + Grafana and set SLOs (e.g., p95 latency under 1.5s).

  • Graduate to orchestrators when ready:

    • Kubernetes + KServe/Seldon Core for canaries and autoscaling
    • Ray Serve for Python-native model composition and scheduling

If you want a ready-to-adapt reference, tell me your distro, GPU/CPU resources, and target model; I’ll generate a bash-first deployment script tailored to your environment.

Related posts