Posted on
• Artificial Intelligence

Artificial Intelligence Server Deployment

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Server Deployment on Linux: From Zero to Inference

You’ve built an AI prototype that dazzles on your laptop—but the moment you try to put it behind an API for real users, reality sets in: inconsistent environments, missing drivers, resource contention, and a dozen “works on my machine” gotchas. This guide shows you how to deploy a reliable, reproducible AI inference server on Linux using battle-tested tools and Bash-friendly steps you can paste straight into the terminal.

We’ll cover why AI server deployment matters, walk through a practical CPU-first setup (with optional GPU acceleration), harden it with a reverse proxy and firewall, and finish with a systemd service so it all survives reboots. All installation snippets include apt, dnf, and zypper where cited.

Why this matters

  • Reliability beats demos: Reproducible builds and services ensure your model works the same way on every restart.

  • Latency and throughput: Proper serving and batching can cut costs significantly and keep users happy.

  • Compliance and control: On-prem or private-cloud serving means you control data and costs.

  • Operability: Logging, health checks, and systemd mean fewer 3 a.m. surprises.

What we’ll build (at a glance)

  • A dedicated Linux user and workspace for the AI server

  • A CPU-friendly LLM server (llama.cpp) exposing an OpenAI-compatible API

  • Optional GPU acceleration flags if your machine has NVIDIA or AMD GPUs

  • A reverse proxy (Nginx) that terminates TLS and protects your backend port

  • A systemd unit to auto-start and auto-recover your AI server

  • Firewall rules to expose only what’s needed

1) Plan your workload and serving strategy

Before you install anything, answer these:

  • Latency vs throughput: Low-latency chats? Consider smaller models or GPU. High-throughput batch scoring? Consider batching and parallelism settings.

  • CPU vs GPU: CPU is simplest and often good for small/medium models or low QPS; GPU shines for large LLMs or vision at scale.

  • Memory budget: Models must fit in RAM/VRAM plus overhead. Quantized GGUF models can run well on CPUs with low memory.

We’ll start CPU-first (works on almost any Linux box), then show how to flip on GPU acceleration later.

2) Prepare the host: packages, user, and firewall

Install base tools (compilers, Python, Nginx, etc.). Pick the command set for your distro.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y build-essential cmake git curl wget python3 python3-pip python3-venv nginx ufw
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake git curl wget python3 python3-pip nginx firewalld
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y -t pattern devel_basis
sudo zypper install -y cmake git curl wget python3 python3-pip nginx firewalld

Optional (faster CPU math via OpenBLAS):

  • apt:
sudo apt install -y libopenblas-dev
  • dnf:
sudo dnf install -y openblas-devel
  • zypper:
sudo zypper install -y openblas-devel

Create a dedicated user and workspace:

sudo useradd -r -m -d /opt/ai -s /bin/bash ai
sudo mkdir -p /opt/ai/models
sudo chown -R ai:ai /opt/ai

Open only the ports you need (choose one firewall family):

  • UFW (Debian/Ubuntu):
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# We'll proxy the backend locally, so keep 8000 internal. If you must expose it:
# sudo ufw allow 8000/tcp
sudo ufw enable
sudo ufw status verbose
  • firewalld (Fedora/openSUSE/RHEL/CentOS):
sudo systemctl enable --now firewalld
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
# Keep backend internal; expose 8000 only if needed:
# sudo firewall-cmd --permanent --add-port=8000/tcp
sudo firewall-cmd --reload

3) Real-world example (CPU): LLM server with llama.cpp

llama.cpp is a fast, portable inference engine for LLMs. Its server binary exposes an OpenAI-compatible HTTP API, so your existing clients can usually connect with zero code changes.

Build llama.cpp:

sudo -iu ai
cd /opt/ai
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Basic CPU build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# Optional: enable BLAS for extra CPU speed (requires OpenBLAS dev pkg, see above)
# cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
# cmake --build build -j

Download a small, permissively available GGUF model (TinyLlama 1.1B Chat). This works without special credentials and is great for testing:

mkdir -p /opt/ai/models
cd /opt/ai/models
curl -L -o TinyLlama-1.1B-Chat.Q4_K_M.gguf \
  https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf

Start the server (run in a terminal first to verify):

cd /opt/ai/llama.cpp
./build/bin/server \
  -m /opt/ai/models/TinyLlama-1.1B-Chat.Q4_K_M.gguf \
  --host 127.0.0.1 --port 8000 \
  -c 2048 -ngl 0

  • -ngl 0 keeps layers on CPU (safest default).

  • The server exposes OpenAI-like endpoints at http://127.0.0.1:8000/v1

Test with curl:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "model":"TinyLlama-1.1B-Chat.Q4_K_M.gguf",
        "messages":[{"role":"user","content":"In one sentence, what is Linux?"}],
        "max_tokens":64
      }' | jq .

You should see JSON with a generated message. If jq isn’t installed, remove the pipe.

4) Put it behind Nginx (TLS-ready and production-friendly)

Install Nginx (already installed above). Create a site config that proxies /v1 to the backend and keeps the raw Authorization header intact (important for OpenAI-compatible clients):

sudo bash -c 'cat >/etc/nginx/conf.d/ai.conf' <<'EOF'
upstream ai_backend {
    server 127.0.0.1:8000;
    keepalive 16;
}

server {
    listen 80;
    server_name _;

    # Increase timeouts for long generations
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;

    location /v1/ {
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Connection "";
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;
        proxy_pass http://ai_backend;
    }

    # Optional simple health check
    location /healthz {
        return 200 "ok\n";
        add_header Content-Type text/plain;
    }
}
EOF
sudo nginx -t
sudo systemctl reload nginx

Optional TLS with Let’s Encrypt (ensure DNS points to this host):

  • apt:
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx
  • dnf:
sudo dnf install -y certbot python3-certbot-nginx
sudo certbot --nginx
  • zypper:
sudo zypper install -y certbot python3-certbot-nginx
sudo certbot --nginx

After this, clients can talk to your AI server through Nginx on port 443. Keep the backend on 127.0.0.1:8000 for safety.

5) Make it a service (systemd)

Create a systemd unit so the server starts at boot and auto-recovers:

sudo bash -c 'cat >/etc/systemd/system/llama-server.service' <<'EOF'
[Unit]
Description=llama.cpp AI Server
Wants=network-online.target
After=network-online.target

[Service]
User=ai
Group=ai
WorkingDirectory=/opt/ai/llama.cpp
ExecStart=/opt/ai/llama.cpp/build/bin/server \
  -m /opt/ai/models/TinyLlama-1.1B-Chat.Q4_K_M.gguf \
  --host 127.0.0.1 --port 8000 \
  -c 2048 -ngl 0
Restart=always
RestartSec=3
# Slow model loads can take time; extend start timeout if needed
TimeoutStartSec=600
# Uncomment to constrain CPU or memory:
# MemoryMax=8G
# CPUQuota=200%

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
sudo systemctl status llama-server --no-pager

Check logs:

journalctl -u llama-server -f

Smoke test through Nginx:

curl -s https://YOUR_DOMAIN/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"TinyLlama-1.1B-Chat.Q4_K_M.gguf","messages":[{"role":"user","content":"Say hi!"}],"max_tokens":32}'

Optional: Turn on GPU acceleration later

If your server has a supported GPU:

  • Install vendor drivers and runtime:

    • NVIDIA: Install proprietary driver and CUDA toolkit (matching driver and libcudart). Follow NVIDIA’s official docs for your distro.
    • AMD: Install ROCm per AMD’s documentation.

  • Rebuild llama.cpp with the correct backend:

    • NVIDIA CUDA:
    cd /opt/ai/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j
    • AMD ROCm (HIPBLAS):
    cd /opt/ai/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIPBLAS=ON
cmake --build build -j

  • Then start the server with GPU layers enabled (e.g., -ngl 999 to offload as many layers as possible):

    ./build/bin/server -m /opt/ai/models/TinyLlama-1.1B-Chat.Q4_K_M.gguf --host 127.0.0.1 --port 8000 -c 2048 -ngl 999

    Adjust model size for available VRAM.

Tip: For larger deployments and multi-model serving, consider specialized servers like vLLM (OpenAI-compatible LLM serving with batching) or NVIDIA Triton (multi-framework model repo). These typically run best in containers with GPU runtime configured.

Operations checklist (quick wins)

  • Health checks: Nginx /healthz; also consider periodic curl checks from cron or a monitoring system.

  • Metrics: Node Exporter and process-level dashboards can help track CPU/RAM saturation. llama.cpp logs tokens/sec—scrape and alert.

  • Logging: journalctl plus Nginx access/error logs provide request-level visibility.

  • Rollouts: Keep binaries and models versioned (e.g., in /opt/ai/releases). Switch systemd ExecStart to roll forward/back safely.

  • Security: Bind backend to 127.0.0.1, restrict ports, use TLS, and consider an API key or mTLS at Nginx for private deployments.

Conclusion and next steps

You now have a reproducible, service-managed AI inference server that speaks an OpenAI-compatible API, sits safely behind Nginx, and starts on boot. From here:

  • Swap in a model that fits your use case (code generation, QA, RAG with embeddings).

  • Tune performance: enable BLAS, adjust context size and threads, or turn on GPU acceleration.

  • Harden: add authentication at Nginx, enable TLS auto-renew via certbot, and set resource limits in systemd.

If you want a follow-up post, ask for:

  • Adding RAG (Retrieval-Augmented Generation) with a vector DB

  • Deploying a GPU-accelerated vLLM server with batching

  • CI/CD for models and blue/green rollouts

Happy shipping!

Related posts