Posted on
• Artificial Intelligence

Artificial Intelligence Web Deployment

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Web Deployment on Linux: From Notebook to Production

You’ve got a model that works in a notebook. Now what? The real challenge begins when you try to deliver AI to real users: reliable, fast, secure, and reproducible. This guide shows you how to deploy an AI inference API on a Linux server using Bash-friendly tooling, with clear, copy-paste commands for apt, dnf, and zypper.

What you’ll get:

  • A minimal, production-grade FastAPI service for inference

  • Systemd and Nginx setup for durability and reverse-proxying

  • Optional containerization with Podman or Docker

  • Practical steps you can adapt to your own models

Why this matters:

  • Speed and reliability: API + ASGI server outperforms ad-hoc scripts

  • Reproducibility: pinned environments and container images

  • Security: unprivileged service user behind a reverse proxy

  • Scalability: workers and statelessness make horizontal scaling possible

1) Prerequisites and Project Skeleton

Pick a clean Linux host (VM or bare-metal). We’ll install Python, Nginx, Git, and build tools. Then we’ll scaffold a small project.

Install prerequisites:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git python3 python3-pip python3-virtualenv nginx build-essential
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y git python3 python3-pip python3-virtualenv nginx gcc gcc-c++ make
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y git python3 python3-pip python3-virtualenv nginx gcc gcc-c++ make

Create a service user and directories:

sudo useradd --system --create-home --shell /usr/sbin/nologin svc-ml || true
sudo mkdir -p /opt/ai-service
sudo chown svc-ml:svc-ml /opt/ai-service

Project files:

sudo -u svc-ml bash -lc '
cd /opt/ai-service
git init .
cat > requirements.txt <<EOF
fastapi
uvicorn[standard]
transformers
torch
EOF

cat > app.py <<EOF
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List
from transformers import pipeline

app = FastAPI(title="AI Inference API")

class Item(BaseModel):
    text: str

class Batch(BaseModel):
    texts: List[str]

# Lazy-load model on startup
@app.on_event("startup")
def load_model():
    app.state.nlp = pipeline(
        "sentiment-analysis",
        model="distilbert-base-uncased-finetuned-sst-2-english"
    )

@app.get("/healthz")
def health():
    return {"status": "ok"}

@app.post("/predict")
def predict(item: Item):
    out = app.state.nlp(item.text)[0]
    return {"label": out["label"], "score": float(out["score"])}

@app.post("/batch_predict")
def batch_predict(batch: Batch):
    out = app.state.nlp(batch.texts)
    return [{"label": r["label"], "score": float(r["score"])} for r in out]
EOF
'

Create and activate a virtual environment, then install dependencies:

sudo -u svc-ml bash -lc '
cd /opt/ai-service
python3 -m virtualenv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
'

Quick local test:

sudo -u svc-ml bash -lc '
cd /opt/ai-service
. .venv/bin/activate
uvicorn app:app --host 0.0.0.0 --port 8000 --workers 2
'

In another terminal:

curl -s http://127.0.0.1:8000/healthz
curl -s -X POST http://127.0.0.1:8000/predict -H "content-type: application/json" -d "{\"text\":\"I love Linux!\"}"

Tip: First run downloads the model. For faster cold-starts in production, pre-warm by calling /predict once during deploy or bake the model into a container image.

2) Service Management with systemd (Keep It Running)

Create a systemd unit so your API restarts on failure and starts at boot:

sudo tee /etc/systemd/system/ai-service.service >/dev/null <<'EOF'
[Unit]
Description=AI Inference API (FastAPI + Uvicorn)
After=network.target

[Service]
User=svc-ml
Group=svc-ml
WorkingDirectory=/opt/ai-service
Environment=HF_HOME=/opt/ai-service/.cache/huggingface
ExecStart=/opt/ai-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000 --workers 2 --timeout-keep-alive 30
Restart=on-failure
RestartSec=5
# Harden a bit
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

sudo mkdir -p /opt/ai-service/.cache/huggingface
sudo chown -R svc-ml:svc-ml /opt/ai-service/.cache

sudo systemctl daemon-reload
sudo systemctl enable --now ai-service
sudo systemctl status ai-service --no-pager

Check:

curl -s http://127.0.0.1:8000/healthz

3) Put Nginx in Front (Security, TLS, and Performance)

Nginx acts as a reverse proxy, adds timeouts, and simplifies TLS. You installed Nginx above; ensure it’s running:

  • Debian/Ubuntu (apt):
sudo systemctl enable --now nginx
  • Fedora/RHEL/CentOS Stream (dnf):
sudo systemctl enable --now nginx
  • openSUSE/SLE (zypper):
sudo systemctl enable --now nginx

Create a site config (replace example.com with your domain or server IP):

sudo tee /etc/nginx/conf.d/ai-service.conf >/dev/null <<'EOF'
server {
    listen 80;
    server_name example.com;

    # Increase timeouts for model inference if needed
    proxy_read_timeout 120s;
    proxy_connect_timeout 5s;
    proxy_send_timeout 120s;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /healthz {
        proxy_pass http://127.0.0.1:8000/healthz;
    }
}
EOF

sudo nginx -t
sudo systemctl reload nginx

Optional TLS:

  • Obtain a certificate with Let’s Encrypt (certbot or your preferred method).

  • Terminate TLS in Nginx and keep Uvicorn on localhost.

Firewall note:

  • Debian/Ubuntu (ufw):
sudo ufw allow "Nginx Full"
sudo ufw enable
  • Fedora/RHEL/CentOS (firewalld):
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
  • openSUSE (firewalld):
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

4) Optional: Containerize with Podman or Docker

Containers improve portability and rollback.

Install a container engine:

  • Podman

    • apt:
    sudo apt update
sudo apt install -y podman
    • dnf:
    sudo dnf install -y podman
    • zypper:
    sudo zypper refresh
sudo zypper install -y podman

  • Docker (package names vary by distro)

    • apt (Ubuntu/Debian):
    sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker
    • dnf (Fedora):
    # Moby (upstream Docker) in Fedora repos
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
    • zypper (openSUSE):
    sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker

Create Dockerfile:

sudo -u svc-ml bash -lc '
cd /opt/ai-service
cat > Dockerfile <<EOF
FROM python:3.11-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    HF_HOME=/root/.cache/huggingface

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000
CMD ["uvicorn", "app:app", "--host","0.0.0.0","--port","8000","--workers","2","--timeout-keep-alive","30"]
EOF
'

Build and run with Podman:

cd /opt/ai-service
sudo podman build -t ai-svc:latest .
sudo podman run --rm -p 8000:8000 ai-svc:latest

Or with Docker:

cd /opt/ai-service
sudo docker build -t ai-svc:latest .
sudo docker run --rm -p 8000:8000 ai-svc:latest

Test:

curl -s http://127.0.0.1:8000/healthz

Tip: For faster cold starts, pre-pull models during build:

RUN python -c "from transformers import pipeline; pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')('warmup')"

5) Scale, Observe, and Optimize

  • Concurrency and workers:

    • Start with workers = CPU cores or cores/2 and tune empirically.
    • Batch requests when possible (see /batch_predict).

  • Caching:

    • Keep HF_HOME on persistent storage to avoid re-downloading models.
    • Reuse models across workers if memory allows (container memory limits apply).

  • Logging and metrics:

    • systemd logs:
    journalctl -u ai-service -f
    • Basic timing in app endpoints; export counters to Prometheus if needed.

  • Security basics:

    • Run as unprivileged user (done).
    • Keep API behind Nginx; lock inbound ports (only 80/443).
    • Store secrets in environment files (not code).

Real-world example patterns:

  • CPU-only inference on small models (as shown) for cost efficiency.

  • GPU-backed nodes for heavy models; containerize and schedule with Kubernetes/Nomad.

  • Canary deploys: run new version on a second port, shift Nginx traffic gradually.

Troubleshooting Cheatsheet

  • venv issues: We used virtualenv to avoid distro-specific python3-venv packages. If you prefer venv and it fails, install python3-venv (apt) or your distro’s pythonX-venv package.

  • OOM or slow cold-starts: Use smaller models, increase instance memory, or warm the cache.

  • 502 from Nginx: Check journalctl -u ai-service, increase proxy_read_timeout, or add more workers.

  • Permission errors writing model cache: Ensure HF_HOME directory is owned by your service user.

Conclusion and Next Steps

You’ve taken an AI model from “works on my laptop” to a production-grade Linux service:

  • FastAPI + Uvicorn inference API

  • Managed by systemd

  • Fronted by Nginx

  • Optionally containerized with Podman or Docker

Next step: adapt the /predict logic to your own model, add input validation and telemetry, and automate deployment with a simple Bash script or CI pipeline. When you’re ready to scale further, place this behind a load balancer and add health checks.

Want a follow-up? Ask for:

  • A GPU-ready Dockerfile and deployment checklist

  • A zero-downtime rolling deploy Bash script

  • Observability with Prometheus + Grafana on Linux

Related posts