Posted on
• Artificial Intelligence

Artificial Intelligence Continuous Delivery

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Continuous Delivery: Ship Models Like Code (From Your Linux Shell)

What if shipping a new model felt as boring—and safe—as shipping a bug fix? Most AI teams still push notebooks by hand, wrangle data in ad‑hoc folders, and “deploy” by copying files to servers. That’s brittle, slow, and impossible to audit. The value of Artificial Intelligence Continuous Delivery (AI CD) is turning ML into a repeatable, testable, and automated delivery system you can trust.

This guide shows you how to stand up AI CD on Linux using nothing but your terminal. You’ll learn why AI CD matters and walk away with a concrete, bash-first blueprint you can implement this week.

Why AI Continuous Delivery?

  • Reproducibility: You can rebuild the same model binary and environment—bit for bit—months later.

  • Safety: Automated tests, evals, and policy gates catch regressions before production.

  • Speed: Small, frequent, automated releases ship value faster and reduce risk.

  • Compliance: Versioned data, models, metrics, and infra are auditable by default.

  • Cost: Early failures in CI are cheaper than late failures in production.

Prerequisites: Install Core Tools

Pick Docker or Podman for container builds. The rest is common.

  • Common packages: git, Python 3, pip, venv, Git LFS, make, curl

  • Container runtime: Docker or Podman (choose one)

  • Optional for Kubernetes CD: kubectl

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip git-lfs make curl

# Docker (option A)
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"   # log out/in to apply

# Podman (option B)
sudo apt install -y podman buildah skopeo

# Optional: kubectl (on some distros this may be kubernetes-client)
sudo apt install -y kubectl || sudo apt install -y kubernetes-client

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y git python3 python3-pip git-lfs make curl

# Docker (option A, Fedora provides moby-engine)
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

# Podman (option B)
sudo dnf install -y podman buildah skopeo

# Optional: kubectl (kubernetes-client provides kubectl)
sudo dnf install -y kubernetes-client

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y git python3 python3-pip git-lfs make curl

# venv is typically included with python3; if missing try:
# sudo zypper install -y python3-venv

# Docker (option A)
sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

# Podman (option B)
sudo zypper install -y podman buildah skopeo

# Optional: kubectl (package may be kubectl or kubernetes-client)
sudo zypper install -y kubectl || sudo zypper install -y kubernetes-client

If python3 -m venv fails on your distro, either install python3-venv (apt) or use pip install virtualenv as a fallback.

A Minimal, Practical AI CD Blueprint (5 Steps)

Below is a concrete path you can copy-paste on a fresh repo. It’s tool-agnostic, shell-first, and production-friendly.

1) Make builds reproducible

Pin your environment and automate repeatable commands.

Repo layout:

ai-cd-demo/
├─ data/                 # DVC will track pointers, not blobs
├─ models/               # Model artifacts (via DVC)
├─ src/
│  ├─ train.py
│  └─ evaluate.py
├─ server/
│  └─ app.py             # FastAPI inference server
├─ tests/
│  └─ test_eval.py
├─ requirements.txt
├─ dvc.yaml
├─ params.yaml
└─ Makefile

Example requirements.txt:

fastapi==0.111.0
uvicorn[standard]==0.30.1
scikit-learn==1.5.0
pandas==2.2.2
numpy==1.26.4
joblib==1.4.2
dvc[s3]==3.51.0
pytest==8.2.2

Create and use a venv:

python3 -m venv .venv
. .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

Makefile for muscle memory:

.PHONY: venv train eval serve test

venv:
    python3 -m venv .venv && . .venv/bin/activate && pip install -U pip -r requirements.txt

train:
    . .venv/bin/activate && python -m src.train

eval:
    . .venv/bin/activate && python -m src.evaluate

serve:
    . .venv/bin/activate && uvicorn server.app:app --host 0.0.0.0 --port 8000

test:
    . .venv/bin/activate && pytest -q

Minimal src/train.py:

# src/train.py
import joblib, pandas as pd
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

def main():
    X, y = load_iris(return_X_y=True, as_frame=True)
    clf = LogisticRegression(max_iter=200).fit(X, y)
    joblib.dump(clf, "models/iris.joblib")
    print("Saved models/iris.joblib")

if __name__ == "__main__":
    main()

Minimal src/evaluate.py:

# src/evaluate.py
import json, joblib
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

def main():
    X, y = load_iris(return_X_y=True, as_frame=True)
    clf = joblib.load("models/iris.joblib")
    acc = accuracy_score(y, clf.predict(X))
    print(json.dumps({"accuracy": acc}))

if __name__ == "__main__":
    main()

2) Version everything: code, data, models, metrics

Use Git for code, Git LFS for large binaries, and DVC for data/model artifacts and pipelines.

Initialize:

git init
git lfs install
dvc init
git add .dvc .gitignore
git commit -m "init dvc"

Track data and model artifacts:

mkdir -p data models
# Add your raw data into data/, then:
dvc add data
git add data.dvc .gitignore
git commit -m "track data with dvc"

Configure a DVC remote (S3 shown; you can use local, SSH, GCS, etc.):

dvc remote add -d origin s3://my-bucket/ai-cd-demo
dvc remote modify origin access_key_id "$AWS_ACCESS_KEY_ID"
dvc remote modify origin secret_access_key "$AWS_SECRET_ACCESS_KEY"
dvc remote modify origin endpointurl "https://s3.your-cloud.example"
git commit .dvc/config -m "configure dvc remote"

Define a simple DVC pipeline in dvc.yaml:

stages:
  train:
    cmd: python -m src.train
    deps:
      - src/train.py
      - data
    outs:
      - models/iris.joblib
  evaluate:
    cmd: python -m src.evaluate > metrics.json
    deps:
      - src/evaluate.py
      - models/iris.joblib
    metrics:
      - metrics.json:
          cache: false

Run, then push artifacts:

dvc repro
dvc push
git add models/.gitignore dvc.yaml metrics.json
git commit -m "train + eval with dvc"

Now your code, data pointers, models, and metrics are versioned together.

3) Automate tests and evals in CI

Gate releases on metrics (e.g., accuracy) and tests.

Example tests/test_eval.py:

import json, subprocess

def test_accuracy_gate():
    out = subprocess.check_output(["python", "-m", "src.evaluate"])
    acc = json.loads(out)["accuracy"]
    assert acc >= 0.90, f"Accuracy too low: {acc}"

GitHub Actions workflow .github/workflows/ci.yml:

name: ci
on:
  push:
    branches: [ "main" ]
  pull_request:

jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install deps
        run: |
          python -m pip install -U pip
          pip install -r requirements.txt
      - name: DVC pull data/models
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          DVC_NO_ANALYTICS: "1"
        run: |
          dvc pull || true  # pull if available
      - name: Train + Eval
        run: |
          python -m src.train
          python -m src.evaluate > metrics.json
      - name: Unit tests + metric gates
        run: |
          pytest -q
      - name: Upload metrics
        uses: actions/upload-artifact@v4
        with:
          name: metrics
          path: metrics.json

This fails fast if accuracy regresses, preventing promotion.

4) Containerize inference and promote via environments

A tiny FastAPI server server/app.py:

from fastapi import FastAPI
import joblib
from pydantic import BaseModel

app = FastAPI()
model = joblib.load("models/iris.joblib")

class Sample(BaseModel):
    features: list[float]  # length 4 for iris

@app.post("/predict")
def predict(s: Sample):
    return {"class": int(model.predict([s.features])[0])}

Dockerfile (works with Docker or Podman):

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -U pip && pip install -r requirements.txt
COPY models/ models/
COPY server/ server/
EXPOSE 8000
CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"]

Build and run locally:

Docker:

docker build -t ghcr.io/youruser/ai-cd-demo:$(git rev-parse --short HEAD) .
docker run --rm -p 8000:8000 ghcr.io/youruser/ai-cd-demo:$(git rev-parse --short HEAD)

Podman:

podman build -t ghcr.io/youruser/ai-cd-demo:$(git rev-parse --short HEAD) .
podman run --rm -p 8000:8000 ghcr.io/youruser/ai-cd-demo:$(git rev-parse --short HEAD)

Test inference:

curl -s -X POST localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"features":[5.1,3.5,1.4,0.2]}'

Kubernetes deployment k8s/deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-cd-demo
spec:
  replicas: 2
  selector:
    matchLabels: { app: ai-cd-demo }
  template:
    metadata:
      labels: { app: ai-cd-demo }
    spec:
      containers:
      - name: app
        image: ghcr.io/youruser/ai-cd-demo:REPLACEME_SHA
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet: { path: /docs, port: 8000 }
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: ai-cd-demo
spec:
  selector: { app: ai-cd-demo }
  ports:
  - port: 80
    targetPort: 8000

Apply to a cluster:

kubectl apply -f k8s/deploy.yaml
kubectl rollout status deploy/ai-cd-demo

Promotion tip:

  • Use tags like :staging-<sha> and :prod-<sha>.

  • Promote by updating just the image tag and letting your CD tool (e.g., Argo CD, Flux) sync.

Optional Argo CD application (argocd/app.yaml):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ai-cd-demo
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/youruser/ai-cd-demo.git
    targetRevision: main
    path: k8s
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

5) Observe and roll back with metric gates

  • Metric gates: Keep the pytest threshold or parse metrics.json in CI to block bad models.

  • Health and latency: Add a /healthz endpoint and expose Prometheus metrics.

  • Rollback: Use kubectl rollout undo deploy/ai-cd-demo or revert the tag in Git; Argo CD/Flux will reconcile automatically.

Example gate in bash (CI step) to block promotion if accuracy < 0.92:

ACC=$(jq -r .accuracy metrics.json)
python - <<PY
import sys
acc = float("$ACC")
sys.exit(0 if acc >= 0.92 else 1)
PY

Real-World Patterns That Work

  • Shadow deployments: Mirror a slice of prod traffic to the new model; compare outputs offline before switching.

  • Blue/Green: Run both versions; flip Service selector when the new one passes SLOs.

  • Feature flags: Gate model usage by tenant or percentage rollouts.

  • Immutable artifacts: Image tags are content-addressed (use commit SHA); never reuse mutable tags for promotion.

Conclusion and Next Steps

AI CD makes ML boring—in the best way. You version everything, test automatically, gate on metrics, ship in containers, and promote with confidence.

Your next steps: 1) Initialize DVC in your repo and pin your Python environment. 2) Add a simple pytest gate that enforces a minimum metric. 3) Containerize your inference server and deploy to a staging namespace. 4) Wire an automated pipeline to build, test, and deploy on every merge.

If you found this useful, pick one step, implement it today, and iterate. Consistent, incremental automation beats big-bang “MLOps transformations” every time.

Related posts