Posted on
• Artificial Intelligence

Artificial Intelligence CI/CD Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Ship AI Like Software: Practical CI/CD for Machine Learning on Linux (with Bash)

Modern ML teams lose time hand-running notebooks, babysitting training jobs, and “fixing it in prod.” The fix is not more heroics—it’s CI/CD built for AI. With a few Linux-friendly tools and some Bash glue, you can turn your model code, data, and containers into a reliable pipeline that ships value continuously.

This article explains why CI/CD is essential for AI projects, then walks you through a practical, Linux-first setup you can copy. You’ll get actionable steps, sample scripts, and distro-specific install commands (apt, dnf, zypper).

Why CI/CD matters for AI (and how it’s different)

  • Reproducibility is fragile: Tiny environment changes or unchecked randomness derail experiments.

  • Data is part of the product: Code alone isn’t enough—version the data and model artifacts too.

  • Hardware constraints: You can’t run full GPU training on every push—split fast checks from heavy jobs.

  • Governance/compliance: You’ll need audit trails for models, dependencies, and images.

Good CI/CD bakes in reproducibility, fast feedback, artifact tracking, and safe delivery—without slowing you down.

Prerequisites: Linux packages you’ll actually use

Install Git, Python, venv/pip, Make, and Docker (or Podman). Pick your distro:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip make docker.io podman pipx
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
newgrp docker
pipx ensurepath

Fedora/RHEL (dnf):

sudo dnf install -y git python3 python3-pip python3-virtualenv make docker podman pipx
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
newgrp docker
pipx ensurepath

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git python3 python3-pip python3-virtualenv make docker podman pipx
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
newgrp docker
pipx ensurepath

Optional user-space tools (keeps your system clean):

pipx install pre-commit
pipx install pip-audit
pipx install "dvc[s3]"

Project venv for Python deps:

python3 -m venv .venv
. .venv/bin/activate
pip install -U pip wheel
pip install -U pytest fastapi uvicorn[standard] scikit-learn

A simple, reproducible project layout

ai-cicd-demo/
├─ src/
│  ├─ train.py
│  └─ serve.py
├─ tests/
│  └─ test_infer.py
├─ data/              # large/raw data (DVC manages, not Git)
├─ models/            # trained artifacts (DVC manages)
├─ dvc.yaml
├─ requirements.txt
├─ Makefile
├─ Dockerfile
└─ .github/workflows/ci.yml

requirements.txt:

fastapi
uvicorn[standard]
scikit-learn
pytest
joblib

Minimal training (src/train.py):

#!/usr/bin/env python3
import json, os, joblib
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import numpy as np

RANDOM_SEED = int(os.environ.get("SEED", 42))
np.random.seed(RANDOM_SEED)

def main():
    X, y = load_iris(return_X_y=True, as_frame=False)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=RANDOM_SEED)
    clf = RandomForestClassifier(n_estimators=50, random_state=RANDOM_SEED)
    clf.fit(Xtr, ytr)
    preds = clf.predict(Xte)
    acc = accuracy_score(yte, preds)
    print(f"accuracy={acc:.4f}")
    os.makedirs("models", exist_ok=True)
    joblib.dump(clf, "models/model.joblib")
    with open("models/metrics.json", "w") as f:
        json.dump({"accuracy": acc}, f)
    # Contract: ensure minimal perf
    assert acc > 0.85, "Accuracy regression detected"

if __name__ == "__main__":
    main()

Tiny service (src/serve.py):

from fastapi import FastAPI
from pydantic import BaseModel, conlist
import joblib

app = FastAPI()
model = joblib.load("models/model.joblib")

class Item(BaseModel):
    features: conlist(float, min_items=4, max_items=4)

@app.post("/predict")
def predict(item: Item):
    y = model.predict([item.features])[0].item()
    return {"prediction": int(y)}

Unit test (tests/test_infer.py):

import joblib
import numpy as np
def test_model_contract():
    clf = joblib.load("models/model.joblib")
    x = np.array([[5.1, 3.5, 1.4, 0.2]])
    y = clf.predict(x)
    assert y.shape == (1,)

Makefile (developer ergonomics + CI parity):

SHELL := /bin/bash
.ONESHELL:
.SHELLFLAGS := -eu -o pipefail -c

PY := . .venv/bin/activate

venv:
    python3 -m venv .venv
    $(PY); pip install -U pip wheel
    $(PY); pip install -r requirements.txt

train:
    $(PY); python src/train.py

test:
    $(PY); pytest -q

serve:
    $(PY); uvicorn src.serve:app --host 0.0.0.0 --port 8000

build:
    docker build -t ai-cicd-demo:latest .

run:
    docker run --rm -p 8000:8000 ai-cicd-demo:latest

Dockerfile (reproducible runtime):

FROM python:3.11-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

RUN --mount=type=cache,target=/var/cache/apt \
    apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/ src/
COPY models/ models/   # CI will ensure model exists before build (or build trains)

EXPOSE 8000
CMD ["uvicorn", "src.serve:app", "--host", "0.0.0.0", "--port", "8000"]

DVC pipeline (dvc.yaml) to track artifacts:

stages:
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
    outs:
      - models/model.joblib
    metrics:
      - models/metrics.json:
          cache: false

Initialize DVC and Git (once):

git init
dvc init
git add .
git commit -m "Initial AI CI/CD demo"

5 actionable steps to production-grade AI CI/CD

1) Standardize environments with containers and lockfiles

  • Why: Reproducibility; “works on my machine” goes away.

  • How: Use requirements.txt (or pip-tools/uv/poetry) plus a Dockerfile you can run anywhere.

  • Action:

make venv
pip freeze > requirements.txt
make build

Tip: Prefer Docker for broad compatibility; Podman is drop-in on many distros:

podman build -t ai-cicd-demo:latest .
podman run --rm -p 8000:8000 ai-cicd-demo:latest

2) Test the right things (fast)

  • Why: Full training is slow; you need fast feedback every push.

  • How: Unit tests, deterministic seeds, smoke training on tiny data, and a performance gate.

  • Action:

SEED=123 make train
make test

Add a nightly job for full training; keep PR checks under a few minutes.

3) Version data and models like code

  • Why: You must know exactly which data and model produced a result.

  • How: Use DVC so artifacts and metrics are traceable to commits.

  • Action:

dvc add models/model.joblib
git add models/model.joblib.dvc .gitignore
git commit -m "Track model with DVC"

Configure a remote (S3, SSH, etc.) to store large files:

dvc remote add -d storage s3://your-bucket/path
dvc push

4) Automate CI for build, test, and security

  • Why: Consistent, repeatable checks protect you from regressions and supply chain risk.

  • How: GitHub Actions (or GitLab CI, Jenkins) runs tests, checks deps, builds images, and scans.

  • GitHub Actions example (.github/workflows/ci.yml):

name: CI

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install deps
        run: |
          python -m venv .venv
          . .venv/bin/activate
          pip install -U pip wheel
          pip install -r requirements.txt pytest joblib
      - name: Train (smoke)
        run: |
          . .venv/bin/activate
          SEED=42 python src/train.py
      - name: Run tests
        run: |
          . .venv/bin/activate
          pytest -q

      - name: Pip audit
        run: |
          pipx install pip-audit
          pip-audit -r requirements.txt

  build_and_scan:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t ghcr.io/${{ github.repository }}:latest .
      - name: Trivy scan (no local install)
        run: |
          docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy:latest image --exit-code 1 --severity HIGH,CRITICAL \
            ghcr.io/${{ github.repository }}:latest
  • GitLab CI quick-start (.gitlab-ci.yml):
stages: [test, build]

test:
  image: python:3.11-slim
  stage: test
  script:
    - python -m venv .venv
    - . .venv/bin/activate
    - pip install -r requirements.txt pytest joblib
    - SEED=42 python src/train.py
    - pytest -q
  artifacts:
    paths:
      - models/

build:
  image: docker:24
  stage: build
  services: [docker:24-dind]
  script:
    - docker build -t registry.gitlab.com/$CI_PROJECT_PATH:latest .
    - docker push registry.gitlab.com/$CI_PROJECT_PATH:latest
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

5) Deliver safely with a staging gate

  • Why: Catch runtime issues before users do.

  • How: Build and push image on main, deploy to staging, run smoke tests, then promote.

  • Action (local/staging smoke):

docker run -d --rm -p 8000:8000 --name ai-demo ghcr.io/OWNER/REPO:latest
sleep 2
curl -s -X POST http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"features":[5.1,3.5,1.4,0.2]}'
docker stop ai-demo

For production, use a GitOps flow (e.g., update a Helm chart or Compose file in a deploy repo after CI passes).

Real-world tips

  • Separate CPU-only PR checks from scheduled GPU jobs. Run full training nightly or on-demand.

  • Pin random seeds and document hardware/software in CI logs.

  • Cache dependencies in CI to speed up runs.

  • Store metrics as JSON and fail CI if they regress past a threshold.

  • Use pre-commit hooks to auto-format and lint before code lands:

pre-commit install

What you get by adopting AI CI/CD

  • Confidence: Every change is tested, scanned, and reproducible.

  • Speed: Developers run the same Make targets locally that CI runs in the cloud.

  • Traceability: You can answer “which code and data produced this model?” anytime.

  • Safety: Performance and security gates stop regressions before production.

Call to action

  • Initialize the layout above in your repo.

  • Get CI green locally:

make venv
SEED=42 make train
make test
make build

  • Turn on the CI workflow and ship your first staging image.

  • Add one improvement per week: data remotes, nightly GPU training, canary tests, or infra-as-code for deployment.

Ship models like software—reliably, repeatedly, and with less stress.

Related posts