Posted on
• Artificial Intelligence

Artificial Intelligence CI/CD Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence CI/CD Best Practices (for Linux Bash Users)

When “it works on my machine” meets machine learning, things get weird fast: a tiny dependency bump changes your model’s accuracy, a teammate retrains on new data and silently regresses the metric, or a container image grows to gigabytes and takes 20 minutes to build in CI. You need CI/CD that understands data, models, and compute — not just code.

This post explains why AI requires its own CI/CD patterns, then gives you battle-tested, shell-first practices you can implement today. You’ll get install commands for apt, dnf, and zypper, practical Bash snippets, and a clear path to production-grade AI automation.

Why AI CI/CD is different (and worth fixing)

  • Data is a first-class input. Code changes don’t tell the whole story; data drift does.

  • Reproducibility matters. The same seed and environment must recreate the same model.

  • Artifacts are big. Datasets, checkpoints, and containers need caching and smart transfer.

  • “Tests” include metrics. Shipping a model means guarding against accuracy/latency regressions.

  • Security and compliance matter. Supply chain risks exist in images, Python deps, and models.

If your pipeline doesn’t address these, you’re paying with flaky builds, surprise regressions, and slow delivery.

Prerequisites (install once, use everywhere)

Pick your package manager and install the basics you’ll use throughout this guide.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y git git-lfs python3 python3-pip python3-venv jq podman
# Optional: Docker instead of Podman
sudo apt install -y docker.io
sudo systemctl enable --now docker

Dnf (Fedora/RHEL/CentOS Stream):

sudo dnf -y install git git-lfs python3 python3-pip jq podman
# Optional: Docker instead of Podman (on Fedora this package may be 'moby-engine')
sudo dnf -y install docker
sudo systemctl enable --now docker

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y git git-lfs python3 python3-pip jq podman
# Optional: Docker instead of Podman
sudo zypper install -y docker
sudo systemctl enable --now docker

Python tooling (pip-based for cross-distro consistency):

python3 -m pip install --upgrade pip
python3 -m pip install dvc mlflow pytest

Initialize Git LFS once per machine:

git lfs install

1) Version everything: code, data, and model artifacts

Treat data and models like code: track, diff, and move them predictably.

  • Use Git for code and metadata.

  • Use Git LFS for large binary blobs (e.g., model weights).

  • Use DVC to version datasets and connect to remote storage.

Quick start:

# In your repo
git init
git lfs track "models/*"
echo "models/" >> .gitignore
git add .gitattributes .gitignore
git commit -m "Track model binaries with Git LFS"

# Initialize DVC and add data
dvc init
mkdir -p data/raw
# Put your raw data in data/raw/
dvc add data/raw
git add data/raw.dvc .dvc/.gitignore
git commit -m "Track raw dataset with DVC"

# Configure a DVC remote (example: local folder; swap for S3, GCS, etc.)
dvc remote add -d storage ./dvc_storage
dvc push

Why it helps:

  • Reproducible training: the exact dataset version that produced a model is fetchable in CI.

  • Lightweight repos: Git only sees small .dvc pointers; data moves via the DVC remote.

2) Build reproducible, portable environments with containers

CI/CD should produce the same bits you’ll run in staging/production. Containers make this boring and reliable.

Minimal Dockerfile/Containerfile:

# Save as Dockerfile (or Containerfile for Podman)
FROM python:3.11-slim

# Speed up pip and ensure reproducibility
ENV PIP_NO_CACHE_DIR=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app
COPY requirements.txt .
RUN python -m pip install --upgrade pip && pip install -r requirements.txt

COPY . .
# Optional: set a default entrypoint
# ENTRYPOINT ["python", "-m", "src.infer"]

Build and run (Podman):

podman build -t myorg/myproj:train -f Dockerfile .
podman run --rm -it -v "$PWD":/app -w /app myorg/myproj:train python -V

Build and run (Docker):

docker build -t myorg/myproj:train -f Dockerfile .
docker run --rm -it -v "$PWD":/app -w /app myorg/myproj:train python -V

Tips:

  • Pin dependencies in requirements.txt or Poetry’s lock file.

  • Bake CUDA/cuDNN base images only when you need GPUs; test on CPU-first to reduce cost.

  • Keep the training image and inference image separate; the latter should be thin and fast.

3) Gate releases with tests and metrics (not just unit tests)

Ship models only when quality holds steady. Add fast tests and a metrics gate to your CI.

Example layout:

# Run unit tests
pytest -q

# Run a fast smoke-train to catch breakage (subset, 1 epoch, fixed seed)
python -m src.train --epochs 1 --sample 1024 --seed 1337 --out metrics.json

# Compare metrics to a previous baseline (commit, artifact, or file)
cat > previous_metrics.json <<'JSON'
{"f1": 0.85}
JSON

python - <<'PY'
import json, sys
prev = json.load(open("previous_metrics.json"))["f1"]
cur = json.load(open("metrics.json"))["f1"]
threshold = 0.98  # Allow up to 2% drop
ok = (cur >= prev * threshold)
print(f"Prev f1={prev} Cur f1={cur} OK={ok}")
sys.exit(0 if ok else 1)
PY

Notes:

  • Keep smoke tests under ~2–5 minutes so they always run in PRs.

  • Use the same random seeds across frameworks and data loaders for determinism.

  • Track baselines per dataset slice or segment when applicable.

4) Use a model registry to track lineage and promote safely

A registry logs metrics, artifacts, and lineage, and gives you controlled promotion across environments. MLflow is simple to start with.

Start MLflow locally:

# Start a local server; replace with a shared host in your team
mlflow server \
  --backend-store-uri sqlite:///mlruns.db \
  --default-artifact-root ./mlruns \
  --host 0.0.0.0 --port 5000 &
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000

Log and register a model:

python - <<'PY'
import mlflow, mlflow.sklearn
from sklearn.linear_model import LogisticRegression

mlflow.set_experiment("demo-exp")
with mlflow.start_run() as run:
    # Pretend we trained something
    mlflow.log_metric("f1", 0.88)
    model = LogisticRegression()
    mlflow.sklearn.log_model(model, "model")
    result = mlflow.register_model(f"runs:/{run.info.run_id}/model", "ChurnModel")
    print("Registered:", result.name, result.version)
PY

Promotion idea:

  • “Staging” on success of smoke tests and security checks.

  • “Production” only after full-data training and canary validation.

5) Scan images and dependencies to secure the supply chain

Scan what you ship. Trivy is easy to run from a container, so you don’t even need to install it natively.

Podman:

# Save image to a tarball and scan from the Trivy container
podman save myorg/myproj:train -o image.tar
podman run --rm -v "$PWD":/work aquasec/trivy:latest image --input /work/image.tar

Docker:

docker save myorg/myproj:train -o image.tar
docker run --rm -v "$PWD":/work aquasec/trivy:latest image --input /work/image.tar

Also consider:

  • Pinning base images by digest (not just tags).

  • Scanning Python dependencies (requirements.txt) with your chosen tool during CI.

Real-world, end-to-end: a minimal CI-style Bash flow

This script shows a local “CI” you can run before wiring into GitHub Actions, GitLab CI, or Jenkins. It mirrors the best practices above.

#!/usr/bin/env bash
set -euo pipefail

# 0) Setup Python env
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install dvc mlflow pytest

# 1) Sync data (DVC)
dvc pull

# 2) Tests + smoke training with fixed seed
pytest -q
python -m src.train --epochs 1 --sample 2048 --seed 42 --out metrics.json

# Create a baseline if it doesn't exist
test -f previous_metrics.json || echo '{"f1": 0.80}' > previous_metrics.json

python - <<'PY'
import json, sys
p = json.load(open("previous_metrics.json"))["f1"]
c = json.load(open("metrics.json"))["f1"]
print(f"Baseline f1={p} current f1={c}")
sys.exit(0 if c >= p*0.98 else 1)
PY

# 3) Build container
if command -v podman >/dev/null 2>&1; then
  RUNTIME=podman
else
  RUNTIME=docker
fi
$RUNTIME build -t myorg/myproj:train -f Dockerfile .

# 4) Security scan via Trivy container
$RUNTIME save myorg/myproj:train -o image.tar
$RUNTIME run --rm -v "$PWD":/work aquasec/trivy:latest image --input /work/image.tar

# 5) Log to MLflow (local demo registry)
mlflow server --backend-store-uri sqlite:///mlruns.db --default-artifact-root ./mlruns \
  --host 127.0.0.1 --port 5000 &
MLFLOW_PID=$!
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
sleep 2

python - <<'PY'
import mlflow, mlflow.sklearn
from sklearn.linear_model import LogisticRegression
mlflow.set_experiment("demo-exp")
with mlflow.start_run() as run:
    mlflow.log_metric("f1", 0.82)
    mlflow.sklearn.log_model(LogisticRegression(), "model")
    mlflow.register_model(f"runs:/{run.info.run_id}/model", "DemoModel")
PY

kill $MLFLOW_PID || true
echo "CI-like flow completed."

Wire this into your CI and add proper caching:

  • Cache DVC remote between jobs to avoid re-downloading unchanged data.

  • Cache pip wheels (e.g., ~/.cache/pip).

  • Cache container layers by reusing the same base image and build context.

Conclusion and next steps

AI needs CI/CD that respects data, artifacts, and metrics — not just code. If you:

1) Version datasets and models with DVC and Git LFS, 2) Build repeatable environments via containers, 3) Enforce tests and metric gates, 4) Track lineage and promotion in a registry, 5) Scan what you ship,

…then your team will ship faster with fewer surprises.

Call to action:

  • Add DVC to your current ML repo and push your first dataset version.

  • Containerize your training job and run a local smoke test under CI.

  • Stand up an MLflow registry and promote your next “good” model through it.

  • Drop the sample Bash flow into your project and adapt it to your stack.

Make reliability your default — your future self (and your on-call rotation) will thank you.

Related posts