Artificial Intelligence CI/CD Case Studies: From Notebook to Production on Linux

If your model hits 94% accuracy in a notebook but customers see stale, slow, or wrong predictions in production, you don’t have an algorithm problem—you have a delivery problem. AI CI/CD (Continuous Integration/Continuous Delivery) brings the discipline of software delivery to data and models so that training, evaluation, deployment, and monitoring are repeatable, auditable, and fast.

This article shares real-world case studies and the exact Linux-friendly steps teams used to build reliable AI delivery pipelines. You’ll get actionable setups you can replicate on your workstation or in your org—using standard Bash, containers, and widely available packages.

Why AI CI/CD is different (and worth your time)

• Data and code both change. You need to version datasets, features, and models to reproduce results and roll back safely.

• Environments drift. GPU/CPU, drivers, dependency trees—containerization and pinned dependencies are essential.

• Performance is non-binary. Models degrade with data drift and concept shift; CI must validate metrics, not just unit tests.

• Governance matters. Registries, lineage, and approvals are required by regulated and enterprise teams.

• Speed wins. Automating training, evaluation, and rollout shortens the path from R&D to value.

Prerequisites: Linux installs you’ll actually use

Install the foundational tools with your package manager. Choose apt, dnf, or zypper as appropriate for your distro.

• Git

  • apt: sudo apt update && sudo apt install -y git
  • dnf: sudo dnf install -y git
  • zypper: sudo zypper install -y git

• Python 3 and pip

  • apt: sudo apt install -y python3 python3-pip
  • dnf: sudo dnf install -y python3 python3-pip
  • zypper: sudo zypper install -y python3 python3-pip

• Podman (rootless containers; Docker alternative)

  • apt: sudo apt install -y podman
  • dnf: sudo dnf install -y podman
  • zypper: sudo zypper install -y podman

• NGINX (for simple blue/green routing in one of the case studies)

  • apt: sudo apt install -y nginx
  • dnf: sudo dnf install -y nginx
  • zypper: sudo zypper install -y nginx

• Useful CLI tools

  • apt: sudo apt install -y curl jq make
  • dnf: sudo dnf install -y curl jq make
  • zypper: sudo zypper install -y curl jq make

Python packages used below can be installed per-project or to your user site:

• python3 -m pip install --user dvc mlflow pytest pre-commit evidently fastapi uvicorn

Tip: If you prefer isolation, create a virtual env with python3 -m venv .venv && . .venv/bin/activate (Debian/Ubuntu may require sudo apt install -y python3-venv).

---

Case Study 1: Reproducible training with DVC + Git

A media analytics team struggled to reproduce model improvements shared via notebooks and cloud buckets. They adopted DVC (Data Version Control) alongside Git to version data and pipelines. Within two sprints they could reproduce any model build by commit and data hash.

Actionable steps: 1. Initialize Git and DVC in your repo - git init - python3 -m pip install --user dvc - dvc init 2. Track datasets and avoid polluting Git with large files - mkdir -p data/raw && dvc add data/raw - git add data/.gitignore data/raw.dvc && git commit -m "Track raw data with DVC" 3. Configure remote storage for datasets (S3/GCS/Azure/local NAS) - Example (S3): dvc remote add -d storage s3://your-bucket/path && dvc push 4. Define a reproducible pipeline - Create a file dvc.yaml describing stages (prepare → train → evaluate). Then run dvc repro to execute with cached artifacts. 5. Add tests and run them in CI and locally - python3 -m pip install --user pytest - Structure tests to fail when metrics regress; e.g., store last-good metrics in metrics.json and assert thresholds. - Locally: pytest -q && dvc repro && dvc metrics show

Why it works:

• Every experiment is tied to code, data versions, parameters, and metrics.

• dvc repro ensures only what changed is recomputed, making CI fast.

---

Case Study 2: Containerized training with MLflow tracking

A retail team moved from ad hoc training to containerized jobs tracked by MLflow. Results: 40% less “it works on my machine,” faster reviews, and easy model lineage.

Actionable steps: 1. Create a minimal training entrypoint that logs to MLflow - python3 -m pip install --user mlflow - In your train.py, use MLflow to log params, metrics, and artifacts (models, plots). Keep it pure-Python with CLI args for seeds and data paths. 2. Build and run training in Podman - Add a Containerfile that installs system deps, then python3 -m pip install -r requirements.txt, and sets CMD ["python", "train.py"]. - Build: podman build -t ml-train:latest . - Run with mounted data and an experiment tag: podman run --rm -v $PWD:/app -w /app -e MLFLOW_TRACKING_URI=file:/app/mlruns ml-train:latest --data data/raw --seed 42 3. Promote best runs by metric threshold - Use MLflow model registry if you host a tracking server; otherwise, mark the blessed artifact with a Git tag referencing the run ID. - Automate selection: a small script parses MLflow runs and emits the model path with jq, gatekeeping merges on performance. 4. Integrate in CI - Your CI job should execute podman build, run training with a fixed seed, and fail if val_f1 drops. Use pytest to validate invariants (e.g., schema checks, no-leakage assertions).

Why it works:

• Containers standardize the environment; MLflow standardizes experiment lineage.

• Model promotion becomes a controlled, testable step instead of tribal knowledge.

---

Case Study 3: Safe blue/green model rollout on a VM with Podman + NGINX

A fintech startup without Kubernetes still achieved zero-downtime model releases. They ran two containerized FastAPI model versions behind NGINX and shifted traffic gradually.

Actionable steps: 1. Package your serving app - python3 -m pip install --user fastapi uvicorn - Containerize with a Containerfile that exposes port 8000 and runs uvicorn app:app --host 0.0.0.0 --port 8000. - Build: podman build -t fraud-model:1.0 . 2. Run blue and green versions on different ports - podman run -d --name model-blue -p 8001:8000 fraud-model:1.0 - Build new image: podman build -t fraud-model:1.1 . - podman run -d --name model-green -p 8002:8000 fraud-model:1.1 3. Install and configure NGINX to load balance - Install: - apt: sudo apt install -y nginx - dnf: sudo dnf install -y nginx - zypper: sudo zypper install -y nginx - Point an upstream to localhost:8001 and localhost:8002 with weights (e.g., 90/10). Update your site config and sudo systemctl reload nginx. 4. Health-check and promote - Probe each version: curl -s http://localhost:8001/health and curl -s http://localhost:8002/health - Canary metrics look good? Shift weights to 50/50, then 0/100. Finally, stop old version: podman rm -f model-blue

Why it works:

• Containers isolate runtime; NGINX gives you reversible, incremental cutovers.

• No Kubernetes needed; Bash-friendly and automatable with make or simple scripts.

---

Case Study 4: Continuous data drift monitoring with Evidently

A healthcare AI team missed a silent feature distribution shift. They added a daily drift job that produced HTML reports and alerted on thresholds.

Actionable steps: 1. Install and script a daily report - python3 -m pip install --user evidently - Write a small script that loads yesterday’s predictions and a reference dataset, computes PSI/KS, and writes an HTML report to reports/$(date +%F).html. 2. Schedule with cron - Edit crontab: crontab -e - Add a line to run at 02:15 daily: 15 2 * * * cd /path/to/repo && /usr/bin/python3 monitor.py >> logs/monitor.log 2>&1 3. Gate deploys on drift - Have your CI check the last drift score; if above threshold, block promotion and open an issue automatically.

Why it works:

• You treat data quality as a first-class citizen in CI/CD.

• Early warnings prevent bad models from reaching or lingering in production.

---

Bash-first building blocks you can copy

• Lock dependencies: pip-compile or pinned requirements.txt to avoid surprise version bumps.

• Fast inner loop: use make train, make test, make serve so CI and humans run the same commands.

• Pre-commit hygiene: python3 -m pip install --user pre-commit && pre-commit install to catch lint/format/test issues before they hit CI.

• Artifacts everywhere: persist models, metrics, and reports to a remote (DVC remote, MLflow, artifact store) so rollbacks and audits are one command away.

---

Conclusion and next steps

AI CI/CD pays off by making results reproducible, promotions trustworthy, and rollbacks painless. You’ve seen four battle-tested patterns—data versioning, containerized training, safe rollouts, and drift monitoring—implemented with standard Linux tools and containers.

Your next step: 1. Pick one case study and implement it this week. 2. Install the prerequisites with your package manager (apt/dnf/zypper). 3. Automate a single gate (e.g., fail CI on metric regression or drift). 4. Iterate toward fully automated training, promotion, and monitoring.

When your next model is ready, you’ll ship it confidently—with Bash commands you understand and a pipeline you control.