Artificial Intelligence CI/CD Pipelines on Linux: From Notebook to Production, Reliably

Ever shipped a model that worked on your laptop but fell apart in production? Data drift, missing dependencies, “it works on my machine” environments, and manual releases are the biggest reasons AI projects stall. A Linux-first CI/CD pipeline solves this by making your experiments reproducible, your builds automated, and your deployments boring—in the best way.

This article shows you how to build a practical AI CI/CD pipeline on Linux with Bash-first tooling, from data and model versioning to tests, builds, and deployment. You’ll get copy-pasteable commands, minimal examples, and installation steps for apt, dnf, and zypper.

Why AI CI/CD on Linux is worth it

Reproducibility: Pin environments, automate training, and replay experiments exactly.
Quality gates: Fail fast on bad data, broken metrics, or flaky tests.
Speed: Cache dependencies, containerize, and reuse artifacts.
Security: Scan containers, freeze dependencies, sign images.
Scale: Run headless on any Linux server or runner—local, on-prem, or cloud.

What we’ll build

A lean, Linux-native workflow that:

Creates a reproducible Python environment with a Makefile
Versions data/models with DVC and logs runs with MLflow
Adds tests and quality checks (pytest, pre-commit, ShellCheck)
Containerizes and pushes your model server image (Podman or Docker)
Automates the lot in CI (example: GitHub Actions)

You can adapt the same ideas to GitLab CI, Jenkins, or any runner with Bash.

Prerequisites: Install core CLI tools

Install these once on your Linux dev box or runner.

Git, Python, venv, pip, Make, jq, curl

Debian/Ubuntu:

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip make jq curl

Fedora/RHEL/CentOS Stream:

sudo dnf install -y git python3 python3-pip make jq curl

openSUSE/SLES:

sudo zypper install -y git python3 python3-pip make jq curl

Optional but recommended: Podman/Buildah/Skopeo (rootless containers)

Debian/Ubuntu:

sudo apt install -y podman buildah skopeo

Fedora/RHEL/CentOS Stream:

sudo dnf install -y podman buildah skopeo

openSUSE/SLES:

sudo zypper install -y podman buildah skopeo

Or Docker Engine (if you prefer Docker)

Debian/Ubuntu:

sudo apt install -y docker.io docker-compose-plugin
sudo usermod -aG docker $USER

Fedora/RHEL/CentOS Stream:

sudo dnf install -y moby-engine docker-compose-plugin
sudo usermod -aG docker $USER

openSUSE/SLES:

sudo zypper install -y docker docker-compose
sudo usermod -aG docker $USER

Log out/in after adding your user to the docker group.

Pre-commit and ShellCheck (for quality gates)

Debian/Ubuntu:

sudo apt install -y pre-commit shellcheck

Fedora/RHEL/CentOS Stream:

sudo dnf install -y pre-commit ShellCheck

openSUSE/SLES:

sudo zypper install -y pre-commit ShellCheck

pipx (for cleanly installing CLI Python tools like DVC)

Debian/Ubuntu:

sudo apt install -y pipx
pipx ensurepath

Fedora/RHEL/CentOS Stream:

sudo dnf install -y pipx
pipx ensurepath

openSUSE/SLES:

sudo zypper install -y pipx
pipx ensurepath

1) Scaffold a reproducible ML project

Use a Makefile as your one-liner interface for everything: environment, tests, training, packaging.

Project layout:

ai-pipeline/
├─ Makefile
├─ requirements.txt
├─ src/
│  ├─ train.py
│  └─ app.py
├─ data/        # tracked with DVC (later)
├─ models/      # outputs
├─ scripts/
│  └─ smoke.sh
├─ .pre-commit-config.yaml
├─ Dockerfile
└─ tests/
   └─ test_basic.py

Minimal Makefile:

PY := python3
VENV := .venv
PIP := $(VENV)/bin/pip
PYTHON := $(VENV)/bin/python

ENGINE ?= podman # set to docker if you prefer

.PHONY: venv deps lint test train package run clean

venv:
    $(PY) -m venv $(VENV)

deps: venv
    $(PIP) install -U pip
    $(PIP) install -r requirements.txt
    $(PIP) install mlflow pytest

lint:
    pre-commit run --all-files
    shellcheck scripts/*.sh

test: deps
    $(PYTHON) -m pytest -q

train: deps
    SEED=42 $(PYTHON) src/train.py --data data/raw --out models/

package:
    $(ENGINE) build -t ghcr.io/yourorg/yourmodel:$(shell git rev-parse --short HEAD) .

run:
    $(ENGINE) run --rm -p 8000:8000 ghcr.io/yourorg/yourmodel:$(shell git rev-parse --short HEAD)

clean:
    rm -rf $(VENV) models/*.pkl .pytest_cache

Sample requirements.txt:

numpy==1.26.4
scikit-learn==1.5.1
fastapi==0.111.0
uvicorn[standard]==0.30.0

Minimal train.py (logs to MLflow, saves a model):

import os, json, mlflow
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
import pickle
import numpy as np

seed = int(os.getenv("SEED", "42"))
np.random.seed(seed)

X, y = load_iris(return_X_y=True, as_frame=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=seed)

with mlflow.start_run():
    model = LogisticRegression(max_iter=200)
    model.fit(Xtr, ytr)
    acc = model.score(Xte, yte)
    mlflow.log_metric("accuracy", acc)

    os.makedirs("models", exist_ok=True)
    with open("models/model.pkl", "wb") as f:
        pickle.dump(model, f)

    with open("models/metrics.json", "w") as f:
        json.dump({"accuracy": acc}, f)

print("Trained. Accuracy:", acc)

Minimal app.py (serves predictions via FastAPI):

from fastapi import FastAPI
import pickle
import numpy as np

app = FastAPI()
model = pickle.load(open("models/model.pkl", "rb"))

@app.get("/healthz")
def health():
    return {"status": "ok"}

@app.post("/predict")
def predict(payload: dict):
    X = np.array(payload["features"]).reshape(1, -1)
    y = model.predict(X).tolist()
    return {"prediction": y}

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt mlflow
COPY models/ models/
COPY src/ src/
ENV PORT=8000
EXPOSE 8000
CMD ["python","-m","uvicorn","src.app:app","--host","0.0.0.0","--port","8000"]

Smoke test script scripts/smoke.sh:

#!/usr/bin/env bash
set -euo pipefail
curl -fsSL "http://localhost:8000/healthz" | jq -e '.status=="ok"' >/dev/null

2) Version data and models with DVC, track runs with MLflow

Use DVC to keep data and model artifacts out of Git history while keeping pointers in Git. Use any remote (S3, GCS, SSH, WebDAV).

Install DVC via pipx:

pipx install dvc
# Also available with extras, e.g.:
# pipx install "dvc[s3,gdrive]"

Initialize and track:

dvc init
git add .dvc .gitignore
git commit -m "init dvc"

mkdir -p data/raw
# put your dataset in data/raw
dvc add data/raw
git add data/.gitignore data/raw.dvc
git commit -m "track raw data with dvc"

# Configure remote (example using a local directory)
mkdir -p ../dvc-remote
dvc remote add -d localremote ../dvc-remote
dvc push

MLflow tracking server (optional, for teams):

. .venv/bin/activate
mlflow ui --backend-store-uri sqlite:///mlruns.db --host 0.0.0.0 --port 5000

Now every git commit + dvc push ties code to data. Your CI can pull exact versions with:

git clone ...
dvc pull

3) Add tests and quality gates (fail fast)

Install pre-commit and ShellCheck (see install section above), then set up pre-commit:

pre-commit install

.pre-commit-config.yaml:

repos:

- repo: https://github.com/psf/black
  rev: 24.4.2
  hooks: [{id: black}]

- repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.5.0
  hooks: [{id: ruff}]

- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v1.10.0
  hooks: [{id: mypy}]

Basic pytest test tests/test_basic.py:

def test_sanity():
    assert 1 + 1 == 2

Data and model checks in CI (bash-friendly):

jq -e '.accuracy >= 0.9' models/metrics.json

Run locally:

make lint
make test
make train

4) Containerize and push images (Podman or Docker)

Build with Podman:

make package
# or explicitly:
podman build -t ghcr.io/yourorg/yourmodel:$(git rev-parse --short HEAD) .

Push with Skopeo or Podman:

echo "$CR_PAT" | podman login ghcr.io -u youruser --password-stdin
podman push ghcr.io/yourorg/yourmodel:$(git rev-parse --short HEAD)

Using Docker:

docker build -t ghcr.io/yourorg/yourmodel:$(git rev-parse --short HEAD) .
echo "$CR_PAT" | docker login ghcr.io -u youruser --password-stdin
docker push ghcr.io/yourorg/yourmodel:$(git rev-parse --short HEAD)

Quick local run:

podman run --rm -p 8000:8000 ghcr.io/yourorg/yourmodel:$(git rev-parse --short HEAD)
curl -s localhost:8000/healthz | jq

5) Automate it in CI (GitHub Actions example)

.github/workflows/ci.yml:

name: ci

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build-test-train-package:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      packages: write

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Cache pip
      uses: actions/cache@v4
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

    - name: Install system deps
      run: sudo apt-get update && sudo apt-get install -y pre-commit shellcheck jq

    - name: Venv + deps
      run: |
        python -m venv .venv
        . .venv/bin/activate
        pip install -U pip
        pip install -r requirements.txt mlflow pytest

    - name: Lint
      run: |
        pre-commit install
        pre-commit run --all-files
        shellcheck scripts/*.sh

    - name: Test
      run: . .venv/bin/activate && pytest -q

    - name: Train (small/ci dataset)
      run: . .venv/bin/activate && SEED=42 python src/train.py --data data/raw --out models/

    - name: Quality gate on metrics
      run: jq -e '.accuracy >= 0.9' models/metrics.json

    - name: Build and push image
      uses: docker/build-push-action@v6
      with:
        context: .
        push: true
        tags: ghcr.io/${{ github.repository_owner }}/yourmodel:${{ github.sha }}
        labels: |
          org.opencontainers.image.source=${{ github.repository }}
          org.opencontainers.image.revision=${{ github.sha }}

What this does:

Recreates the environment
Lints and tests
Trains a fast model
Fails if accuracy is below threshold
Builds and pushes a versioned image

Adapt the same steps to GitLab CI (.gitlab-ci.yml) or Jenkins (Jenkinsfile) with shell stages.

Real-world tips that pay off

Pin everything: requirements.txt, Docker base image tag, and random seeds.
Makefile as the one API: if it’s not in make, it’s not “standard.”
Separate data sizes: tiny “ci” dataset for quick training; full dataset for nightly or GPU runners.
Cache smartly: pip cache, DVC cache, and container layers to speed CI.
Track lineage: use DVC + MLflow to trace which data, code, and params produced each model.
Security: run ShellCheck, scan images (e.g., trivy), and sign images (cosign) in CI.

Troubleshooting quick hits

“Module venv not found” on Debian: install python3-venv.
Permissions denied for Docker: add your user to the docker group and re-login.
Slow CI: switch to Podman rootless or cache pip and build layers.
Inconsistent metrics across runs: fix seeds, pin versions, and freeze CPU/GPU BLAS threads for determinism.

Your next step (CTA)

Start small: 1) Copy the Makefile, requirements.txt, train.py, app.py, and Dockerfile into a fresh repo. 2) make train then make package and run the container locally. 3) Add the GitHub Actions workflow (or your CI of choice) and watch it go green on every push. 4) Layer in DVC and MLflow to lock down reproducibility and tracking.

When your pipeline is boring, your releases can be bold. Build it once on Linux—then run it anywhere.

Artificial Intelligence CI/CD Pipelines on Linux: From Notebook to Production, Reliably

Why AI CI/CD on Linux is worth it

What we’ll build

Prerequisites: Install core CLI tools

1) Scaffold a reproducible ML project

2) Version data and models with DVC, track runs with MLflow

3) Add tests and quality gates (fail fast)

4) Containerize and push images (Podman or Docker)

5) Automate it in CI (GitHub Actions example)

Real-world tips that pay off

Troubleshooting quick hits

Your next step (CTA)

Related posts