Posted on
• Artificial Intelligence

Artificial Intelligence Continuous Integration

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Continuous Integration (AI CI) on Linux with Bash

If your model “works on my machine” but breaks in production, you don’t have a model problem—you have a process problem. Continuous Integration (CI) changed how we ship software; AI CI does the same for machine learning by automating fast checks for data, code, and model quality before anything merges to main.

This guide shows you how to set up a lean, Linux-first AI CI pipeline you can run locally and in GitHub Actions or GitLab CI, using only Bash and a few essential tools.

What you’ll get:

  • Reproducible Python environment with pre-commit formatting and linting

  • Data/model versioning with Git LFS and DVC

  • Fast unit/data tests and a “smoke” training run with accuracy gates

  • Ready-to-paste CI configurations for GitHub and GitLab

Why AI needs CI (and why now)

  • AI integrates code, data, and models—any one of these can silently regress. CI catches regressions early.

  • Data distribution drifts and changes subtly. Data checks in CI flag schema/quality issues before training.

  • Reproducibility isn’t optional. CI codifies environments, seeds, and dependency locks.

  • Governance and compliance are easier when quality gates are automated and auditable in logs.

Prerequisites (install via your package manager)

Install core tooling: Git, Git LFS, Python 3, virtual environments, Make, jq, and curl.

APT (Debian/Ubuntu):

sudo apt update
sudo apt install -y git git-lfs python3 python3-venv python3-pip make jq curl

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y git git-lfs python3 python3-pip make jq curl

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y git git-lfs python3 python3-pip make jq curl

Optional: container engine for local runners

  • APT:

    • Docker: sudo apt install -y docker.io
    • Podman: sudo apt install -y podman

  • DNF:

    • Podman: sudo dnf install -y podman
    • Docker (moby): sudo dnf install -y moby-engine (package name may vary)

  • Zypper:

    • Podman: sudo zypper install -y podman
    • Docker: sudo zypper install -y docker

Initialize Git LFS once:

git lfs install

Project bootstrap (one-time)

Create and enter a project folder, then prepare a virtual environment and install minimal packages:

mkdir ai-ci-demo && cd ai-ci-demo
git init

python3 -m venv .venv
. .venv/bin/activate

python -m pip install --upgrade pip
pip install black isort flake8 pytest pre-commit dvc pandas scikit-learn joblib

Pin dependencies for reproducibility:

pip freeze > requirements-lock.txt
git add requirements-lock.txt

Initialize DVC for data/model tracking:

dvc init
git add .dvc .dvcignore
git commit -m "Init DVC"

Set up pre-commit:

cat > .pre-commit-config.yaml <<'YAML'
repos:
  - repo: https://github.com/psf/black
    rev: 24.4.2
    hooks: [{id: black}]
  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks: [{id: isort}]
  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks: [{id: flake8}]
  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks: [{id: nbstripout}]
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks: [{id: detect-secrets}]
YAML

pre-commit install
git add .pre-commit-config.yaml
git commit -m "Add pre-commit (black/isort/flake8/nbstripout/detect-secrets)"

Add a simple trainer, accuracy gate, and tests:

mkdir -p scripts tests
cat > train.py <<'PY'
import argparse, json, os, random
import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import joblib

def set_seeds(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--sample", type=int, default=120, help="samples to use for quick smoke run")
    parser.add_argument("--out", type=str, default="metrics.json")
    parser.add_argument("--model-path", type=str, default="model.joblib")
    args = parser.parse_args()

    set_seeds(42)
    iris = datasets.load_iris()
    X, y = iris.data, iris.target
    if args.sample < len(X):
        X, y = X[:args.sample], y[:args.sample]

    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
    clf = LogisticRegression(max_iter=1000, random_state=42, n_jobs=1)
    clf.fit(Xtr, ytr)
    preds = clf.predict(Xte)
    acc = accuracy_score(yte, preds)

    with open(args.out, "w") as f:
        json.dump({"accuracy": acc}, f)

    joblib.dump(clf, args.model_path)
    print(f"accuracy={acc:.4f}")

if __name__ == "__main__":
    main()
PY

cat > scripts/validate_metrics.py <<'PY'
import argparse, json, sys

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--path", default="metrics.json")
    p.add_argument("--min-acc", type=float, default=0.9)
    args = p.parse_args()
    with open(args.path) as f:
        m = json.load(f)
    acc = float(m.get("accuracy", 0.0))
    if acc < args.min-acc:
        print(f"FAIL: accuracy {acc:.4f} < threshold {args.min-acc:.4f}")
        sys.exit(1)
    print(f"OK: accuracy {acc:.4f} >= threshold {args.min-acc:.4f}")

if __name__ == "__main__":
    main()
PY

cat > tests/test_train.py <<'PY'
from subprocess import check_call
import json, os

def test_smoke_training(tmp_path):
    mpath = tmp_path/"m.json"
    check_call(["python", "train.py", "--sample", "120", "--out", str(mpath)])
    assert mpath.exists()
    m = json.loads(mpath.read_text())
    assert "accuracy" in m and 0 <= m["accuracy"] <= 1
PY

Optional Makefile to run common tasks:

cat > Makefile <<'MK'
.PHONY: fmt lint test train-smoke eval ci
fmt:
    . .venv/bin/activate && black . && isort .
lint:
    . .venv/bin/activate && flake8 .
test:
    . .venv/bin/activate && pytest -q
train-smoke:
    . .venv/bin/activate && python train.py --sample 120 --out metrics.json
eval:
    . .venv/bin/activate && python scripts/validate_metrics.py --path metrics.json --min-acc 0.9
ci: fmt lint test train-smoke eval
MK

Commit:

git add .
git commit -m "Add trainer, tests, metrics gate, and Makefile"

5 Actionable Steps to AI CI on Linux

1) Standardize the environment

  • Always use a venv and pin dependencies.

  • Seed all randomness.

  • Automate formatting/linting with pre-commit so CI isn’t fighting style drift.

Run locally:

. .venv/bin/activate
make fmt lint

2) Version data and models with DVC + Git LFS

  • Keep datasets and binary models out of Git history; track pointers instead.

  • Use DVC to add data and models; push to a remote (local dir, S3, SSH, etc.).

Example (local directory remote):

mkdir -p data
echo "placeholder" > data/README.md
dvc add data
git add data.dvc .gitignore
dvc remote add -d local ./dvcstore
git commit -m "Track data with DVC (local remote)"

3) Add fast tests that fail fast

  • Unit tests for utilities.

  • Data sanity checks (row counts, nulls).

  • Smoke training with a small sample and a metrics gate.

Run locally:

. .venv/bin/activate
pytest -q
python train.py --sample 120 --out metrics.json
python scripts/validate_metrics.py --path metrics.json --min-acc 0.9

4) Enforce quality gates

  • Treat metrics like unit tests: fail the build on regression.

  • Keep thresholds realistic and revisit them as the model improves.

Adjust the threshold in scripts/validate_metrics.py or pass --min-acc.

5) Wire it into CI

GitHub Actions (.github/workflows/ai-ci.yml):

mkdir -p .github/workflows
cat > .github/workflows/ai-ci.yml <<'YAML'
name: AI CI
on:
  push:
  pull_request:
jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Git LFS
        run: git lfs install
      - name: Install deps
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-lock.txt || \
          pip install black isort flake8 pytest pre-commit dvc pandas scikit-learn joblib
      - name: Pre-commit
        run: |
          pip install pre-commit
          pre-commit run -a || (git status --porcelain && exit 1)
      - name: Tests
        run: pytest -q
      - name: Smoke train + eval
        run: |
          python train.py --sample 120 --out metrics.json
          python scripts/validate_metrics.py --path metrics.json --min-acc 0.9
YAML

GitLab CI (.gitlab-ci.yml):

cat > .gitlab-ci.yml <<'YAML'
stages: [test]
ai_ci:
  stage: test
  image: python:3.11-slim
  before_script:
    - apt-get update
    - apt-get install -y --no-install-recommends git git-lfs build-essential jq curl
    - git lfs install
    - python -m pip install --upgrade pip
    - pip install -r requirements-lock.txt || pip install black isort flake8 pytest pre-commit dvc pandas scikit-learn joblib
  script:
    - pre-commit run -a || (git status --porcelain && exit 1)
    - pytest -q
    - python train.py --sample 120 --out metrics.json
    - python scripts/validate_metrics.py --path metrics.json --min-acc 0.9
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "merge_request_event"
YAML

Push and watch the pipeline:

git add .github/workflows/ai-ci.yml .gitlab-ci.yml || true
git commit -m "Add CI"
git branch -M main
git remote add origin <your-remote-url>
git push -u origin main

Real-world tips

  • Cache smartly: keep smoke tests under ~60 seconds. Use small samples or lightweight models in CI; run full training in scheduled workflows or CD.

  • Separate data checks from model checks: break failures down into actionable signals.

  • Log everything: save metrics to JSON and print summaries so CI logs are self-explanatory.

  • Security: run detect-secrets in pre-commit and CI; block accidental key commits.

Conclusion and next steps

AI CI turns “hope-driven” merges into measurable, repeatable quality. You now have:

  • Reproducible environments

  • Data/model versioning

  • Automated tests with metrics thresholds

  • Working CI configs for GitHub and GitLab

Next steps:

  • Add dataset-specific checks (nulls, ranges, schema)

  • Track artifacts with DVC remotes (S3/SSH) and push/pull in CI

  • Promote models via CD only when CI gates pass

Have questions or want a follow-up on adding CD and model registry integration (MLflow/Hugging Face Hub) using Bash and containers? Tell me what stack you’re on, and I’ll tailor the next guide.

Related posts