Posted on
• Artificial Intelligence

Artificial Intelligence Release Engineering

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Release Engineering with Bash: Ship Models Like Software

If your best model still lives in a notebook, you don’t have a model—you have a hypothesis. AI Release Engineering turns that hypothesis into a reproducible, testable, signed artifact you can ship with confidence. In this article you’ll learn a practical, Bash-first way to bring software rigor to ML: deterministic builds, versioned data, containerized models, cryptographic signing, and automated promotion.

Why this matters:

  • Reproducibility: “Works on my GPU” isn’t a release strategy.

  • Compliance and security: You’ll need provenance, SBOMs, signatures, and audit trails.

  • Speed with safety: Faster iterations, fewer surprises in production.

Below you’ll find concrete steps and copy‑pasteable shell and Makefile snippets to get there on any major Linux distro.

Prerequisites: Install the core tooling

Use your package manager to install the essentials. Run one of the following blocks.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y \
  git git-lfs make jq \
  python3 python3-venv python3-pip \
  podman skopeo cosign
git lfs install

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y \
  git git-lfs make jq \
  python3 python3-pip python3-virtualenv \
  podman skopeo cosign
git lfs install

Zypper (openSUSE/SLE):

sudo zypper install -y \
  git git-lfs make jq \
  python3 python3-pip python3-virtualenv \
  podman skopeo cosign
git lfs install

Notes:

  • If python3 -m venv isn’t available, ensure python3-venv (apt) or python3-virtualenv (dnf/zypper) is installed.

  • Podman is rootless-friendly and works great on all three families; you can substitute docker if you prefer.

  • cosign packages are available on most recent distros; if not, see sigstore’s install docs.

The high-level flow

1) Lock the environment and build deterministically
2) Version data and artifacts
3) Package the model with metadata
4) Sign and verify the release
5) Automate tests and promotion

Each step below includes real commands you can adopt today.

1) Lock the environment and build deterministically

Deterministic builds are the backbone of reproducible ML. Pin dependencies, capture build metadata, fix seeds, and make it one command to rebuild.

Makefile:

# Makefile
PY ?= python3
VENV ?= .venv
REQ ?= requirements.txt
ART ?= artifacts
DIST ?= dist
MODEL_NAME ?= churn-classifier
MODEL_VER ?= $(shell git describe --tags --always --dirty)
REGISTRY ?= registry.example.com/ml/$(MODEL_NAME)
SOURCE_URL ?= https://example.com/your-repo

.PHONY: venv deps train eval package image push sign release clean meta

venv:
    $(PY) -m venv $(VENV)
    $(VENV)/bin/pip install -U pip wheel

deps: venv
    # Install and freeze exact versions for reproducibility
    $(VENV)/bin/pip install -r $(REQ)
    $(VENV)/bin/pip freeze > freeze.txt

meta:
    mkdir -p $(ART) $(DIST)
    printf '{\n  "git_commit": "%s",\n  "built_at": "%s",\n  "python": "%s"\n}\n' \
      "$$(git rev-parse HEAD)" "$$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
      "$$($(VENV)/bin/python -V)" | jq '.' > $(ART)/build_meta.json

train: deps meta
    mkdir -p $(ART)
    # Example training with fixed seed; replace with your code
    $(VENV)/bin/python src/train.py \
      --seed 123 \
      --data data/train.csv \
      --out $(ART)/model.onnx

eval:
    # Fail the build if quality gates are not met
    $(VENV)/bin/python src/eval.py \
      --model $(ART)/model.onnx \
      --report $(ART)/metrics.json
    jq -e '.f1 >= 0.82' $(ART)/metrics.json

package:
    mkdir -p $(DIST)
    tar -C $(ART) -czf $(DIST)/$(MODEL_NAME)-$(MODEL_VER).tar.gz .

image:
    podman build \
      --pull=always \
      --label org.opencontainers.image.source=$(SOURCE_URL) \
      --label ai.model=$(MODEL_NAME) \
      --label ai.version=$(MODEL_VER) \
      -t $(REGISTRY):$(MODEL_VER) -f Containerfile .

push:
    podman push $(REGISTRY):$(MODEL_VER)

sign:
    # Requires cosign.key (generate with: cosign generate-key-pair)
    cosign sign --key cosign.key $(REGISTRY):$(MODEL_VER)

release: train eval package image push sign

clean:
    rm -rf $(VENV) $(ART) $(DIST)

Minimal training script (replace with your framework):

# src/train.py
import argparse, json, random, numpy as np
random.seed(123); np.random.seed(123)

parser = argparse.ArgumentParser()
parser.add_argument("--seed", type=int, default=123)
parser.add_argument("--data", required=True)
parser.add_argument("--out", required=True)
args = parser.parse_args()

# TODO: load data, train model. We'll fake it here:
with open(args.out, "wb") as f:
    f.write(b"FAKE_ONNX_BYTES")
print("Wrote", args.out)

Simple evaluator with a quality gate:

# src/eval.py
import argparse, json
parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
parser.add_argument("--report", required=True)
args = parser.parse_args()

# TODO: real evaluation; we’ll simulate a metric
metrics = {"f1": 0.84, "latency_ms_p95": 12.3}
with open(args.report, "w") as f:
    json.dump(metrics, f, indent=2)
print("Wrote", args.report)

Containerfile to ship the runtime:

# Containerfile
FROM python:3.11-slim@sha256:cf1b1762d9c3c7b8d8a1c7df3a7b1d2c0f0e1b4b9e9a...  # pin by digest
WORKDIR /opt/model
COPY artifacts/model.onnx /opt/model/model.onnx
COPY freeze.txt /opt/model/requirements.txt
# Optional: add a tiny runner
COPY tools/infer.py /opt/model/infer.py
ENTRYPOINT ["python","/opt/model/infer.py","--model","/opt/model/model.onnx"]

Example runner:

# tools/infer.py
import argparse, sys
parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
parser.add_argument("--input", default="-")
args = parser.parse_args()
data = sys.stdin.read() if args.input == "-" else open(args.input).read()
print('{"ok": true, "model": "%s"}' % args.model)

Run everything:

make release REGISTRY=registry.example.com/ml/churn-classifier

2) Version data and artifacts (and make large files painless)

Models and datasets don’t fit comfortably in plain Git. Use Git LFS for large, immutable blobs and keep pointers in the repo.

Initialize and track:

git lfs install
git lfs track "*.onnx"
git lfs track "data/*.csv"
echo ".gitattributes" >> .gitignore || true
git add .gitattributes
git add data/ artifacts/
git commit -m "Track model and data with Git LFS"

To capture integrity alongside your metadata:

sha256sum artifacts/model.onnx | awk '{print $$1}' > artifacts/model.sha256
jq --arg sha "$(< artifacts/model.sha256)" \
   '. + {artifact_sha256: $sha}' artifacts/build_meta.json > artifacts/build_meta2.json \
   && mv artifacts/build_meta2.json artifacts/build_meta.json

This makes it trivial to attest exactly which data, code, and binary you shipped.

3) Package with rich metadata

Use OCI labels to ship a self-describing artifact, and verify it remotely.

Build and inspect:

make image REGISTRY=registry.example.com/ml/churn-classifier
skopeo inspect docker://registry.example.com/ml/churn-classifier:$(git describe --tags --always) | jq '.Labels'

Add your own labels (owner, dataset snapshot, license) in the Makefile’s podman build step:

--label ai.dataset="s3://ml-bucket/snapshots/2024-07-01" \
--label ai.license="Apache-2.0" \
--label ai.metrics="$(shell cat artifacts/metrics.json | tr -d '\n')" \

Tip: keep labels small; for larger metadata, add a model card file into the image under /opt/model/.

4) Sign and verify your release

Signing prevents tampering and enables policy enforcement in CI/CD and clusters.

Generate keys once:

cosign generate-key-pair
# Produces cosign.key and cosign.pub

Sign and verify:

make sign REGISTRY=registry.example.com/ml/churn-classifier
cosign verify --key cosign.pub registry.example.com/ml/churn-classifier:$(git describe --tags --always)

You can attach additional attestations (e.g., build metadata, metrics) with cosign attest.

5) Automate tests and promotion

Quality gates make “go/no-go” decisions automatic.

Add an end-to-end check to the Makefile’s eval target (already shown) and a promotion rule:

.PHONY: promote
promote:
    # Tag "stable" only if previous steps passed
    podman tag $(REGISTRY):$(MODEL_VER) $(REGISTRY):stable
    podman push $(REGISTRY):stable

A tiny CI job (GitHub Actions example) to wire it up:

# .github/workflows/release.yml
name: Release Model
on:
  push:
    tags: ["v*"]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install deps
        run: |
          sudo apt update
          sudo apt install -y git git-lfs make jq python3 python3-venv python3-pip podman skopeo cosign
          git lfs install
      - name: Build, test, package, sign
        run: make release REGISTRY=${{ secrets.REGISTRY }}
      - name: Promote stable
        if: success()
        run: make promote REGISTRY=${{ secrets.REGISTRY }}

Replace secrets and registry with your environment.

Why this approach works for AI specifically

  • Non-determinism tamed: pinned containers, frozen Python deps, and fixed seeds make model training repeatable.

  • Data as a first-class citizen: tracking datasets and artifacts alongside code makes rollbacks and audits practical.

  • Supply-chain ready: OCI labels, remote inspect, and cosign signatures align with modern platform policies.

  • Bash-native: everything is simple, reviewable, and portable—ideal for Linux teams that automate with Make and shell.

Quick start checklist

  • Install the tools using apt, dnf, or zypper (see prerequisites).

  • Create the Makefile, Containerfile, and minimal train/eval scripts.

  • Track large files with Git LFS.

  • Run make release and verify the image with skopeo and cosign.

Conclusion / Call to Action

Treat your model like a product, not a prototype. Start today: 1) Bootstrap the Makefile above into your repo.
2) Wire in your real training and evaluation scripts.
3) Enforce quality gates and signatures before promotion.

When your next breakthrough is ready, you’ll ship it the same day—deterministically, securely, and with the audit trail your platform expects. If you want a ready-made skeleton, adapt the snippets here into your own “ml-release-template” and iterate from there.

Related posts