Posted on
• Artificial Intelligence

Artificial Intelligence GitOps Best Practices

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence GitOps Best Practices: a Linux Bash Playbook

Shipping a machine learning model isn’t the hard part—shipping it repeatedly, safely, and predictably is. If your AI services drift from what’s in Git, or if “the model worked on my laptop” is a recurring theme, it’s time to bring GitOps to your ML stack. In this guide, we’ll show how to apply GitOps best practices to AI systems using familiar Linux and Bash tooling. You’ll get a practical blueprint, shell-ready snippets, and distro-friendly install commands.

Why GitOps for AI is worth your time

  • Reproducibility: Models depend on code, data, and environments. GitOps enforces declarative, versioned deployments so what you serve matches what’s in Git.

  • Auditability and compliance: Regulated domains need provenance. Git history plus immutable images and pinned digests create an auditable trail.

  • Speed with safety: Automate CI checks, model quality gates, and progressive rollouts to move fast without surprise regressions.

Prerequisites: install the essentials (apt, dnf, zypper)

Update your package index, then install core tools. Pick the commands for your distro.

  • Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y git make python3 python3-venv python3-pip pipx jq podman curl gnupg lsb-release
  • Fedora/RHEL/CentOS (dnf)
sudo dnf -y install git make python3 python3-pip pipx jq podman curl gnupg2 redhat-lsb-core
  • openSUSE/SLE (zypper)
sudo zypper refresh
sudo zypper install -y git make python3 python3-pip python3-virtualenv pipx jq podman curl gpg2 lsb-release

Add pipx to your PATH (if needed), then install CLI tools we’ll use in the repo:

# Ensure pipx is on PATH for current shell
python3 -m pipx ensurepath

# Install pre-commit and DVC for data/model versioning
pipx install pre-commit
pipx install dvc

Install kubectl (official repositories):

  • Debian/Ubuntu (apt)
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl
  • Fedora/RHEL/CentOS (dnf)
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
sudo dnf config-manager --add-repo https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
sudo dnf install -y kubectl
  • openSUSE/SLE (zypper)
sudo rpm --import https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
sudo zypper addrepo https://pkgs.k8s.io/core:/stable:/v1.29/rpm/ kubernetes
sudo zypper refresh
sudo zypper install -y kubectl

Install Helm:

  • Debian/Ubuntu (apt)
curl https://baltocdn.com/helm/signing.asc | sudo gpg --dearmor -o /usr/share/keyrings/helm.gpg
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm
  • Fedora/RHEL/CentOS (dnf)
sudo tee /etc/yum.repos.d/helm.repo <<'EOF'
[helm]
name=Helm
baseurl=https://baltocdn.com/helm/stable/rpm
enabled=1
gpgcheck=1
gpgkey=https://baltocdn.com/helm/signing.asc
EOF
sudo dnf install -y helm
  • openSUSE/SLE (zypper)
sudo rpm --import https://baltocdn.com/helm/signing.asc
sudo zypper addrepo https://baltocdn.com/helm/stable/rpm/ helm
sudo zypper refresh
sudo zypper install -y helm

Optional: Flux CLI for GitOps

curl -s https://fluxcd.io/install.sh | sudo bash

Best practices you can apply today

1) Design repos for immutability and clarity

Keep training, packaging, and deployment declarative and separate responsibilities. A commonly successful layout:

.
├── data/                  # Small samples only; use DVC for large data
├── models/
│   ├── MODEL_VERSION      # e.g., 2024-07-01-bert-base-cased
│   └── card.md            # Model card: purpose, data, metrics, caveats
├── src/                   # Training/inference code
├── containers/
│   └── inference/Containerfile
├── charts/                # Helm chart for serving
├── k8s/                   # Raw manifests or kustomize overlays
│   ├── base/
│   └── overlays/
│       ├── staging/
│       └── prod/
├── ci/
│   └── check-metrics.sh
├── Makefile
└── .pre-commit-config.yaml

Version data and models without bloating Git:

dvc init
# Track large artifacts
dvc add data/raw/
# Configure a remote (S3, GCS, SSH, etc.)
dvc remote add -d storage s3://your-bucket/path
git add .dvc .gitignore
git commit -m "Initialize DVC for large data"

Pin everything that deploys:

  • Pin base images by digest, not tag.

  • Record model file hash:

sha256sum models/model.onnx | tee models/MODEL_SHA256

2) Make training and packaging reproducible (containers + Make)

Build once, run anywhere. Bake exact versions and hashes into your image.

containers/inference/Containerfile:

FROM docker.io/python:3.11@sha256:bd66e...   # pin by digest
ENV PIP_NO_BINARY=:all: PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
COPY src/ src/
COPY models/ models/
COPY requirements.txt .
RUN pip install --no-cache-dir --require-hashes -r requirements.txt
CMD ["python", "-m", "src.serve"]

Generate a locked requirements file with hashes:

python3 -m venv .venv && . .venv/bin/activate
pip install --upgrade pip pip-tools
pip-compile --generate-hashes -o requirements.txt pyproject.toml

Makefile (core targets):

REGISTRY ?= docker.io/yourname
IMAGE ?= $(REGISTRY)/ml-inference
GIT_SHA := $(shell git rev-parse --short HEAD)

.PHONY: train
train:
    . .venv/bin/activate && python src/train.py --out models/model.onnx

.PHONY: metrics
metrics:
    . .venv/bin/activate && python src/evaluate.py --in models/model.onnx --out metrics.json

.PHONY: build
build:
    podman build -f containers/inference/Containerfile -t $(IMAGE):$(GIT_SHA) .
    podman inspect $(IMAGE):$(GIT_SHA) --format '{{.Id}}' | sed 's/sha256://g' > image.digest

.PHONY: push
push:
    podman push $(IMAGE):$(GIT_SHA)
    @echo "Pushed $(IMAGE):$(GIT_SHA)"

.PHONY: tag-immutable
tag-immutable:
    @echo "Use digests in manifests for immutability."
    @podman inspect $(IMAGE):$(GIT_SHA) --format '{{.RepoDigests}}'

.PHONY: all
all: train metrics build

Tip: prefer Podman for rootless builds on Linux; it’s available via apt/dnf/zypper as shown above.

3) Enforce quality and security gates in CI

Before Git merges trigger deployments, catch issues locally and in CI.

Install and enable pre-commit hooks:

pre-commit install
pre-commit run --all-files

.pre-commit-config.yaml:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-yaml
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: https://github.com/psf/black
    rev: 24.4.2
    hooks:
      - id: black
        language_version: python3

Block regressions with a simple metric gate:

ci/check-metrics.sh:

#!/usr/bin/env bash
set -euo pipefail
THRESHOLD="${THRESHOLD:-0.89}"

acc=$(jq -r '.accuracy' metrics.json)
if [ -z "$acc" ] || [ "$acc" = "null" ]; then
  echo "No accuracy in metrics.json"; exit 1
fi

awk -v a="$acc" -v t="$THRESHOLD" 'BEGIN { exit (a>=t)?0:1 }' \
  || { echo "Accuracy $acc below threshold $THRESHOLD"; exit 1; }

echo "Accuracy $acc meets threshold $THRESHOLD"

Use this script in your CI pipeline before allowing merges into the GitOps “deploy” branch.

4) GitOps the deployment (Flux + Kubernetes manifests)

Bootstrap Flux (once, against your cluster and Git repository). If you use GitHub/GitLab, follow Flux’s bootstrap docs; for a quick local install:

flux check --pre
# If not bootstrapping via provider, you can apply manifests tracked in Git.

Declare your model service in Kubernetes and manage it via Git. Example Deployment (pin by image digest):

k8s/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-inference
spec:
  replicas: 3
  selector:
    matchLabels: { app: ml-inference }
  template:
    metadata:
      labels: { app: ml-inference }
    spec:
      containers:
        - name: server
          image: docker.io/yourname/ml-inference@sha256:abc123...   # immutable
          ports:
            - containerPort: 8080
          env:
            - name: MODEL_VERSION
              valueFrom:
                configMapKeyRef:
                  name: model-config
                  key: version

Simple Service:

k8s/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: ml-inference
spec:
  selector: { app: ml-inference }
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name: http

Flux sources and kustomization to sync from your Git repository:

k8s/flux-source.yaml:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: ai-platform
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/yourorg/yourrepo.git
  ref:
    branch: main

k8s/flux-kustomization.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: ml-inference
  namespace: flux-system
spec:
  interval: 1m
  prune: true
  wait: true
  path: ./k8s/overlays/prod
  sourceRef:
    kind: GitRepository
    name: ai-platform

Workflow:

  • A PR updates the image digest to a new, immutable build.

  • CI runs ci/check-metrics.sh and other tests.

  • Merge to main; Flux reconciles and applies the change to the cluster.

  • Roll back by reverting the commit; Flux restores the previous state.

For staged rollouts, use Helm’s canary values or separate overlays (staging vs prod) with different replica counts and traffic policies.

5) Record provenance and make rollbacks trivial

Track what’s running, where it came from, and how to go back fast.

  • Capture build metadata:
printf '{"git_sha":"%s","built_at":"%s"}\n' \
  "$(git rev-parse --short HEAD)" "$(date -u +%FT%TZ)" | tee build.json
  • Store model checksum and version next to your manifests; reference them via ConfigMaps:
kubectl create configmap model-config \
  --from-file=models/MODEL_SHA256 \
  --from-file=models/MODEL_VERSION \
  -n your-namespace \
  --dry-run=client -o yaml | tee k8s/base/model-config.yaml
  • Always pin image digests in manifests. To update after make push:
DIGEST=$(podman inspect docker.io/yourname/ml-inference:$(git rev-parse --short HEAD) \
  --format '{{index .RepoDigests 0}}' | sed 's/.*@//')
sed -i "s|@sha256:[a-f0-9]\+|@$DIGEST|g" k8s/overlays/prod/deployment.yaml
git commit -am "Bump image digest to $DIGEST"

Real-world example flow (end-to-end)

1) Develop and train:

make train
make metrics
ci/check-metrics.sh  # fails fast if quality regressed

2) Build, push, update manifest:

make build push
# Update deployment to new digest and commit

3) Merge to main; Flux deploys:

  • Flux picks up the new commit and reconciles the cluster.

  • If something goes wrong, git revert and Flux restores the last known good.

Conclusion and next steps

GitOps for AI is about turning messy, one-off heroics into repeatable, reviewable operations. Start small:

  • Add pre-commit hooks and a metrics gate.

  • Containerize inference and pin digests in Kubernetes manifests.

  • Let Flux (or your GitOps controller of choice) reconcile from Git.

Your next step: bootstrap a minimal repo with the structure above, wire up your CI to run ci/check-metrics.sh, and make your first digest-pinned deployment. From there, extend with progressive delivery, richer policy checks, and full data lineage.

If you want a follow-up post with a complete demo repository and CI workflow files, tell me which platform you use (GitHub/GitLab) and your target distro—I’ll tailor it to your stack.

Related posts