Posted on
• Artificial Intelligence

Artificial Intelligence DevOps Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence DevOps Checklists: A Bash-First Guide

Shipping AI isn’t just “deploy and forget.” Models drift, datasets grow stale, GPUs fail in odd ways, and dependency hell lurks everywhere. The result: broken builds, silent regressions, and late-night pages. This Bash-first guide delivers practical AI DevOps checklists you can copy into your repos today—so your experiments are reproducible, your deployments stay healthy, and your team sleeps better.

Why checklists for AI DevOps?

  • AI has more moving parts: data, features, models, hardware accelerators, and inference services. A failure in any layer can invalidate results.

  • Traditional web/service DevOps patterns don’t fully cover data drift, model regression, or artifact governance.

  • Bash-native checklists make best practices painless: one-liners, shell scripts, and Makefile targets your whole team can run on any Linux distro.

Below you’ll find five actionable checklists with copy-paste snippets and distro-agnostic install commands.

Prerequisites: set up your Linux AI DevOps toolbox

Run the commands appropriate to your distro.

  • Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y git make curl jq yq python3 python3-venv pipx podman
pipx ensurepath
  • Fedora/RHEL/CentOS Stream (dnf)
sudo dnf install -y git make curl jq yq python3 pipx podman
pipx ensurepath
  • openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y git make curl jq yq python3 pipx podman
pipx ensurepath

Optional: if you prefer Docker instead of Podman

  • Debian/Ubuntu
sudo apt install -y docker.io docker-compose-plugin
sudo usermod -aG docker "$USER"
# Re-login for group to take effect
  • Fedora/RHEL/CentOS Stream
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
  • openSUSE
sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

Pro tip: Prefer Podman for rootless containers on dev machines; it’s available across distros and works well with CI.

Checklist 1: Reproducible environments (no “works on my machine”)

Actions

  • Pin OS and Python dependencies.

  • Provide a one-command setup for local dev and CI.

  • Verify system prerequisites before anything else.

Makefile to standardize workflows:

# Makefile
PYTHON ?= python3
VENV ?= .venv
PIP ?= $(VENV)/bin/pip
PY ?= $(VENV)/bin/python

.PHONY: venv
venv:
    $(PYTHON) -m venv $(VENV)
    $(PIP) install --upgrade pip wheel
    # Use a locked requirements file if you have one; else, requirements.txt
    [ -f requirements.lock ] && $(PIP) install -r requirements.lock || $(PIP) install -r requirements.txt

.PHONY: check
check:
    ./scripts/checklist_env.sh

.PHONY: test
test: venv
    $(PY) -m pytest -q

.PHONY: fmt
fmt: venv
    $(PY) -m black src tests
    $(PY) -m isort src tests

Environment checklist script:

# scripts/checklist_env.sh
#!/usr/bin/env bash
set -euo pipefail

need() { command -v "$1" >/dev/null 2>&1 || { echo "Missing: $1"; exit 1; }; }

echo "Checking core tools…"
for c in git make curl jq yq python3; do need "$c"; done

echo "Checking container engine…"
if command -v podman >/dev/null 2>&1; then podman --version
elif command -v docker >/dev/null 2>&1; then docker --version
else
  echo "Missing: podman or docker"; exit 1
fi

echo "Python:"
python3 -c 'import sys; print(sys.version)'
python3 -m venv --help >/dev/null 2>&1 || { echo "python3-venv/venv support missing"; exit 1; }

echo "Disk space (need >5GB free):"
df -h .

echo "GPU (optional):"
if command -v nvidia-smi >/dev/null 2>&1; then nvidia-smi || true; else echo "No NVIDIA GPU detected."; fi

echo "OK ✅"

Deterministic container builds (Podman or Docker):

# Dockerfile
FROM python:3.11-slim@sha256:26c5a4d0b1d5f83c...  # pin by digest for reproducibility
ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential git ca-certificates && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.lock requirements.lock
RUN pip install --upgrade pip && pip install -r requirements.lock
COPY . .
CMD ["python", "-m", "your_package.api"]

Tip: lock Python deps with pip-tools and commit requirements.lock:

pipx install pip-tools
pip-compile -o requirements.lock pyproject.toml  # or requirements.in

Checklist 2: Data and model artifact governance

Actions

  • Store large binaries (models, embeddings) with versioning.

  • Keep cryptographic checksums alongside artifacts.

  • Automate verification in CI and before deploys.

Install Git LFS

  • Debian/Ubuntu
sudo apt install -y git-lfs
  • Fedora/RHEL/CentOS Stream
sudo dnf install -y git-lfs
  • openSUSE
sudo zypper install -y git-lfs

Initialize LFS and track models:

git lfs install
git lfs track "models/**"
echo "/models/*.sha256" >> .gitignore
git add .gitattributes

Create and verify checksums:

# Generate checksums
find models -type f ! -name "*.sha256" -print0 | while IFS= read -r -d '' f; do
  sha256sum "$f" > "${f}.sha256"
done

# Verify checksums
find models -type f -name "*.sha256" -print0 | while IFS= read -r -d '' s; do
  sha256sum -c "$s" || { echo "Checksum failed for $s"; exit 1; }
done

Minimal model registry folder structure:

models/
  2024-07-01/
    model.pt
    model.pt.sha256
    metadata.yaml   # training data snapshot, commit hash, metrics
latest -> 2024-07-01

Checklist 3: CI/CD quality gates for ML

Actions

  • Run unit tests and fast data contracts on every PR.

  • Fail the build if the new model underperforms a baseline.

  • Build and push a container only after checks pass.

Evaluation gate script:

# ci/ci_evaluate.sh
#!/usr/bin/env bash
set -euo pipefail
# Expect a metrics.json with {"f1": 0.842, "latency_ms_p95": 42.1}
baseline_f1="${BASELINE_F1:-0.82}"
max_p95_ms="${MAX_P95_MS:-100}"

f1=$(jq -r '.f1' metrics.json)
p95=$(jq -r '.latency_ms_p95' metrics.json)

echo "Baseline F1: $baseline_f1 | New F1: $f1"
echo "Max p95: $max_p95_ms | New p95: $p95"

awk "BEGIN{exit !($f1 >= $baseline_f1)}" || { echo "F1 regression"; exit 1; }
awk "BEGIN{exit !($p95 <= $max_p95_ms)}" || { echo "Latency regression"; exit 1; }
echo "Metrics gate passed ✅"

Example GitHub Actions workflow:

# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
  test-and-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install OS deps
        run: |
          sudo apt update
          sudo apt install -y python3 python3-venv pipx jq
          pipx ensurepath
      - name: Setup venv
        run: |
          python3 -m venv .venv
          . .venv/bin/activate
          pip install -U pip wheel
          pip install -r requirements.txt
      - name: Unit tests
        run: . .venv/bin/activate && pytest -q
      - name: Evaluate model
        run: |
          . .venv/bin/activate
          python scripts/eval.py --out metrics.json
          bash ci/ci_evaluate.sh
      - name: Build container
        run: docker build -t ghcr.io/yourorg/yourapp:${{ github.sha }} .

Checklist 4: Observability for inference (logs + metrics)

Actions

  • Use structured JSON logs for easy grep/jq.

  • Expose Prometheus metrics for latency and error rates.

  • Keep everything debuggable from Bash.

Structured logs example (Python):

# src/logging_setup.py
import json, sys, time

def log(event, **kw):
    print(json.dumps({"ts": time.time(), "event": event, **kw}), file=sys.stdout, flush=True)

Use in your service:

from logging_setup import log
try:
    start = time.time()
    y = model.predict(x)
    log("inference_ok", latency_ms=1000*(time.time()-start), batch=len(x))
except Exception as e:
    log("inference_err", error=str(e))
    raise

Slice and dice logs on Linux:

# Show p95 latency over last 200 lines
tail -n 200 service.log | jq -r 'select(.event=="inference_ok") | .latency_ms' | sort -n | awk 'NR==int(NR*0.95)'

Minimal Prometheus exporter (Python):

# src/metrics.py
from prometheus_client import Counter, Histogram, start_http_server
inferences = Counter("inferences_total", "Total inferences")
latency = Histogram("inference_latency_ms", "Inference latency (ms)", buckets=[5,10,20,50,100,200,500,1000])

def observe_inference(ms): inferences.inc(); latency.observe(ms)

if __name__ == "__main__":
    start_http_server(8000)  # /metrics
    import time, random
    while True:
        observe_inference(random.uniform(5, 100))
        time.sleep(1)

Run:

python -m pip install prometheus-client
python src/metrics.py
# curl http://localhost:8000/metrics

Checklist 5: Secrets and compliance hygiene

Actions

  • Keep secrets out of Git; inject at runtime via environment or container secrets.

  • Add pre-commit hooks to block obvious mistakes.

  • Document who can rotate what and how.

Pre-commit with secret detection:

pipx install pre-commit
cat > .pre-commit-config.yaml <<'YAML'
repos:

- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.6.0
  hooks:
    - id: detect-private-key
    - id: end-of-file-fixer
    - id: trailing-whitespace

- repo: https://github.com/Yelp/detect-secrets
  rev: v1.4.0
  hooks:
    - id: detect-secrets
YAML
pre-commit install

Containerized secrets (Podman example):

# Create a secret from stdin and mount as env var API_KEY
printf "%s" "super-secret" | podman secret create api_key -
podman run --rm --secret api_key,type=env,target=API_KEY \
  -e OTHER_CONFIG=prod docker.io/library/python:3.11-slim \
  python -c 'import os; print(os.getenv("API_KEY"))'

Systemd env files for services:

# /etc/yourapp/yourapp.env (root-readable only)
API_KEY=super-secret
MODEL_DIR=/var/lib/yourapp/models

# yourapp.service snippet:
EnvironmentFile=/etc/yourapp/yourapp.env

Real-world example flow (pulling it all together)

1) Developer runs:

git clone https://github.com/yourorg/yourapp.git
cd yourapp
make check
make venv
make test

2) Train and snapshot model:

. .venv/bin/activate
python scripts/train.py --out models/2024-07-01/model.pt
sha256sum models/2024-07-01/model.pt > models/2024-07-01/model.pt.sha256
ln -sfn 2024-07-01 models/latest
git add models/.gitattributes metadata/  # binaries tracked via LFS
git commit -m "Model 2024-07-01 with F1=0.845"

3) CI runs unit tests and ci_evaluate.sh, builds/pushes the image if metrics pass.

4) Prod deploy mounts secrets via Podman or systemd env file, exposes /metrics, and logs in JSON for jq-friendly troubleshooting.

Conclusion and next steps

Checklists turn tribal AI ops knowledge into repeatable safety rails. Start small:

  • Today: add scripts/checklist_env.sh and a Makefile.

  • This week: lock dependencies and enable Git LFS + checksums for models.

  • This sprint: wire up CI gates and structured logs with a minimal /metrics endpoint.

Need a head start? Copy the snippets above into your repo’s scripts/, ci/, and src/ folders. Run make check, fix any gaps it reports, and keep iterating. Your future self (and on-call engineer) will thank you.

Related posts