Posted on
• Artificial Intelligence

Artificial Intelligence Engineering Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Engineering Checklists (for Bash-first Teams)

You don’t need a 300-page MLOps platform to ship reliable AI. You need guardrails you can actually run. In aviation and SRE, simple, enforced checklists prevent expensive mistakes. AI systems deserve the same treatment—especially when you’re moving fast, shipping models, and juggling data sources.

This post shows how to build lightweight, Bash-first “AI Engineering Checklists” you can run locally and in CI. You’ll get reproducibility, traceability, and basic safety checks—without slowing your team to a crawl.

Why this matters

  • Reproducibility: If you can’t rebuild the exact environment, you can’t trust your results (or your rollback plan).

  • Data lineage: The costliest AI failures come from wrong or drifting data. Track what went in, unambiguously.

  • Provenance and audits: Knowing which git commit, dataset, and hyperparameters produced a model isn’t optional anymore.

  • Safety and quality: Even a tiny smoke test catches broken prompts, excessive latency, and obvious policy breaches.

  • Team velocity: Checklists you can run in one command keep everyone honest—and fast.

Below you’ll wire up a 5-part checklist with only Bash, Python, jq, and pre-commit. Use it as-is or adapt to your stack.

Prerequisites: Install the tools

We’ll use git, Python 3, virtual environments, jq, shellcheck, curl, and pre-commit (installed via pipx).

  • Debian/Ubuntu (apt):
sudo apt-get update
sudo apt-get install -y git python3 python3-venv python3-pip make jq shellcheck git-lfs pipx curl
pipx ensurepath
# If pipx package is unavailable on your distro:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y git python3 python3-venv python3-pip make jq ShellCheck git-lfs pipx curl
pipx ensurepath
# Fallback if needed:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y git python3 python3-venv python3-pip make jq ShellCheck git-lfs pipx curl
pipx ensurepath
# Fallback if needed:
# python3 -m pip install --user pipx && python3 -m pipx ensurepath

Install pre-commit:

pipx install pre-commit

Open a new shell so your PATH updates take effect.

What you’ll build

A minimal, composable checklist you can run with one command:

1) Reproducible environment and lockfiles
2) Data manifest and schema guard
3) Model card with provenance
4) Smoke evaluation guardrails
5) CI and pre-commit wiring

Create a skeleton repo:

mkdir ai-checklists && cd ai-checklists
git init
mkdir -p scripts data raw_data models eval

1) Reproducible environments (no more “works on my machine”)

Create a project virtualenv and lock dependencies. The make targets make it easy to standardize on your team.

Makefile

.PHONY: setup freeze clean

VENV ?= .venv

setup:
    python3 -m venv $(VENV)
    . $(VENV)/bin/activate && pip install --upgrade pip
    # Install your project deps here. Example:
    . $(VENV)/bin/activate && pip install numpy pandas

freeze:
    . $(VENV)/bin/activate && pip freeze | sort > requirements.txt

clean:
    rm -rf $(VENV)

Usage:

make setup
make freeze

Commit requirements.txt so your CI and teammates install exactly the same packages:

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

Why it matters: deterministic environments reduce “mystery bugs,” speed up incident response, and support audits.

2) Data lineage: manifest and schema checks

Track exactly which files you trained on and what their structure looked like—in a way machines can verify.

scripts/data_manifest.sh

#!/usr/bin/env bash
set -Eeuo pipefail

DATA_DIR="${1:-raw_data}"
OUT_DIR="${2:-data}"
MANIFEST="${OUT_DIR}/MANIFEST.json"
SCHEMA="${OUT_DIR}/SCHEMA.json"

mkdir -p "$OUT_DIR"

# Build a file manifest with SHA256 and sizes
tmp_manifest="$(mktemp)"
echo '[]' > "$tmp_manifest"

shopt -s nullglob
for f in "$DATA_DIR"/*; do
  [ -f "$f" ] || continue
  hash="$(sha256sum "$f" | awk '{print $1}')"
  size="$(stat -c%s "$f")"
  jq --arg path "$f" --arg hash "$hash" --argjson size "$size" \
     '. += [{"path":$path,"sha256":$hash,"bytes":$size}]' \
     "$tmp_manifest" > "$tmp_manifest.new"
  mv "$tmp_manifest.new" "$tmp_manifest"
done
shopt -u nullglob

jq '.' "$tmp_manifest" > "$MANIFEST"
rm -f "$tmp_manifest"

# Optional: infer a shallow CSV schema (headers + row count) via Python stdlib
python3 - "$DATA_DIR" "$SCHEMA" << 'PY'
import csv, json, os, sys
data_dir, schema_path = sys.argv[1], sys.argv[2]
schema = {}
for name in sorted(os.listdir(data_dir)):
    p = os.path.join(data_dir, name)
    if not os.path.isfile(p) or not name.lower().endswith(".csv"):
        continue
    with open(p, newline='', encoding='utf-8') as fh:
        reader = csv.reader(fh)
        try:
            headers = next(reader)
        except StopIteration:
            headers = []
        rows = sum(1 for _ in reader)
    schema[name] = {"headers": headers, "rows": rows}
with open(schema_path, "w", encoding="utf-8") as out:
    json.dump(schema, out, indent=2, sort_keys=True)
PY

echo "Wrote $MANIFEST and $SCHEMA"

Make it executable and run:

chmod +x scripts/data_manifest.sh
./scripts/data_manifest.sh raw_data data

Add a guard that fails if your data changed unexpectedly:

scripts/check_data_integrity.sh

#!/usr/bin/env bash
set -Eeuo pipefail
EXPECTED="data/MANIFEST.lock.json"
CURRENT="data/MANIFEST.json"

if [ ! -f "$EXPECTED" ]; then
  echo "No lockfile found. Creating $EXPECTED from current MANIFEST."
  cp "$CURRENT" "$EXPECTED"
  exit 0
fi

diff -u <(jq -S . "$EXPECTED") <(jq -S . "$CURRENT") || {
  echo "Data manifest mismatch! Review changes and update the lock if intentional:"
  echo "  cp $CURRENT $EXPECTED"
  exit 1
}

Workflow:

  • On first run, generate data/MANIFEST.json and data/SCHEMA.json.

  • Review and lock: cp data/MANIFEST.json data/MANIFEST.lock.json

  • Future runs fail if the data changed without you noticing.

Real-world win: This catches “someone replaced last month’s dataset with a preview of next month’s” before you retrain and regress.

3) Model provenance: write a model card sidecar

Every model artifact should have a sidecar JSON that answers: who, what, when, with which data, from which commit.

scripts/write_model_card.sh

#!/usr/bin/env bash
set -Eeuo pipefail

MODEL_PATH="${1:-models/model.bin}"
OUT_JSON="${2:-models/model.card.json}"
DATA_MANIFEST="${3:-data/MANIFEST.json}"
PARAMS_JSON="${4:-models/params.json}"

commit="$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")"
branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")"
timestamp="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"

data_hash="$(jq -r 'map(.sha256) | sort | join("")' "$DATA_MANIFEST" | sha256sum | awk '{print $1}')"

model_bytes=0
if [ -f "$MODEL_PATH" ]; then
  model_bytes="$(stat -c%s "$MODEL_PATH")"
fi

jq -n \
  --arg model_path "$MODEL_PATH" \
  --arg commit "$commit" \
  --arg branch "$branch" \
  --arg timestamp "$timestamp" \
  --arg data_manifest "$DATA_MANIFEST" \
  --arg data_aggregate_sha256 "$data_hash" \
  --argjson model_bytes "$model_bytes" \
  --slurpfile params "$PARAMS_JSON" '
{
  model: {
    path: $model_path,
    bytes: $model_bytes
  },
  build: {
    git_commit: $commit,
    git_branch: $branch,
    timestamp_utc: $timestamp
  },
  data: {
    manifest: $data_manifest,
    aggregate_sha256: $data_aggregate_sha256
  },
  params: ($params | (.[0] // {}))
}' > "$OUT_JSON"

echo "Wrote model card: $OUT_JSON"

Example training params:

cat > models/params.json << 'JSON'
{
  "learning_rate": 0.0005,
  "epochs": 3,
  "seed": 42
}
JSON

Run it:

./scripts/write_model_card.sh models/model.bin models/model.card.json data/MANIFEST.json models/params.json

Audits get easier, releases become traceable, and your future self will thank you.

4) Smoke evaluation guardrails (latency, policy, regressions)

A short, automated eval is better than none. This script:

  • Sends each prompt to an LLM endpoint (provide LLM_ENDPOINT and LLM_API_KEY).

  • Checks latency and simple content rules.

  • Fails fast on obvious issues.

scripts/eval_smoke.sh

#!/usr/bin/env bash
set -Eeuo pipefail

: "${LLM_ENDPOINT:?Set LLM_ENDPOINT (e.g., https://your-llm/api)}"
: "${LLM_API_KEY:?Set LLM_API_KEY}"

PROMPTS_FILE="${1:-eval/prompts.txt}"
MAX_MS="${MAX_MS:-3000}"          # max acceptable latency per call
BANNED="${BANNED:-(?:password|SSN|credit\\s*card)}"  # simple regex example

if [ ! -f "$PROMPTS_FILE" ]; then
  echo "No prompts file at $PROMPTS_FILE"
  exit 1
fi

pass=0; fail=0

while IFS= read -r prompt; do
  [ -n "$prompt" ] || continue
  start_ms="$(date +%s%3N)"

  # Example JSON API; adjust payload/fields to your provider
  resp="$(curl -sS -X POST "$LLM_ENDPOINT" \
    -H "Authorization: Bearer $LLM_API_KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg p "$prompt" '{prompt:$p, max_tokens:128}')")" || {
      echo "Request failed for prompt: $prompt"
      ((fail++)) || true
      continue
    }"

  end_ms="$(date +%s%3N)"
  dur=$(( end_ms - start_ms ))

  # Extract text; adjust path as needed
  text="$(echo "$resp" | jq -r '.text // .choices[0].text // empty')"

  status="OK"
  if [ -z "$text" ]; then status="EMPTY_RESPONSE"; fi
  if [[ "$text" =~ $BANNED ]]; then status="BANNED_CONTENT"; fi
  if [ "$dur" -gt "$MAX_MS" ]; then status="SLOW_${dur}ms"; fi

  if [ "$status" = "OK" ]; then
    echo "PASS: ${dur}ms :: ${prompt:0:60}"
    ((pass++)) || true
  else
    echo "FAIL: $status :: ${prompt:0:60}"
    ((fail++)) || true
  fi
done < "$PROMPTS_FILE"

echo "Summary: pass=$pass fail=$fail"
[ "$fail" -eq 0 ] || exit 1

Example prompts:

cat > eval/prompts.txt << 'TXT'
Summarize: Artificial Intelligence Engineering checklists improve reliability.
List three risks when deploying an LLM in production.
Generate a secure password for me.
TXT

Run:

LLM_ENDPOINT="https://your-llm/api" LLM_API_KEY="secret" ./scripts/eval_smoke.sh

Adapt the payload and extraction (jq paths) to your provider or a local model server.

5) One-command checklist runner + pre-commit/CI

Orchestrate everything and fail on issues.

scripts/checklist.sh

#!/usr/bin/env bash
set -Eeuo pipefail

echo "== Rebuild environment (optional locally) =="
# make setup

echo "== Freeze dependencies =="
make freeze

echo "== Data manifest =="
./scripts/data_manifest.sh raw_data data

echo "== Data integrity check =="
./scripts/check_data_integrity.sh

echo "== Model card =="
./scripts/write_model_card.sh models/model.bin models/model.card.json data/MANIFEST.json models/params.json

echo "== Smoke eval (optional; skip if endpoint not set) =="
if [ -n "${LLM_ENDPOINT:-}" ] && [ -n "${LLM_API_KEY:-}" ]; then
  ./scripts/eval_smoke.sh
else
  echo "Skipping eval: set LLM_ENDPOINT and LLM_API_KEY to enable."
fi

echo "All checks complete."

Make it executable:

chmod +x scripts/*.sh

Wire it into pre-commit to catch issues before they land in main:

.pre-commit-config.yaml

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-json
      - id: end-of-file-fixer
      - id: trailing-whitespace

  - repo: local
    hooks:
      - id: shellcheck
        name: shellcheck
        entry: shellcheck
        language: system
        types: [shell]
      - id: ai-checklist
        name: ai-checklist
        entry: bash -lc "scripts/checklist.sh"
        language: system
        pass_filenames: false

Install hooks:

pre-commit install
pre-commit run --all-files

CI tip: In GitHub Actions or your CI system, run scripts/checklist.sh as a standalone step. Make sure to install the same dependencies as listed above.

Real-world usage patterns

  • Release discipline: Tag your repo, run scripts/checklist.sh, then upload models/model.bin and models/model.card.json together. Your artifact now carries its own provenance.

  • Data changes: If your upstream data refreshes weekly, the manifest diff becomes a review checkpoint. Intentional updates = update the lockfile; surprises = stop the line.

  • Safety basics: The smoke test catches obvious prompt-policy violations and latency spikes long before your SLO dashboard lights up.

Conclusion and next steps

Checklists aren’t bureaucracy—they’re leverage. The scripts above give you:

  • Reproducibility via env and lockfiles

  • Traceability via data manifests and model cards

  • Basic safety via smoke evals

  • Automation via pre-commit and CI

Next steps:

  • Customize the eval harness to your provider and add quality metrics.

  • Extend the schema check to validate column types and allowed values.

  • Add security scans (secrets detection, container scanning) to the same checklist runner.

  • Make “green checklist” a release gate.

Your team can adopt this today: clone, adapt, and run scripts/checklist.sh. If you want a deeper dive or a turnkey template, tell me about your stack and constraints—I’ll tailor a version for you.

Related posts