Posted on
• Artificial Intelligence

Artificial Intelligence Change Management

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Change Management (for Bash-first teams)

AI is moving faster than your change board. A single prompt tweak can swing conversion rates, a model upgrade can double costs overnight, and a silent API deprecation can break production at 3 a.m. The value of AI is real—but so is the operational risk. This article shows how to bring disciplined, Linux-first change management to your AI stack using Git, Bash, and a handful of standard tools.

What you’ll get:

  • A lightweight toolkit to inventory, gate, deploy, canary-test, and roll back AI changes

  • Scripts you can drop into any repo

  • Installation instructions for apt, dnf, and zypper

Why AI needs explicit change management

  • Models drift and APIs change: Outputs shift as providers retrain, update defaults, or deprecate endpoints.

  • Prompts are production code: A “minor” wording change can materially change behavior, safety, and cost.

  • Reproducibility is hard: Data, hyperparameters, and context windows all influence outputs.

  • Compliance and auditability: You must know which model, prompt, and settings were live when a decision was made.

  • Blast radius control: Canary and staged rollouts prevent organization-wide regressions.

Prerequisites

Tools we’ll use: git, curl, jq, inotify-tools, tree, make

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git curl jq inotify-tools tree make

Fedora/RHEL (dnf):

sudo dnf install -y git curl jq inotify-tools tree make

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git curl jq inotify-tools tree make

Optional universal installer that detects your package manager:

#!/usr/bin/env bash
set -euo pipefail
pkgs=(git curl jq inotify-tools tree make)
if command -v apt-get >/dev/null 2>&1; then
  sudo apt-get update
  sudo apt-get install -y "${pkgs[@]}"
elif command -v dnf >/dev/null 2>&1; then
  sudo dnf install -y "${pkgs[@]}"
elif command -v zypper >/dev/null 2>&1; then
  sudo zypper refresh
  sudo zypper install -y "${pkgs[@]}"
else
  echo "Unsupported package manager. Install: ${pkgs[*]}" >&2
  exit 1
fi

Repo layout to copy

Keep your AI assets predictable:

ai/
  prompts/
    onboarding_v2.txt
    onboarding_v3.txt
    current.txt -> onboarding_v3.txt
  configs/
    config.json
  datasets/
    canary_inputs.jsonl
  scripts/
    ai-inventory.sh
    deploy.sh
    ai-router.sh
    ai-canary-check.sh
CHANGELOG.md

Example config (ai/configs/config.json):

{
  "model_id": "acme/gpt-1.4",
  "prompt_version": "onboarding_v3.txt",
  "temperature": 0.2,
  "endpoint": "http://localhost:8000/infer"
}

1) Inventory the AI surface area (and checksum it)

Know exactly what’s in production: prompts, configs, and datasets. This creates a JSON “bill of materials” you can pin to a release.

ai/scripts/ai-inventory.sh:

#!/usr/bin/env bash
set -euo pipefail

cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"

mkdir -p ai/artifacts

find ai/prompts ai/configs ai/datasets -type f -print0 \
| while IFS= read -r -d '' f; do
    sha=$(sha256sum "$f" | awk '{print $1}')
    printf '%s\t%s\n' "$f" "$sha"
  done \
| jq -R -s '
    split("\n")[:-1]
    | map(split("\t"))
    | { generated_at: now | todate,
        git_rev: (input_filename | sub(".+"; "'"$(git rev-parse --short HEAD 2>/dev/null || echo unknown)"'")),
        files: map({path: .[0], sha256: .[1]}) }' \
> ai/artifacts/inventory.json

echo "Wrote ai/artifacts/inventory.json"

Run it after each meaningful change:

bash ai/scripts/ai-inventory.sh
git add ai/artifacts/inventory.json

2) Put guardrails on prompt and model changes (Git hooks)

Treat prompts/configs as production code. Require a change ticket for any prompt change.

Create .git/hooks/commit-msg:

#!/usr/bin/env bash
set -euo pipefail
msg_file="$1"

# If prompts changed, require PROMPT-<id> in commit message
if git diff --cached --name-only | grep -q '^ai/prompts/'; then
  if ! grep -Eq 'PROMPT-[0-9]+' "$msg_file"; then
    echo "ERROR: Prompt changes require a PROMPT-<id> reference in the commit message." >&2
    exit 1
  fi
fi

# If configs changed, require CONFIG-<id>
if git diff --cached --name-only | grep -q '^ai/configs/'; then
  if ! grep -Eq 'CONFIG-[0-9]+' "$msg_file"; then
    echo "ERROR: Config changes require a CONFIG-<id> reference in the commit message." >&2
    exit 1
  fi
fi

Make it executable:

chmod +x .git/hooks/commit-msg

You can extend this with automated checks (e.g., max prompt size, allowed temperature range) using jq:

jq -e '.temperature >= 0 and .temperature <= 1' ai/configs/config.json >/dev/null \
  || { echo "Temperature out of range (0..1)"; exit 1; }

3) Staged rollouts with feature flags and canaries

Use a symlink to point to the active prompt and an environment flag to control model versions. Then canary a small percentage before a full cutover.

ai/scripts/deploy.sh:

#!/usr/bin/env bash
set -euo pipefail

CFG="ai/configs/config.json"
PROMPTS_DIR="ai/prompts"

model_id=$(jq -r '.model_id' "$CFG")
prompt_ver=$(jq -r '.prompt_version' "$CFG")

ln -sfn "$prompt_ver" "$PROMPTS_DIR/current.txt"

stamp="releases/$(date -u +'%Y%m%dT%H%M%SZ')_${model_id//\//-}_$prompt_ver"
mkdir -p releases
git rev-parse --short HEAD > "$stamp.gitrev"
cp "$CFG" "$stamp.config.json"

echo "Deployed -> model: $model_id | prompt: $prompt_ver"
echo "Symlink: $PROMPTS_DIR/current.txt -> $prompt_ver"

Basic 5% canary router (route a fraction of traffic to a new model):

#!/usr/bin/env bash
# ai/scripts/ai-router.sh
set -euo pipefail

CFG="ai/configs/config.json"
ENDPOINT=$(jq -r '.endpoint' "$CFG")

BASE_MODEL=$(jq -r '.model_id' "$CFG")
NEW_MODEL="${NEW_MODEL_ID:-}"     # set via env to activate canary
CANARY_PCT="${CANARY_PCT:-5}"     # e.g., 5%

read -r input
model="$BASE_MODEL"
if [[ -n "$NEW_MODEL" ]]; then
  r=$(( RANDOM % 100 ))
  if (( r < CANARY_PCT )); then
    model="$NEW_MODEL"
  fi
fi

payload=$(jq -n --arg m "$model" --arg i "$input" '{model: $m, input: $i}')

curl -sS -H 'Content-Type: application/json' -d "$payload" "$ENDPOINT"

Usage:

export NEW_MODEL_ID="acme/gpt-1.4"
export CANARY_PCT=5
echo "Summarize: The user ordered 3 widgets." | bash ai/scripts/ai-router.sh

Watch-and-reload when the active prompt changes:

inotifywait -m -e close_write,move,create,delete ai/prompts/current.txt \
  | while read -r _; do
      pkill -HUP your-ai-service-binary || true
      echo "Reloaded service after prompt change"
    done

4) Drift monitoring with a simple canary probe

Continuously check latency and basic output sanity on known inputs. Alert early.

ai/scripts/ai-canary-check.sh:

#!/usr/bin/env bash
set -euo pipefail

CFG="ai/configs/config.json"
ENDPOINT=${ENDPOINT:-"$(jq -r '.endpoint' "$CFG")"}
CANARY="ai/datasets/canary_inputs.jsonl"
LOG="ai/artifacts/canary.log"
mkdir -p "$(dirname "$LOG")"

while IFS= read -r line; do
  [[ -z "$line" ]] && continue
  input=$(jq -r '.input' <<<"$line")
  http_and_time=$(curl -s -o /tmp/resp.json -w '%{http_code} %{time_total}' \
    -H 'Content-Type: application/json' \
    -d "$(jq -n --arg i "$input" '{input: $i}')" \
    "$ENDPOINT") || http_and_time="000 999"
  code=$(awk '{print $1}' <<<"$http_and_time")
  tsec=$(awk '{print $2}' <<<"$http_and_time")

  out_len=$(jq -r '.output | tostring | length' /tmp/resp.json 2>/dev/null || echo 0)

  status="OK"
  if [[ "$code" != "200" ]] || (( $(echo "$tsec > 2.0" | bc -l) )) || (( out_len < 4 )); then
    status="ALERT"
  fi

  printf '%s\tcode=%s\tlat=%.3fs\tlen=%s\tstatus=%s\n' \
    "$(date -u +'%FT%TZ')" "$code" "$tsec" "$out_len" "$status" | tee -a "$LOG"
done < "$CANARY"

Example canary inputs (ai/datasets/canary_inputs.jsonl):

{"input":"Say 'OK'."}
{"input":"Return a JSON object with key 'ready' true."}
{"input":"What is 2+2?"}

Run it via cron/systemd and alert on “ALERT” lines.

5) Auditable releases and quick rollbacks

Tag releases, ship an inventory, and keep a one-command rollback.

Create a release:

bash ai/scripts/ai-inventory.sh
bash ai/scripts/deploy.sh
git add ai/artifacts/inventory.json ai/configs/config.json releases/
git commit -m "CONFIG-123 release with acme/gpt-1.4 and onboarding_v3"
git tag -a "ai-$(date -u +'%Y%m%dT%H%M%SZ')" -m "AI release"
git push --tags

Rollback to the previous tag:

last_tag=$(git tag --list 'ai-*' --sort=-creatordate | sed -n '2p')
git checkout "$last_tag" -- ai/configs/config.json ai/prompts/
jq -r '.prompt_version' ai/configs/config.json | xargs -I{} ln -sfn "{}" ai/prompts/current.txt
git commit -am "Rollback to $last_tag"

Append a human-readable audit entry:

echo "$(date -u +'%F %T')Z  Rolled back to $last_tag due to elevated latency in canary" >> CHANGELOG.md
git add CHANGELOG.md && git commit -m "Doc: rollback rationale"

Real-world example: upgrading a model safely

  • Context: Your onboarding assistant is on acme/gpt-1.3 with onboarding_v2.txt. You want acme/gpt-1.4 and onboarding_v3.txt.

  • Plan:

    • Update ai/configs/config.json with model_id and prompt_version.
    • Run deploy.sh to update the symlink and stamp the release.
    • Set NEW_MODEL_ID to gpt-1.4 and CANARY_PCT to 5 for canary routing.
    • Run ai-canary-check.sh for 30 minutes. If stable, raise CANARY_PCT to 50, then 100.
    • Tag the release and record the inventory for audit.
    • If latency spikes or quality drops, run the rollback recipe.

Conclusion and next steps

AI change management doesn’t require a new platform—just consistent, automatable practices. Start small:

  • Add the inventory script and commit hook today.

  • Route 5% of traffic to your next model behind a simple Bash router.

  • Monitor drift with a canary probe and keep a clean rollback path.

Call to action: 1) Copy the repo layout and scripts into your project. 2) Install the prerequisites using apt, dnf, or zypper. 3) Pilot a canary for your next prompt or model change. 4) Share results with your team and codify them in your runbooks.

If you want a ready-to-fork starter, package these scripts into a Makefile target and wire them to your CI. Your future 3 a.m. self will thank you.

Related posts