Posted on
• Artificial Intelligence

Artificial Intelligence Backup Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Backup Checklists: A Bash-first Guide to Never Losing Your Models, Data, or Mind

If your GPU dies tonight or your cloud project gets accidentally deleted, how quickly can you rebuild your AI stack? In machine learning and LLM work, “the project” isn’t just code—it’s datasets, model weights, experiment metadata, prompts, feature stores, and the exact environment that produced your results. Without a solid, automatable backup checklist, you’re betting your research and product timelines on luck.

This guide gives you a practical, Bash-centric backup checklist for AI projects, with concrete scripts you can drop into your workflow. You’ll learn what to back up, how to store it securely, and how to automate verification and restores.

Why an AI-specific backup checklist?

  • AI artifacts are large, diverse, and volatile: datasets, pre-trained weights, fine-tuned checkpoints, vector indexes, and logs are all moving targets.

  • Reproducibility and compliance matter: you need to prove what data and config produced a model version, sometimes under regulation.

  • Time is money: recomputing models or re-labeling data is costly; losing experiment history is worse.

  • Security is non-negotiable: sensitive data and secrets demand encryption at rest and in transit.

The AI Backup Checklist (what to protect)

Use this inventory to define your backup scope:

  • Source and Infrastructure as Code: git repos, CI/CD configs, IaC (Terraform/Ansible), notebooks.

  • Datasets and Data Manifests: raw, processed, and split data; schema; data versions or manifests (hash lists).

  • Models and Artifacts: pre-trained weights, fine-tuned checkpoints, tokenizer files, embeddings, vector store snapshots.

  • Experiments and Metadata: hyperparameters, training configs, metrics, run logs (e.g., W&B), random seeds, commit hashes.

  • Environment and Dependencies: requirements.txt, conda env export, container image references/Dockerfile, CUDA/cuDNN versions.

  • Secrets and Access: do not store secrets in backups; store references to where they live (Vault/KMS). Keep backup encryption keys separate.

  • Orchestration and Infra Snapshots: job configs, SLURM specs, Kubernetes manifests, storage class definitions.

Tip: Keep a human-readable manifest at the root of each project that lists paths to all of the above. Back it up with everything else.

Tools you’ll use (install once)

We’ll lean on proven, scriptable tools:

  • rsync: fast local copies

  • restic: deduplicated, encrypted backups to local or object storage (S3, B2, etc.)

  • borgbackup: deduplicated backups over SSH/NAS

  • rclone: mirrors and moves data across clouds

  • jq: compose small manifests in JSON

Install them with your distro’s package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y rsync rclone restic borgbackup jq
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y rsync rclone restic borgbackup jq
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y rsync rclone restic borgbackup jq

No extra package is required for systemd timers (present on modern distros).

1) Build your AI backup manifest (lightweight, provable)

Create a quick, reproducible manifest that captures versions, hashes, and environment. Run this at the start of each backup.

#!/usr/bin/env bash
set -euo pipefail

PROJECT_ROOT="${1:-$PWD}"
OUT_DIR="${PROJECT_ROOT}/.backup-meta"
mkdir -p "$OUT_DIR"

# Capture Python dependency state
if command -v python >/dev/null 2>&1 && python -c "import pkgutil" >/dev/null 2>&1; then
  python -m pip freeze > "${OUT_DIR}/requirements.lock" || true
fi

# Conda env (if present)
if command -v conda >/dev/null 2>&1; then
  conda env export > "${OUT_DIR}/conda-env.yaml" || true
fi

# Git state
if command -v git >/dev/null 2>&1 && [ -d "${PROJECT_ROOT}/.git" ]; then
  git -C "$PROJECT_ROOT" rev-parse HEAD > "${OUT_DIR}/git-commit.txt" || true
  git -C "$PROJECT_ROOT" status --porcelain=v1 > "${OUT_DIR}/git-status.txt" || true
  git -C "$PROJECT_ROOT" remote -v > "${OUT_DIR}/git-remotes.txt" || true
fi

# Data manifest (hash a sample of large files to keep it fast)
find "$PROJECT_ROOT/data" -type f 2>/dev/null | head -n 200 | sort | xargs -r sha256sum > "${OUT_DIR}/data.sha256"

# Minimal machine info
uname -a > "${OUT_DIR}/system.txt" || true
nvidia-smi -L > "${OUT_DIR}/gpus.txt" 2>/dev/null || true

# JSON index for tooling
jq -n \
  --arg time "$(date -Is)" \
  --arg project "$PROJECT_ROOT" \
  --arg commit "$(cat "${OUT_DIR}/git-commit.txt" 2>/dev/null || echo unknown)" \
  '{generated_at:$time, project:$project, git_commit:$commit}' \
  > "${OUT_DIR}/index.json"

What this gives you:

  • Fast-to-compare checksums of representative data

  • Exact dependency fingerprints

  • Git provenance to map models to code

2) Pick a storage pattern that won’t betray you

Follow 3-2-1:

  • 3 copies of your data

  • 2 different media (e.g., object storage and local disk/NAS)

  • 1 offsite/immutable copy

Common patterns:

  • Primary encrypted restic repo in S3-compatible storage (S3, B2, MinIO)

  • Secondary borg repo on an SSH-accessible NAS

  • Optional local rsync mirror for quick restores

Security:

  • Use restic’s built-in encryption

  • Store encryption password in a root-only file on a separate host or secrets manager

  • For S3, enable Object Lock/Versioning if available to resist accidental deletion or ransomware

3) Script the backup (Bash + restic + rsync)

This script:

  • Builds the manifest

  • Backs up project directories to restic (S3 or local)

  • Keeps a local rsync mirror for fast restores

  • Applies exclude rules and retention

  • Runs health checks

#!/usr/bin/env bash
set -euo pipefail

# Required: set these environment variables ahead of time (export or a .env file sourced here)
# export RESTIC_REPOSITORY="s3:https://s3.us-east-1.amazonaws.com/my-bucket/ai-backups"
# export RESTIC_PASSWORD_FILE="$HOME/.config/restic/ai.pw"      # chmod 600
# For S3-compatible: export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION

PROJECTS=(
  "$HOME/projects/llm-finetune"
  "$HOME/projects/vision-model"
)

LOCAL_MIRROR="/mnt/backup-mirror/ai"
TAGS="ai,$(hostname -s)"
EXCLUDES=(
  "--exclude" "**/__pycache__"
  "--exclude" "**/.mypy_cache"
  "--exclude" "**/.ruff_cache"
  "--exclude" "**/.venv"
  "--exclude" "**/.ipynb_checkpoints"
  "--exclude" "**/wandb/tmp"
  "--exclude" "**/*.tmp"
  "--exclude" "**/*.log"
  # Do not back up secrets files directly; store them in a vault.
  "--exclude" "**/.env"
  "--exclude" "**/*.pem"
  "--exclude" "**/id_rsa*"
)

# Init repo if needed
restic snapshots >/dev/null 2>&1 || restic init

for P in "${PROJECTS[@]}"; do
  [ -d "$P" ] || continue
  echo ">> Preparing manifest for $P"
  "$(dirname "$0")/ai-build-manifest.sh" "$P" || true

  echo ">> Restic backing up $P"
  restic backup "${EXCLUDES[@]}" --tag "$TAGS" "$P"

  echo ">> Local mirror rsync for $P"
  mkdir -p "$LOCAL_MIRROR"
  rsync -aHAX --delete --info=stats1 \
    --exclude '__pycache__' --exclude '.venv' --exclude '.ipynb_checkpoints' \
    "$P/" "$LOCAL_MIRROR/$(basename "$P")/"
done

echo ">> Apply retention (7 daily, 4 weekly, 6 monthly)"
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

echo ">> Integrity check (lightweight)"
restic check --read-data-subset=1% || true

echo ">> Done"

Notes:

  • For a purely local setup, point RESTIC_REPOSITORY to a path like /srv/restic/ai.

  • For an SSH/NAS alternative, see the borg snippet below.

Optional: Borg over SSH/NAS

#!/usr/bin/env bash
set -euo pipefail

export BORG_REPO="ssh://user@nas.lan:22/volume/backups/ai"
export BORG_PASSPHRASE="$(cat $HOME/.config/borg/ai.pw)"

borg init --encryption=repokey-blake2 || true

borg create --stats --progress \
  ::'{hostname}-{now:%Y-%m-%d_%H-%M}' \
  "$HOME/projects/llm-finetune" \
  "$HOME/projects/vision-model" \
  --exclude '**/.venv' --exclude '**/__pycache__' --exclude '**/.ipynb_checkpoints' \
  --exclude '**/.env' --exclude '**/*.pem'

borg prune --keep-daily=7 --keep-weekly=4 --keep-monthly=6
borg check --verify-data --max-duration=120 || true

4) Automate and verify with systemd timers

Use user-level systemd units (no root needed). Save these files and enable.

~/.config/systemd/user/ai-backup.service

[Unit]
Description=AI project backup (restic + rsync)

[Service]
Type=oneshot
Environment=RESTIC_REPOSITORY=s3:https://s3.us-east-1.amazonaws.com/my-bucket/ai-backups
Environment=RESTIC_PASSWORD_FILE=%h/.config/restic/ai.pw
Environment=AWS_DEFAULT_REGION=us-east-1
# Expect AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be in an env file:
EnvironmentFile=%h/.config/restic/aws.env
ExecStart=%h/bin/ai-backup.sh

~/.config/systemd/user/ai-backup.timer

[Unit]
Description=Run AI backup daily at 02:00 with jitter

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

systemctl --user daemon-reload
systemctl --user enable --now ai-backup.timer
systemctl --user list-timers

Verification and alerts:

  • Parse restic output and send to your alerting (email/Slack). Example with journalctl:
journalctl --user -u ai-backup.service -n 200 --no-pager
  • Weekly deeper verification:
restic check --read-data-subset=10%

5) Practice restores (and document them)

A backup you’ve never restored is a hypothesis. Test often.

Restore the latest snapshot for a single project:

TARGET="$HOME/restore-test"
mkdir -p "$TARGET"
SNAP=$(restic snapshots --json | jq -r '.[-1].short_id')
restic restore "$SNAP" --target "$TARGET" --path "$HOME/projects/llm-finetune"

Sanity-check hashes against your manifest:

pushd "$TARGET/$HOME/projects/llm-finetune/.backup-meta" >/dev/null
sha256sum -c data.sha256 --ignore-missing || true
popd >/dev/null

Disaster drill: recover only what you need fast

  • Pull a specific checkpoint or tokenizer file
restic restore latest --target "$TARGET" --include '*/checkpoints/epoch-3/*'
restic restore latest --target "$TARGET" --include '*/tokenizer.json'

Security tip:

  • Keep RESTIC_PASSWORD_FILE off the same machine (or in a password manager/secret store). If you must keep a local copy, ensure strict perms: chmod 600.

Real-world examples

  • LLM fine-tuning project

    • Back up: train/val JSONL, tokenizer files, final and intermediate checkpoints, wandb run summaries, training_args.json, your fine-tune script, requirements.lock, and git-commit.txt.
    • Avoid: huge ephemeral batch logs and caches; capture a sample instead.
    • Bonus: store the model card and evaluation report alongside the artifact.

  • Vision model on-prem + NAS

    • Primary: restic to B2/S3 with object lock enabled.
    • Secondary: borg to office NAS over SSH.
    • Restore drill: monthly random class from dataset and last month’s best checkpoint.

  • Compliance-conscious team

    • Don’t back up raw PII; back up de-identification code + manifests of derived data.
    • Keep key material in KMS/Vault, never in backups. Rotate regularly.
    • Apply bucket policies/retention and log reads/writes for audits.

Common gotchas

  • Backing up .venv instead of environment lockfiles. Prefer requirements.lock, conda-env.yaml, and container image references.

  • Not excluding secrets. Use a vault; add safe excludes.

  • Skipping verification. Schedule restic check and a quarterly restore drill.

  • One copy only. Embrace 3-2-1 with at least one immutable/offsite copy.

Conclusion and next steps

Backups are boring—until they save weeks of compute and months of learning. Start now: 1) Copy the manifest and backup scripts into your repo. 2) Configure your restic repo and credentials; run a first full backup. 3) Enable the systemd timer and schedule a restore drill on your calendar.

Your models, data, and research deserve production-grade resilience. Ship your AI with a backup plan you trust.

Related posts