Posted on
• Artificial Intelligence

Artificial Intelligence Snapshot Management

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Snapshot Management with Bash: Reproducible Experiments at Your Fingertips

If you’ve ever trained an AI model that performed brilliantly—only to find you can’t reproduce it later—you know the pain. Was it the dataset? The random seed? A minor library update? Without snapshots, you’re guessing. With snapshots, you’re in control: you can roll back, audit, and reproduce exactly what happened.

This article shows how to build a practical, Bash-first snapshot management workflow for AI projects. You’ll get reproducible experiments, fast rollbacks, and offsite backups—using tools available in most Linux distros. We’ll cover code/data snapshots, filesystem snapshots, and automation.

Why snapshot management matters for AI

  • Reproducibility: Debug anomalies and publish verifiable results.

  • Compliance and audit: Track exactly which data, code, and environment produced a model.

  • Disaster recovery: Roll back after a bad update or a corrupted training run.

  • Cost and time savings: Avoid retraining when you can restore artifacts and states.

What “snapshot” means here

  • Code/config snapshot: Git commit + tracked configs.

  • Data/model artifacts snapshot: Large binary artifacts versioned without bloating Git history.

  • System/environment snapshot: Instant filesystem-level rollback for your project directory or data volume.

  • Run-level metadata: Automatic logging of seeds, hardware, commit, parameters, and outputs.

Prerequisites: Install the essentials

We’ll use Git, Git LFS or Git Annex for large files, Btrfs or LVM for filesystem snapshots, and Restic for offsite backups. Install with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git git-lfs git-annex btrfs-progs lvm2 restic
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y git git-lfs git-annex btrfs-progs lvm2 restic
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y git git-lfs git-annex btrfsprogs lvm2 restic

Notes:

  • You only need Btrfs OR LVM for snapshots, not both. Choose what your system uses.

  • Git LFS and Git Annex both handle large files—pick one for your project (Annex is more flexible for external storage; LFS is simpler if your Git host supports it).

Step 1: Prepare a reproducible project skeleton and metadata logging

Initialize Git and create a consistent structure for runs and artifacts.

mkdir -p ~/ai-project/{data,models,runs,scripts}
cd ~/ai-project
git init

Option A — Git LFS (simple, requires LFS-supporting remote like GitHub):

git lfs install
git lfs track "*.pt" "*.pth" "*.onnx" "*.ckpt" "*.h5" "data/*" "models/*"
echo ".gitattributes" >> .gitignore  # only if you keep it ignored elsewhere
git add .gitattributes
git commit -m "Configure Git LFS for large artifacts"

Option B — Git Annex (flexible external remotes, doesn’t bloat repo):

git annex init "ai-project"
# Example: use a local directory or SSH/S3 as a remote later:
# git remote add storage ssh://user@host:/srv/annex
# git annex enableremote storage type=git

Create a lightweight metadata script to capture run context:

cat > scripts/capture_metadata.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

RUN_DIR="$1"
SEED="${SEED:-1337}"

{
  echo "timestamp=$(date -Is)"
  echo "hostname=$(hostname)"
  echo "kernel=$(uname -srmo)"
  echo "user=$USER"
  echo "pwd=$PWD"
  echo "git_commit=$(git rev-parse --verify HEAD 2>/dev/null || echo 'no-git')"
  echo "git_status=$(git status --porcelain | wc -l) changes"
  command -v python3 >/dev/null && echo "python=$(python3 -V 2>&1)" || true
  command -v nvidia-smi >/dev/null && echo "gpu=$(nvidia-smi --query-gpu=name,driver_version --format=csv,noheader 2>/dev/null | tr '\n' '; ')" || true
  echo "seed=$SEED"
} > "$RUN_DIR/metadata.txt"
EOF
chmod +x scripts/capture_metadata.sh

Step 2: Snapshot code and artifacts with Git + LFS or Git Annex

During/after each run, capture the code state and store outputs deterministically.

Basic workflow with Git LFS:

# Before run
git add -A
git commit -m "Pre-run: configs and code"

# After run, stage outputs tracked by LFS:
git add runs/ models/
git commit -m "Run outputs"
git push
git lfs push origin --all

Basic workflow with Git Annex:

# Track large files without bloating Git
git annex add runs/ models/ data/
git commit -m "Annex: add artifacts"
# Sync to configured remote(s)
git annex sync --content

Tip:

  • Keep raw datasets and large models in Annex or LFS.

  • Keep code, configs, and small text logs in normal Git history.

Step 3: Fast filesystem snapshots (Btrfs or LVM)

For instant project-wide rollback (e.g., before a risky upgrade or big training run), use filesystem snapshots. Pick one method that matches your setup.

A) Btrfs subvolume snapshots (project lives on a Btrfs filesystem)

  • Ensure your project is a subvolume (recommended):
# If ~/ai-project is not yet a subvolume, create and move into it (one-time)
# CAUTION: Adjust paths carefully and back up first.

cd ~
mv ai-project ai-project.bak
sudo btrfs subvolume create ai-project
rsync -aHAX --info=progress2 ai-project.bak/ ai-project/
rm -rf ai-project.bak
  • Take a read-only snapshot before a run:
SNAP_NAME="pre-run-$(date +%Y%m%d-%H%M%S)"
sudo mkdir -p ~/ai-project/.snapshots
sudo btrfs subvolume snapshot -r ~/ai-project ~/ai-project/.snapshots/$SNAP_NAME
  • List and delete snapshots:
sudo btrfs subvolume list -o ~/ai-project
sudo btrfs subvolume delete ~/ai-project/.snapshots/$SNAP_NAME

B) LVM snapshots (project directory lives on an LVM logical volume)

  • Create a snapshot LV (allocate enough space for changes; 20G example):
# Example names: VG=vg0, LV=lv_data; mountpoint contains your project
sudo lvcreate -s -n lv_data_pre_run -L 20G /dev/vg0/lv_data
  • Optionally mount snapshot read-only to verify:
sudo mkdir -p /mnt/data-snap
sudo mount -o ro /dev/vg0/lv_data_pre_run /mnt/data-snap
# ...inspect...
sudo umount /mnt/data-snap
  • Remove snapshot when done:
sudo lvremove -y /dev/vg0/lv_data_pre_run

Notes:

  • Snapshots are cheap initially but consume space as data changes.

  • Prefer Btrfs snapshots for directory-level agility; use LVM for block-level rollback.

Step 4: Offsite archives with Restic (S3/SSH/local disk)

Restic gives encrypted, deduplicated backups of your runs and metadata. Initialize a repo and back up your runs directory.

Set environment variables (example: S3-compatible storage):

export RESTIC_REPOSITORY="s3:https://s3.example.com/my-bucket/ai-project"
export AWS_ACCESS_KEY_ID="YOURKEY"
export AWS_SECRET_ACCESS_KEY="YOURSECRET"
export RESTIC_PASSWORD="choose-a-strong-password"

Initialize and backup:

restic init
restic backup ~/ai-project/runs

List, forget, and prune:

restic snapshots
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

Other targets:

  • Local disk: export RESTIC_REPOSITORY=/mnt/backup/restic-ai

  • Over SSH: export RESTIC_REPOSITORY=sftp:user@host:/backups/restic-ai

Store credentials securely (e.g., in a password file with 600 permissions) instead of environment variables in production.

Step 5: Automate the workflow with a single Bash entrypoint

Wrap your training command with pre/post snapshot steps, metadata capture, and backups. Adapt paths and training command to your project.

cat > run.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

PROJECT_DIR="$HOME/ai-project"
RUNS_DIR="$PROJECT_DIR/runs"
SEED="${SEED:-1337}"
RUN_ID="$(date +%Y%m%d-%H%M%S)-$RANDOM"
RUN_DIR="$RUNS_DIR/$RUN_ID"

mkdir -p "$RUN_DIR"

cd "$PROJECT_DIR"

# 1) Pre-run: commit code/config state
if [ -d .git ]; then
  git add -A
  git commit -m "Pre-run $RUN_ID: snapshot code/config" || true
fi

# 2) Pre-run: fast filesystem snapshot (Btrfs if available)
if findmnt -n -o FSTYPE --target "$PROJECT_DIR" | grep -qi btrfs; then
  SNAP_NAME="pre-$RUN_ID"
  sudo mkdir -p "$PROJECT_DIR/.snapshots"
  sudo btrfs subvolume snapshot -r "$PROJECT_DIR" "$PROJECT_DIR/.snapshots/$SNAP_NAME"
  echo "Btrfs read-only snapshot created: $PROJECT_DIR/.snapshots/$SNAP_NAME"
fi

# 3) Capture metadata
scripts/capture_metadata.sh "$RUN_DIR"

# 4) Launch training (replace with your command)
# Example:
#   python3 train.py --data data/imagenet --epochs 90 --seed "$SEED" --out "$RUN_DIR"
# Below is a placeholder that simulates output:
echo "Simulating training with SEED=$SEED" | tee "$RUN_DIR/train.log"
sleep 2
echo "model_dummy.bin" > "$RUN_DIR/model_dummy.bin"

# 5) Post-run: record artifacts with Git LFS or Git Annex
if command -v git-annex >/dev/null; then
  git annex add "$RUN_DIR"
  git commit -m "Run $RUN_ID artifacts (annex)"
  git annex sync --content || true
elif command -v git >/dev/null; then
  # If using Git LFS, ensure patterns were tracked earlier
  git add "$RUN_DIR"
  git commit -m "Run $RUN_ID artifacts"
  git push || true
  git lfs push origin --all || true
fi

# 6) Offsite backup with Restic (optional, if configured)
if [ -n "${RESTIC_REPOSITORY:-}" ] && [ -n "${RESTIC_PASSWORD:-}" ]; then
  restic backup "$RUN_DIR" || true
fi

echo "Run complete: $RUN_DIR"
EOF
chmod +x run.sh

Run it:

SEED=2025 ./run.sh

Optional scheduling with cron (daily 02:00 backup of runs directory with Restic):

crontab -e
# Add:
0 2 * * * RESTIC_REPOSITORY=/mnt/backup/restic-ai RESTIC_PASSWORD=... /usr/bin/restic backup $HOME/ai-project/runs >> $HOME/ai-project/restic.log 2>&1

Real-world example: What you’ll see after a run

ai-project/
├─ data/
├─ models/
├─ runs/
│  └─ 20260711-142233-12345/
│     ├─ metadata.txt
│     ├─ train.log
│     └─ model_dummy.bin
├─ scripts/
│  └─ capture_metadata.sh
├─ .gitattributes   # if using Git LFS
└─ run.sh

  • metadata.txt captures commit, kernel, Python version, GPU info, and seed.

  • Artifacts live under a unique run ID directory, tracked by LFS or Annex.

  • A Btrfs/LVM snapshot lets you roll back the entire project if needed.

  • Restic archives your runs offsite for disaster recovery.

Conclusion and next steps

A little Bash glue goes a long way. By combining:

  • Git (+ LFS or Annex) for code and artifacts,

  • Btrfs or LVM for instant, space-efficient snapshots,

  • Restic for encrypted offsite backups,

you get a reliable, auditable, and reproducible AI workflow.

Your next step:

  • Implement Step 5’s run.sh in your project and run one experiment end to end.

  • Then add policies: retention (Restic forget/prune), snapshot naming conventions, and a small README documenting how to restore.

Want to go further? Add dataset checksums, seed normalization across libraries, containerized runtimes, or CI that automatically validates a snapshot can reproduce a baseline metric. Snapshots aren’t just safety nets—they’re the backbone of professional AI operations.

Related posts