Posted on
• Artificial Intelligence

Artificial Intelligence VM Backup Strategies

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence VM Backup Strategies: Keep Your GPU Hours Safe

Ever watched a multi-day training run crash and take your VM with it? That’s not just frustrating—it’s expensive. AI VMs often hold massive datasets, model checkpoints, and specialized OS/GPU stacks that aren’t trivial to rebuild. A solid backup strategy means your experiments survive hardware failures, operator errors, and cloud hiccups.

This post lays out practical, bash-friendly strategies to back up AI virtual machines (KVM/libvirt focus), get consistent snapshots while models are training, deduplicate big datasets, and automate offsite retention. By the end, you’ll have concrete commands and scripts to protect your GPU time and your results.

Why AI VM backups are different

  • High write rates and big files: Datasets and checkpoints can be 100s of GB to TB, continuously updated.

  • Application consistency: Filesystem-consistent backups may still yield corrupted checkpoints. Quiescing training before snapshots helps.

  • Cost of downtime: Losing a tuned CUDA stack, drivers, and environment can cost days to recreate.

  • Storage economics: Deduplication and compression matter when your data explodes.

What we’ll use

  • libvirt/virsh and QEMU for hot, application-consistent snapshots

  • qemu-guest-agent inside the guest for quiescing filesystems

  • Borg or Restic for deduplicated, encrypted backups

  • rsync for frequent lightweight checkpoint syncs

  • Optional: LVM/ZFS snapshots if your storage supports them

Install on the host

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y libvirt-clients libvirt-daemon-system qemu-utils borgbackup restic rsync pigz zstd
sudo systemctl enable --now libvirtd

Fedora/RHEL (dnf):

sudo dnf install -y libvirt-client libvirt-daemon qemu-img borgbackup restic rsync pigz zstd
sudo systemctl enable --now libvirtd

openSUSE (zypper):

sudo zypper install -y libvirt libvirt-client qemu-tools borgbackup restic rsync pigz zstd
sudo systemctl enable --now libvirtd

Install inside each VM (guest)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

Fedora/RHEL (dnf):

sudo dnf install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

openSUSE (zypper):

sudo zypper install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

Strategy 1: Hot, consistent snapshots with libvirt + qemu-guest-agent

Goal: Back up a running VM without corruption—even while training—by quiescing the filesystem and using an external snapshot. Then back up the stable base image while the VM writes to a temporary overlay. Finally, merge and clean up.

Important:

  • Works best with file-based disks (qcow2/raw). For LVM/ZFS-backed disks, see Strategy 4 for native snapshots.

  • You can also pause training briefly before snapshot (see Strategy 3).

Example backup script (host):

#!/usr/bin/env bash
# backup-ai-vm.sh: create external snapshot, back up base image, then blockcommit
# Requires: libvirt/virsh, qemu-utils, borgbackup or restic (choose one)

set -euo pipefail

VM="${1:-ai-vm}"          # libvirt domain name
DISK_TARGET="${2:-vda}"   # disk target in libvirt (e.g., vda, vdb)
BACKUP_METHOD="${3:-borg}" # "borg" or "restic"

timestamp() { date +%F-%H%M%S; }

# Optional: Ask the guest to checkpoint training (if you have a handler)
# ssh aiuser@"$VM" "pkill -USR1 -f train.py || true; sleep 10" || true

echo "[*] Creating external snapshot for $VM ..."
virsh snapshot-create-as --domain "$VM" \
  --name "bk-$(timestamp)" \
  --description "AI VM backup snapshot" \
  --disk-only --atomic --quiesce

# Find overlay and base
OVERLAY=$(virsh domblklist "$VM" --details | awk -v t="$DISK_TARGET" '$3=="disk" && $4==t {print $5}')
if [[ -z "$OVERLAY" ]]; then
  echo "[-] Could not determine overlay path for $DISK_TARGET" >&2
  exit 1
fi
# Extract backing file path robustly
BASE=$(qemu-img info "$OVERLAY" | sed -n 's/^backing file: \(.*\) (.*/\1/p')
if [[ -z "$BASE" ]]; then
  BASE=$(qemu-img info "$OVERLAY" | awk -F': ' '/backing file:/ {print $2}')
fi
if [[ -z "$BASE" ]]; then
  echo "[-] Could not determine base image backing file from $OVERLAY" >&2
  exit 1
fi

echo "[*] Overlay is $OVERLAY"
echo "[*] Base image is $BASE (stable for backup)"

# Choose a backup root path for file-based backups
SRC_DIR=$(dirname "$BASE")

# Option A: Borg (local or mounted repo)
if [[ "$BACKUP_METHOD" == "borg" ]]; then
  : "${BORG_REPO:?Set BORG_REPO}"
  : "${BORG_PASSPHRASE:?Set BORG_PASSPHRASE}"
  echo "[*] Running borg create ..."
  borg create --stats --compression zstd,10 \
    "$BORG_REPO"::"${VM}-$(timestamp)" \
    "$BASE"  # You can include other disks/paths here
  borg prune -v --list --keep-daily=7 --keep-weekly=4 --keep-monthly=6 "$BORG_REPO"
  borg check "$BORG_REPO"
fi

# Option B: Restic (works great to S3-compatible storage)
if [[ "$BACKUP_METHOD" == "restic" ]]; then
  : "${RESTIC_REPOSITORY:?Set RESTIC_REPOSITORY (e.g., s3:https://...)}"
  : "${RESTIC_PASSWORD:?Set RESTIC_PASSWORD}"
  echo "[*] Running restic backup ..."
  restic backup --one-file-system "$BASE"
  restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
  restic check
fi

echo "[*] Committing overlay back into base (pivot) ..."
virsh blockcommit "$VM" "$DISK_TARGET" --active --pivot --verbose --bandwidth 0

# Clean up stray overlay if present
if [[ -f "$OVERLAY" ]]; then
  rm -f -- "$OVERLAY"
fi

echo "[+] Backup complete for $VM"

Usage:

# With Borg
export BORG_REPO=/backups/borg/ai-vm
export BORG_PASSPHRASE='change-me'
borg init --encryption repokey-blake2 "$BORG_REPO"
./backup-ai-vm.sh ai-vm vda borg

# With Restic (to S3/MinIO)
export RESTIC_REPOSITORY="s3:https://s3.example.com/backups/ai-vm"
export RESTIC_PASSWORD='change-me'
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
restic init
./backup-ai-vm.sh ai-vm vda restic

Notes:

  • Add more disks by repeating the sequence or adjusting the script.

  • If quiesce fails, ensure qemu-guest-agent runs in the guest and that filesystems support freezing.

  • For very busy disks, consider throttling or snapshotting during a lull.

Strategy 2: Deduplicated, encrypted repositories with Borg or Restic

Big models and datasets benefit from block-level deduplication and compression:

Borg (great local/NFS/SSH repos):

# Install (see package manager sections above), then:
export BORG_REPO=/srv/backups/borg/ai-vm
export BORG_PASSPHRASE='change-me'
borg init --encryption repokey-blake2 "$BORG_REPO"

# Back up VM images, configs, and checkpoints
borg create --stats --compression zstd,19 \
  "$BORG_REPO"::ai-vm-{now:%Y-%m-%d_%H-%M} \
  /var/lib/libvirt/images/ai-vm/ /etc/libvirt/qemu/ai-vm.xml /data/checkpoints/

# Retention and consistency
borg prune -v --list --keep-daily=7 --keep-weekly=4 --keep-monthly=6 "$BORG_REPO"
borg check "$BORG_REPO"

Restic (shines with S3-compatible object storage and offsite):

export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/my-bucket/ai-vm"
export RESTIC_PASSWORD='change-me'
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

restic init
restic backup /var/lib/libvirt/images/ai-vm/ /etc/libvirt/qemu/ai-vm.xml /data/checkpoints/
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check

Tip:

  • Store OS disk, data disk, and checkpoints in the same repo for better deduplication.

  • Use zstd compression for speed/ratio sweet spot.

Strategy 3: Split data domains and checkpoint smartly

Don’t treat everything the same. Assign policies by data type:

  • OS/driver stack: Back up nightly (smallish, but painful to rebuild).

  • Datasets: Immutable or slowly changing—snapshot monthly plus incremental sends.

  • Checkpoints: Frequent and critical—sync every N minutes.

Example: frequent checkpoint sync from inside the VM:

# Run within the VM; sync every 10 minutes via cron/systemd timer
rsync -avh --partial --inplace --info=progress2 \
  /mnt/checkpoints/ backup@nas:/backups/ai-vm/checkpoints/

Make training backup-aware. For example, catch SIGUSR1 to save a clean checkpoint before a host snapshot:

# Host before snapshot (optional but recommended)
ssh aiuser@ai-vm "pkill -USR1 -f train.py; sleep 10" || true

Strategy 4: Use your storage’s native snapshots (LVM/ZFS) when available

If your VM disks live on LVM or ZFS, native snapshots are fast and efficient.

LVM example (host):

# Snapshot a logical volume, mount read-only, back it up, then remove
sudo lvcreate -s -n ai-vm-root-snap -L 20G /dev/vg/ai-vm-root
sudo mkdir -p /mnt/ai-vm-root-snap
sudo mount -o ro /dev/vg/ai-vm-root-snap /mnt/ai-vm-root-snap
restic backup /mnt/ai-vm-root-snap
sudo umount /mnt/ai-vm-root-snap
sudo lvremove -y /dev/vg/ai-vm-root-snap

ZFS example (dataset for VM images or training data):

# Create and send an incremental snapshot offsite
SNAP="ai-vm@$(date +%F-%H%M)"
zfs snapshot pool/ai-vm-images@$SNAP
zfs send -Lce pool/ai-vm-images@$SNAP | zstd -T0 -19 | ssh backup@remote "zstd -d | zfs recv -F pool/ai-vm-images"

Strategy 5: Automate and test restores

Backups you can’t restore aren’t backups.

Create a systemd timer to run the snapshot+backup script:

backup-ai-vm.service:

[Unit]
Description=AI VM backup

[Service]
Type=oneshot
Environment="BORG_REPO=/backups/borg/ai-vm"
Environment="BORG_PASSPHRASE=change-me"
ExecStart=/usr/local/sbin/backup-ai-vm.sh ai-vm vda borg

backup-ai-vm.timer:

[Unit]
Description=Run AI VM backup nightly

[Timer]
OnCalendar=03:00
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo cp backup-ai-vm.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now backup-ai-vm.timer

Quick restore test (Restic example):

# Restore into a scratch directory and verify key files exist
export RESTIC_REPOSITORY="s3:https://s3.example.com/backups/ai-vm"
export RESTIC_PASSWORD='change-me'
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

mkdir -p /restore/ai-vm-test
restic restore latest --target /restore/ai-vm-test
ls -lh /restore/ai-vm-test/var/lib/libvirt/images/

Optionally boot a throwaway VM from the restored disk to validate end-to-end:

virt-install --name ai-vm-restore --memory 16384 --vcpus 8 \
  --import --disk path=/restore/ai-vm-test/var/lib/libvirt/images/ai-vm.qcow2,bus=virtio \
  --network network=default --noautoconsole

Real-world pattern that works

  • Put OS and tooling on one disk, datasets on another, and checkpoints on a fast, separate volume.

  • Hourly: rsync checkpoints off-VM.

  • Nightly: libvirt external snapshot → backup base image with Borg/Restic → blockcommit.

  • Weekly: offsite Restic to S3 with retention; enable object lock/immutability at the bucket level if available.

  • Monthly: ZFS/LVM snapshot of datasets + incremental sends.

Conclusion and next steps

GPU hours are precious. A few shell commands can be the difference between a near-miss and a week lost. Start by:

1) Installing qemu-guest-agent in your VMs and setting up the host packages.
2) Dropping the backup-ai-vm.sh script on your host and running a manual test.
3) Scheduling nightly backups and weekly offsite syncs.
4) Proving you can restore—at least once a month.

Call to action: Block out one hour this week to implement Strategy 1 + 2. Run a test restore. Then iterate with checkpoint syncs and offsite retention. Your future self (and your training timelines) will thank you.

Related posts