Posted on
• Artificial Intelligence

Artificial Intelligence Disaster Recovery Planning

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Disaster Recovery Planning on Linux: A Bash-First Playbook

When a model checkpoint corrupts at 3 a.m. or a cloud region goes dark, your AI SLA won’t accept “we’re retraining” as an answer. AI workloads are uniquely stateful and fast-changing; without a disaster recovery (DR) plan, you risk losing not just data—but the reproducibility and trust in your system. This article gives you a concrete, Bash-first approach to build, automate, and test AI disaster recovery on Linux.

What you’ll get:

  • Why AI needs DR that goes beyond generic IT backups

  • A practical toolkit (with apt, dnf, zypper install commands)

  • 5 actionable steps with copy-pasteable Bash

  • A simple way to test your recovery end-to-end

Why AI Disaster Recovery Is Different (and Urgent)

Modern AI stacks pack in unique failure modes:

  • State sprawl: models, checkpoints, vector indexes, datasets, feature stores, experiment metadata, container images, and secret configs—all change at different cadences.

  • High churn and size: multi-GB to TB of artifacts mean “just zip it” won’t cut it.

  • Consistency requirements: vector DBs, Postgres metadata, and training checkpoints must be captured atomically to be restorable.

  • Reproducibility: backups are useless if you can’t recover the exact environment that produced results.

  • Compliance: auditability and retention policies often require immutable, encrypted backups and recovery drills.

If you can’t answer “What’s our RPO/RTO for model artifacts versus vector indexes? When did we last test a full restore?”—you need this plan.

Tools We’ll Use (Install on your distro)

  • restic (encrypted, deduplicated backups)

  • borgbackup (popular alternative to restic)

  • rclone (move/sync data across clouds)

  • jq and curl (APIs and JSON parsing)

  • lvm2 (optional filesystem snapshots)

  • postgresql client tools (pg_dump/psql)

  • podman (OCI containers, rootless by default)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y restic borgbackup rclone jq curl lvm2 postgresql-client podman

Fedora/RHEL/CentOS Stream (dnf):

# On RHEL/CentOS, you may need: sudo dnf install -y epel-release
sudo dnf install -y restic borgbackup rclone jq curl lvm2 postgresql podman

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y restic borgbackup rclone jq curl lvm2 postgresql podman

Tip: For long-lived credentials, prefer environment files with 600 permissions instead of exporting in shell history.

The 5-Step AI DR Plan (with Bash)

1) Inventory and Classify AI State (Set RPO/RTO)

Start by listing and hashing the artifacts you must recover. Classify each with target RPO/RTO. Typical classes:

  • Hot: inference models, vector indexes, feature store snapshots, secrets/config (RPO: minutes–hours; RTO: minutes)

  • Warm: training checkpoints and datasets deltas (RPO: hours; RTO: hours)

  • Cold: raw datasets and archived experiments (RPO: days; RTO: days)

Create a manifest with checksums so you can verify integrity later:

AI_ROOT=/srv/ai
mkdir -p "$AI_ROOT/manifests"

# Hash common AI artifacts (extend patterns to your stack)
find "$AI_ROOT" -type f -regextype posix-extended \
  -regex '.*\.(pt|safetensors|onnx|ckpt|bin|gguf|json|yaml|toml|parquet|feather|csv|pkl)$' \
  -print0 | xargs -0 sha256sum > "$AI_ROOT/manifests/artifacts-$(date +%F).sha256"

# Summarize biggest items (helps spot surprise growth)
du -ah "$AI_ROOT" | sort -h | tail -n 50 > "$AI_ROOT/manifests/top50-$(date +%F).txt"

Outcome: You know what matters, how fresh it must be (RPO), and how fast you must restore it (RTO).

2) Versioned, Encrypted, Offsite Backups (restic or borg)

Restic is simple, fast, encrypted by default, and supports S3/B2/MinIO/etc.

Environment file for credentials (recommended):

sudo install -m 600 /dev/stdin /etc/restic.env <<'EOF'
export RESTIC_REPOSITORY="s3:https://s3.your-endpoint.example.com/ai-backups"
export RESTIC_PASSWORD="change_me_strong_passphrase"
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
EOF

Initialize and run a backup:

source /etc/restic.env
restic init

restic backup /srv/ai/models /srv/ai/checkpoints /srv/ai/datasets /srv/ai/configs \
  --tag daily --exclude-if-present .nobackup

# Retention policy (tune to your compliance)
restic forget --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

Automate with systemd (rootless or system):

sudo tee /etc/systemd/system/restic-backup.service >/dev/null <<'UNIT'
[Unit]
Description=Restic backup of AI data

[Service]
Type=oneshot
EnvironmentFile=/etc/restic.env
ExecStart=/usr/bin/restic backup /srv/ai/models /srv/ai/checkpoints /srv/ai/datasets /srv/ai/configs --tag scheduled
ExecStartPost=/usr/bin/restic forget --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
UNIT

sudo tee /etc/systemd/system/restic-backup.timer >/dev/null <<'UNIT'
[Unit]
Description=Run restic backup hourly

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer

Optional: Borg alternative

borg init --encryption=repokey-blake2 /mnt/backup/borg-repo
borg create --stats /mnt/backup/borg-repo::ai-$(date +%F-%H%M) /srv/ai
borg prune -v --list /mnt/backup/borg-repo --keep-daily=7 --keep-weekly=4 --keep-monthly=6

Optional: Replicate snapshots cross-cloud with rclone

rclone copy --immutable --transfers=8 s3:ai-backups s3:ai-backups-dr

3) Consistent Snapshots for Vector DBs and Metadata

Application-aware snapshots beat “copy while running.”

  • Postgres/Timescale (experiment tracking, metadata):
# Requires postgresql/postgresql-client installed
PGHOST=127.0.0.1 PGUSER=aiuser PGDATABASE=aimeta \
pg_dump -Fc > /srv/ai/db_backups/aimeta-$(date +%F-%H%M).dump
  • Qdrant (vector DB) snapshots via API:
# Create collection snapshot
curl -s -X POST http://localhost:6333/collections/my_collection/snapshots | jq

# Or whole storage snapshot
curl -s -X POST http://localhost:6333/snapshots | jq

Copy resulting snapshot files to your backup set.

  • Milvus/Local index directories: If DB-level snapshots aren’t available, coordinate a short “freeze → copy → unfreeze” window or use LVM:
# LVM snapshot of a logical volume that holds your vector DB data
sudo lvcreate --size 20G --snapshot --name ai_data_snap /dev/vg0/ai_data
sudo mkdir -p /mnt/ai_data_snap
sudo mount /dev/vg0/ai_data_snap /mnt/ai_data_snap

# Now back up snapshot contents (consistent point-in-time)
restic backup /mnt/ai_data_snap --tag vecdb-snapshot

sudo umount /mnt/ai_data_snap
sudo lvremove -y /dev/vg0/ai_data_snap

Note: For containerized stacks, prefer app-native snapshot APIs or stop-the-world minutes off-peak rather than risking torn copies.

4) Make Artifacts Immutable and Environments Reproducible

Backups restore data; reproducibility restores trust.

  • Capture exact Python deps and configs:
# Inside your training/inference venv
pip freeze > /srv/ai/configs/requirements-$(date +%F).txt

# Lock runtime configs
cp -a /etc/yourapp/*.yaml /srv/ai/configs/
  • Containerize and version with Podman:
# Build immutable image
podman build -t registry.example.com/ai/inference:$(date +%Y%m%d) -f Containerfile .

# Push to your registry (ensure credentials configured)
podman push registry.example.com/ai/inference:$(date +%Y%m%d)

# Save critical images into backups too
podman save -o /srv/ai/images/inference-$(date +%F).tar registry.example.com/ai/inference:$(date +%Y%m%d)
  • Capture infra and pipelines in Git: Keep IaC (Terraform/Ansible), CI/CD workflows, and data processing DAGs versioned. Tag releases that correspond to production deployments; reference tags in recovery docs.

5) Run DR Drills with Automated Validation

Recovery you haven’t tested is recovery you don’t have. Make a one-button drill that restores to a clean path and runs smoke tests.

Example smoke-restore script:

#!/usr/bin/env bash
set -euo pipefail

RESTORE_DIR=/tmp/ai-restore-$(date +%s)
mkdir -p "$RESTORE_DIR"

# 1) Restore from restic
source /etc/restic.env
restic restore latest --target "$RESTORE_DIR"

# 2) Restore Postgres metadata (example assumes local dev DB)
# Adjust host and credentials for your test target
createdb -h 127.0.0.1 -U aiuser aimeta_restore || true
pg_restore -h 127.0.0.1 -U aiuser -d aimeta_restore \
  "$RESTORE_DIR/srv/ai/db_backups/aimeta-"*.dump

# 3) Quick artifact integrity check
sha256sum -c "$RESTORE_DIR/srv/ai/manifests/"artifacts-*.sha256

# 4) Optional: spin up containerized inference and test
pushd "$RESTORE_DIR/srv/ai"
podman run --rm -d --name ai_infer -p 8080:8080 \
  -v "$PWD/models:/app/models:ro" \
  registry.example.com/ai/inference:latest

# Give it a moment to start, then hit a health endpoint
sleep 5
curl -fsS http://127.0.0.1:8080/health

podman rm -f ai_infer || true
popd

echo "DR drill SUCCESS in $(basename "$RESTORE_DIR")"

Time your runs to measure RTO:

time ./dr-smoke-restore.sh

Automate this weekly and alert on failures. Keep a runbook with screenshots or logs of successful drills for audits.

Real-World Example (Why This Works)

An applied-research lab lost a primary NVMe drive mid-epoch. Their plan:

  • Hourly restic backups of checkpoints with 7-day retention

  • Daily Postgres dumps of experiment metadata

  • Weekly DR drill to a staging node with a hash check + inference smoke test

Outcome: They resumed training from an hourly checkpoint, recovered experiment lineage, and restored inference in under 30 minutes. No data loss beyond the last hour (RPO met), and SLA impact was minimal (RTO met).

Security and Hygiene Tips

  • Never hardcode secrets in scripts; store them in root-readable env files or a secrets manager.

  • Encrypt everything at rest and in transit (restic does this by default).

  • Use immutable retention or object-lock where compliance requires.

  • Monitor backup sizes and prune policies—surprise growth can silently break RPO.

  • Document the DR contact tree and access paths; limit who can delete backups.

Conclusion and Next Steps (CTA)

AI DR isn’t just “having backups.” It’s knowing exactly what state matters, capturing it consistently, storing it securely offsite, and proving you can restore it—fast.

Your move: 1) Run the inventory scripts and define RPO/RTO per data class. 2) Stand up restic with a daily/hourly policy and test one full restore this week. 3) Add application-level snapshots for your vector DB and metadata store. 4) Version your environments and images; include them in backups. 5) Automate a weekly DR drill with a health check.

If you want a ready-to-run “AI DR Starter Kit” (systemd units, scripts, and example playbooks), tell me your stack (vector DB, metadata store, model formats), and I’ll tailor a minimal repo you can drop in today.

Related posts