Posted on
• Artificial Intelligence

Artificial Intelligence Restore Testing

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Restore Testing with Bash: Turn “We Have Backups” into “We Can Recover”

Ever said “we have backups” and still sweated through a 3 a.m. outage? The difference between a backup and a recovery is a tested restore. The problem: manual restore tests are slow, brittle, and rarely done at scale. The value: automate them, collect signals, and add a small dose of AI to detect hidden problems and drift before production depends on it.

In this post, you’ll build a repeatable, Bash-first restore testing pipeline that:

  • Spins up a disposable restore environment

  • Restores from your backup tool (restic or borg)

  • Runs health checks and collects metrics

  • Uses a lightweight ML anomaly detector to flag subtle issues

  • Automates the whole thing on a timer

No vendor lock-in, no black boxes—just Linux tools and a tiny Python model.

Why AI-assisted restore testing is worth it

  • Config drift is real: a restore might “appear fine” (files exist, services return 200) while response times double or a small set of files mismatch. AI anomaly detection compares today’s restore against your own historical baseline to catch subtle deviations.

  • Scale and repetition: testing weekly or nightly creates trends that humans won’t manually analyze. Let the model highlight outliers automatically.

  • Defense in depth: checksums and exit codes are necessary but not sufficient. Combining signals (file counts, sizes, checksums, service latencies) yields stronger confidence.

What we’ll build (overview)

1) Disposable target: a clean environment for every restore (container, chroot, or VM). 2) Restore run: pull from restic or borg. 3) Post-restore checks: create a manifest, compare checksums, and probe app health. 4) AI anomaly detection: IsolationForest flags deviations from normal. 5) Automation: systemd timer schedules tests, logs results, and can alert on anomalies.

Install prerequisites

We’ll use:

  • Backup tools: restic and/or borg

  • Shell utilities: curl, jq

  • Python for anomaly detection: pandas, scikit-learn, numpy

On Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y restic borgbackup python3 python3-venv python3-pip curl jq

On Fedora/RHEL/CentOS (dnf):

sudo dnf install -y restic borgbackup python3 python3-pip curl jq
# Note: on some RHEL-like systems you may need EPEL for borg:
# sudo dnf install -y epel-release
# sudo dnf install -y borgbackup

On openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y restic borgbackup python3 python3-pip curl jq

Set up a virtual environment for the AI step:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install scikit-learn pandas numpy

Optional (containerized test target with Podman):

  • apt:

    sudo apt install -y podman

  • dnf:

    sudo dnf install -y podman

  • zypper:

    sudo zypper install -y podman

Step 1: Create a disposable restore target

You can use a temp directory, a chroot, or a container. Here’s a minimal container example using Podman so the host stays clean:

# Launch a Fedora container with your backups and scripts mounted in
sudo podman run --rm -it \
  -v /backups:/backups:ro \
  -v "$PWD":/work \
  -w /work \
  quay.io/fedora/fedora:latest bash

Inside the container, install tools with dnf (already shown above), then proceed with the next steps. If you prefer bare metal for speed, just use a per-run temp directory like /tmp/restore-12345.

Step 2: Perform the restore (restic or borg)

Pick the tool you actually use in production. Below are skeletons you can adapt.

Example: restic restore into a clean target:

# Environment variables you likely already have configured:
# export RESTIC_REPOSITORY="s3:s3.amazonaws.com/your-bucket"
# export RESTIC_PASSWORD="..."
# export AWS_ACCESS_KEY_ID=...
# export AWS_SECRET_ACCESS_KEY=...

TARGET="/tmp/restore-app-$$"
mkdir -p "$TARGET"
restic -r "$RESTIC_REPOSITORY" restore latest --target "$TARGET"

Example: borg restore:

# export BORG_REPO="ssh://user@backup-host:22/var/backup/borg-repo"
TARGET="/tmp/restore-app-$$"
mkdir -p "$TARGET"
# Choose an archive; here we pick the newest
ARCHIVE=$(borg list "$BORG_REPO" --last 1 --format="{archive}{NL}")
borg extract "$BORG_REPO"::"$ARCHIVE" --destination "$TARGET"

Tip: keep your restore command configurable with an environment variable so your script doesn’t care which tool you use.

Step 3: Post-restore checks and metrics collection

We’ll generate checksums for the restored files, compare against a known-good manifest (if you have one), and probe application health endpoints to gather metrics. These metrics are fed into the AI step.

Create a known-good manifest once (optional but powerful):

# On a known-good production-like checkout (e.g., /srv/app)
( cd /srv/app && find . -type f -printf "%P\0" | sort -z \
  | xargs -0 -n1 -I{} sha256sum "{}" ) > /manifests/app-prod.sha256

Now the restore/metrics script (save as restore_test.sh):

#!/usr/bin/env bash
set -euo pipefail

DATASET=${DATASET:-"app-prod"}
TARGET=${TARGET:-"/tmp/restore-${DATASET}-$$"}

# Example: restic restore command; override via RESTORE_CMD if you use borg
: "${REPO:=/backups/restic}"
RESTORE_CMD=${RESTORE_CMD:-"restic -r ${REPO} restore latest --target ${TARGET}"}

MANIFEST_EXPECTED=${MANIFEST_EXPECTED:-"/manifests/${DATASET}.sha256"}

mkdir -p "$TARGET"

echo "[*] Restoring into $TARGET ..."
eval "$RESTORE_CMD"

echo "[*] Building post-restore manifest ..."
MANIFEST_ACTUAL="$TARGET/.postrestore.sha256"
( cd "$TARGET" && find . -type f -printf "%P\0" | sort -z \
  | xargs -0 -n1 -I{} sha256sum "{}" ) > "$MANIFEST_ACTUAL"

files_restored=$(wc -l < "$MANIFEST_ACTUAL")
bytes_restored=$(du -sb "$TARGET" | awk '{print $1}')

sha_mismatch=0
if [[ -f "$MANIFEST_EXPECTED" ]]; then
  sha_mismatch=$(comm -3 <(cut -d' ' -f1 "$MANIFEST_EXPECTED" | sort) \
                      <(cut -d' ' -f1 "$MANIFEST_ACTUAL" | sort) | wc -l)
fi

# Example health probe (replace with your service)
http_code_latency=$(curl -s -o /dev/null -w "%{http_code} %{time_total}\n" http://127.0.0.1:8080/health || echo "000 0")
http_code=$(awk '{print $1}' <<<"$http_code_latency")
latency_ms=$(awk '{printf "%.0f", $2*1000}' <<<"$http_code_latency")

timestamp=$(date -Iseconds)

# Columns: ts,dataset,files,bytes,mismatch,http,lat_ms
printf "%s,%s,%s,%s,%s,%s,%s\n" \
  "$timestamp" "$DATASET" "$files_restored" "$bytes_restored" "$sha_mismatch" "$http_code" "$latency_ms" \
  | tee -a metrics_history.csv > metrics_latest.csv

echo "[*] Metrics written to metrics_history.csv and metrics_latest.csv"

Make it executable:

chmod +x restore_test.sh

Customize the health checks:

  • Add database sanity queries

  • Validate presence of critical config files

  • Measure directory sizes that should be stable

  • Include app-specific smoke tests

Each metric becomes a feature for the model.

Step 4: Add AI anomaly detection

We’ll use a small IsolationForest model to learn your “normal” restore metrics and flag outliers. This isn’t a giant LLM—just practical ML with a few lines of Python.

Save as score_restore.py:

#!/usr/bin/env python3
import pandas as pd
from sklearn.ensemble import IsolationForest
import sys

hist = pd.read_csv('metrics_history.csv', header=None, names=['ts','dataset','files','bytes','mismatch','http','lat_ms'])
latest = pd.read_csv('metrics_latest.csv', header=None, names=['ts','dataset','files','bytes','mismatch','http','lat_ms'])

ds = latest['dataset'].iloc[0]
base = hist[hist['dataset'] == ds].tail(200)  # use recent history

def fe(df):
    return df[['files','bytes','mismatch','http','lat_ms']].astype(float)

if len(base) < 10:
    print("WARN: not enough baseline, using rules only")
    ok = (float(latest['mismatch'].iloc[0]) == 0.0) and (str(int(float(latest['http'].iloc[0]))) == '200')
    print("OK" if ok else "ANOMALY")
    sys.exit(0 if ok else 2)

X = fe(base)
clf = IsolationForest(n_estimators=200, contamination=0.05, random_state=42)
clf.fit(X)

pred = clf.predict(fe(latest))[0]   # 1 = normal, -1 = outlier
score = clf.decision_function(fe(latest))[0]

print(f"SCORE={score:.4f} PRED={'OK' if pred==1 else 'ANOMALY'}")
sys.exit(0 if pred == 1 else 2)

Run the full flow:

./restore_test.sh
. .venv/bin/activate
python3 score_restore.py

  • Exit code 0 = OK

  • Exit code 2 = anomaly (useful for CI/systemd to alert)

Tip: Version-control your scripts and keep metrics_history.csv per dataset for clean baselines.

Step 5: Automate and alert

Use systemd timers for a nightly run.

Service unit (save as /etc/systemd/system/restore-test@.service):

[Unit]
Description=Automated restore test for %i

[Service]
Type=oneshot
WorkingDirectory=/opt/restore-tests/%i
Environment=DATASET=%i
ExecStart=/bin/bash restore_test.sh
ExecStartPost=/opt/restore-tests/.venv/bin/python3 /opt/restore-tests/%i/score_restore.py
# Optional simple alert to syslog if anomaly:
ExecStartPost=/bin/sh -c '[ "$$?" -eq 0 ] || logger -t restore-test "%i anomaly detected"'

Timer (save as /etc/systemd/system/restore-test@.timer):

[Unit]
Description=Nightly restore test for %i

[Timer]
OnCalendar=02:30
Persistent=true
Unit=restore-test@%i.service

[Install]
WantedBy=timers.target

Enable it for your dataset “app-prod”:

sudo systemctl daemon-reload
sudo systemctl enable --now restore-test@app-prod.timer

Hook into your notifier of choice:

  • Sendmail/mailx: run on non-zero exit

  • Slack/Matrix/Webhooks: curl a JSON payload if score_restore.py exits 2

Real-world-style example

  • Symptom: A service restored “fine” and returned HTTP 200, but p95 latency doubled due to a missing tmpfs mount restored to disk (slow I/O path).

  • Old approach: Green checkmark because exit codes were 0.

  • With AI scoring: Historical latency was ~25 ms; restored latency hit 80–100 ms—flagged as an anomaly even though “it worked.” The fix landed before the next incident.

Hardening tips

  • Keep manifests for mission-critical data sets—checksum mismatches are a fast fail.

  • Maintain multiple features: file counts, sizes, p95 latency, 200/500 ratios, DB sanity checks.

  • Start with generous thresholds; let the model learn a few weeks of history before tightening alerting.

  • Treat AI as augmentation, not replacement, for periodic full disaster recovery drills.

Conclusion and Call to Action

Backups without restores are hope. Restores without tests are risk. Put this into action: 1) Install the prerequisites and drop the scripts into version control. 2) Wire your actual restore commands and health checks. 3) Schedule the nightly systemd timer and let the baseline grow. 4) Review anomalies and add features that capture your app’s real health.

When the next pager goes off, you won’t be guessing—you’ll be restoring with confidence.

Related posts