Posted on
• Artificial Intelligence

Artificial Intelligence Backup Validation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Assisted Backup Validation on Linux: Catch Silent Failures Before They Bite

Backups only matter when you can restore them. Yet too many teams discover—during a crisis—that their “successful” backups are corrupted, incomplete, or encrypted by ransomware. The logs looked fine. Checksums matched. But something subtle was wrong.

This post shows how to add a thin layer of “artificial intelligence” to your Linux backup validation using Bash and a small Python helper. You’ll still use your favorite tools (restic, borg, rsync…), but you’ll also teach a simple model what “normal” looks like for your backups and alert when a new snapshot looks off.

Value in a nutshell:

  • Verify sample restores with checksums (hands-on proof you can recover).

  • Track compact snapshot “fingerprints” over time.

  • Use anomaly detection to flag weirdness: sudden file-count drops, entropy spikes, binary/text mix shifts—symptoms often seen with misconfigurations and ransomware.

What follows is practical, privacy-safe, and lightweight enough to run from cron.

What “AI Backup Validation” Is (and isn’t)

  • Is: Metrics-driven sanity checks enhanced with machine learning. You collect compact metadata (counts, sizes, extension diversity, compression proxy/entropy, etc.) per snapshot and teach a model to spot odd changes.

  • Isn’t: Shipping your data to the cloud or handing your backup contents to a black box. The model learns from simple numeric metrics only.

Prerequisites and Installation

We’ll use:

  • A backup tool (examples with restic; works similarly with borg)

  • Standard UNIX tools (find, gzip, file)

  • Python 3 plus a few pip packages

  • Optional: FUSE (if you prefer mounting repositories)

Install system packages with your package manager:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y restic borgbackup jq python3 python3-pip gzip file fuse3

Fedora/RHEL/CentOS (dnf):

# On RHEL/CentOS, you may need EPEL for restic/borg
sudo dnf -y install epel-release || true
sudo dnf -y install restic borgbackup jq python3 python3-pip gzip file fuse3

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y restic borgbackup jq python3 python3-pip gzip file fuse3

Install the Python libraries (user-local is fine):

pip3 install --user --upgrade pip
pip3 install --user pandas scikit-learn joblib
# Ensure ~/.local/bin is on your PATH (if needed):
# echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

Set your restic environment (example):

export RESTIC_REPOSITORY=/mnt/backups/myrepo
export RESTIC_PASSWORD_FILE=/root/.restic-pass

Core Workflow: 4 Steps That Actually Catch Problems

1) Before the backup: capture a small sample manifest with checksums

You don’t need to hash everything—just a random sample of small-to-medium files is enough as a smoke test for restore integrity.

Create sample-manifest.sh:

#!/usr/bin/env bash
set -euo pipefail
SRC="${1:-/data}"       # directory you back up
OUT="${2:-sample.sha}"  # output manifest (sha256sum format)
N="${3:-30}"            # sample size (keep small for speed)

# Random sample of files under 100MB, then compute sha256 for each
find "$SRC" -type f -size -100M -print0 | shuf -z -n "$N" | xargs -0 -r sha256sum > "$OUT"
echo "Wrote $(wc -l < "$OUT") entries to $OUT"

Usage:

bash sample-manifest.sh /data /var/backups/sample.sha 30

Run this just before your backup job so the manifest reflects current state.

2) After the backup: perform a sample restore and verify checksums (restic)

This confirms your repository can actually produce files that match the originals. We’ll restore a small sample and compare hashes.

Create validate-restore-restic.sh:

#!/usr/bin/env bash
set -euo pipefail
SAMPLE="${1:-/var/backups/sample.sha}"  # file created by sample-manifest.sh
TARGET="${2:-/tmp/restore-check}"       # where we temporarily restore
SNAP="${3:-latest}"                     # which snapshot to test (restic syntax)

mkdir -p "$TARGET"

fail=0
total=$(wc -l < "$SAMPLE" | awk '{print $1}')
echo "Validating restore for $total sampled files into $TARGET"

# Iterate the sample; restore each path and verify checksum
# sha256sum output format: <SUM><two spaces><PATH>
while IFS= read -r line; do
  sum="${line%% *}"
  path="$(printf '%s' "$line" | sed -E 's/^[0-9a-fA-F]{64}[[:space:]]{2}//')"
  # Ensure containing directory exists in target
  mkdir -p "$(dirname "$TARGET$path")"
  # Restore just this file
  restic restore "$SNAP" --target "$TARGET" --include "$path" >/dev/null

  if [ ! -f "$TARGET$path" ]; then
    echo "MISSING: $path"
    fail=$((fail+1))
    continue
  fi
  rsum=$(sha256sum "$TARGET$path" | awk '{print $1}')
  if [ "$rsum" != "$sum" ]; then
    echo "MISMATCH: $path"
    fail=$((fail+1))
  fi
done < "$SAMPLE"

if [ "$fail" -gt 0 ]; then
  echo "Sample restore FAILED ($fail mismatches)"
  exit 2
else
  echo "Sample restore PASSED"
fi

Notes:

  • Keep your sample small (e.g., 20–50 files) for speed.

  • For borg users: swap the “restore” logic with borg extract --dry-run or real extracts for samples; the checksum logic is the same.

3) Record compact snapshot metrics (“fingerprints”) over time

We’ll compute a lightweight set of metrics—no content leaves the machine. These features often reveal ransomware or broken jobs (e.g., file count collapse, entropy spike, sudden binary/text mix changes).

Create metrics.sh:

#!/usr/bin/env bash
set -euo pipefail
ROOT="${1:-/tmp/restore-check}"  # analyze the sample restore dir or your backup target
OUT="${2:-metrics.csv}"

ts=$(date -Iseconds)

total_files=$(find "$ROOT" -type f | wc -l | awk '{print $1}')
total_bytes=$(find "$ROOT" -type f -printf '%s\n' | awk '{s+=$1} END{print s+0}')
avg_size=$(awk -v b="$total_bytes" -v n="$total_files" 'BEGIN{ if(n>0) printf "%.2f", b/n; else printf "0" }')
ext_div=$(find "$ROOT" -type f -printf '%f\n' | awk -F. 'NF>1 {print tolower($NF)}' | sort -u | wc -l | awk '{print $1}')
mean_depth=$(find "$ROOT" -mindepth 1 -printf '%d\n' | awk '{s+=$1;c++} END{ if(c>0) printf "%.2f", s/c; else printf "0" }')

# Sample up to 200 files under 20MB for compression and binary/text ratios
mapfile -t sample < <(find "$ROOT" -type f -size -20M | shuf -n 200)
orig_sum=0; comp_sum=0; bin_count=0; n=0
for f in "${sample[@]}"; do
  [ -f "$f" ] || continue
  s=$(wc -c < "$f")
  c=$(gzip -c "$f" | wc -c)
  mime=$(file -b --mime-type "$f" || true)
  ((orig_sum+=s))
  ((comp_sum+=c))
  if [[ "$mime" != text/* ]]; then ((bin_count++)); fi
  ((n++))
done
comp_ratio=$(awk -v c="$comp_sum" -v o="$orig_sum" 'BEGIN{ if(o>0) printf "%.4f", c/o; else printf "0" }')
bin_ratio=$(awk -v b="$bin_count" -v n="$n" 'BEGIN{ if(n>0) printf "%.4f", b/n; else printf "0" }')

if [ ! -f "$OUT" ]; then
  echo "timestamp,total_files,total_bytes,avg_size,ext_diversity,mean_depth,comp_ratio,bin_ratio" > "$OUT"
fi
echo "$ts,$total_files,$total_bytes,$avg_size,$ext_div,$mean_depth,$comp_ratio,$bin_ratio" >> "$OUT"
echo "Appended snapshot metrics to $OUT"

Tip: Point ROOT to your full restore location if you do periodic full tests. For speed, it’s fine to analyze the sample restore; the anomaly detector focuses on trend shifts rather than absolute numbers.

4) Train a simple anomaly detector and alert on weird snapshots

We’ll use IsolationForest (unsupervised ML) to learn “normal” from historical metrics and flag anomalies. We’ll also include straightforward rule checks so you get useful reasons.

Create train-validate.py:

#!/usr/bin/env python3
import sys
import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest
from joblib import dump

def main(csv_path):
    df = pd.read_csv(csv_path)
    if df.empty:
        print("No metrics to analyze.")
        sys.exit(0)

    feats = ["total_files","total_bytes","avg_size","ext_diversity","mean_depth","comp_ratio","bin_ratio"]
    for col in feats:
        if col not in df.columns:
            print(f"Missing feature: {col}")
            sys.exit(1)

    # Use all but the last row for training when we have enough history
    if len(df) >= 6:
        train = df.iloc[:-1][feats].astype(float)
        test = df.iloc[[-1]][feats].astype(float)
        model = IsolationForest(random_state=42, contamination=0.05)
        model.fit(train)
        pred = model.predict(test)[0]  # 1=normal, -1=anomaly
        score = model.decision_function(test)[0]
        dump(model, "backup_anomaly_model.joblib")
    else:
        # Not enough history; default to "normal" and lean on heuristics
        pred = 1
        score = 0.0

    # Heuristic checks with explanations
    reasons = []
    if len(df) >= 3:
        hist = df.iloc[:-1][feats].astype(float)
        cur = df.iloc[-1][feats].astype(float)
        med = hist.median()
        eps = 1e-9

        def rel_change(new, base):
            return float((new - base) / (base + eps))

        # Core sanity checks (tweak thresholds to taste)
        if rel_change(cur["total_files"], med["total_files"]) < -0.15:
            reasons.append("total_files down >15% vs median")
        if rel_change(cur["total_bytes"], med["total_bytes"]) < -0.15:
            reasons.append("total_bytes down >15% vs median")
        if cur["comp_ratio"] > max(0.95, med["comp_ratio"] + 0.15):
            reasons.append("comp_ratio unusually high (high entropy / encrypted?)")
        if cur["bin_ratio"] > min(1.0, med["bin_ratio"] + 0.20):
            reasons.append("bin_ratio jumped >20% (more binaries than usual)")
        if rel_change(cur["ext_diversity"], med["ext_diversity"]) < -0.25:
            reasons.append("extension diversity collapsed >25% (missing file types?)")

    anomalous = (pred == -1) or bool(reasons)

    # Human-friendly output and exit code for CI/cron
    status = "FAIL" if anomalous else "PASS"
    print(f"AI validation: {status} (iso_forest_score={score:+.4f})")
    if reasons:
        print("Reasons:")
        for r in reasons:
            print(f" - {r}")

    sys.exit(2 if anomalous else 0)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python3 train-validate.py metrics.csv")
        sys.exit(1)
    main(sys.argv[1])

Run it:

python3 train-validate.py metrics.csv
# Exit code 0 = PASS, non-zero = anomaly detected

That’s it: you now have both a concrete restore check and an AI-backed sanity classifier for your snapshots.

Glue It Together (Cron/Systemd)

Example daily flow with restic:

  • Pre-backup: sample-manifest.sh to pick files + checksums.

  • Run your restic backup.

  • Post-backup: validate-restore-restic.sh on the sample; metrics.sh to record features; train-validate.py to flag anomalies.

Sample cron (runs at 02:30):

SHELL=/bin/bash
MAILTO=you@example.com
30 2 * * * export RESTIC_REPOSITORY=/mnt/backups/myrepo RESTIC_PASSWORD_FILE=/root/.restic-pass && \
  /usr/local/bin/sample-manifest.sh /data /var/backups/sample.sha 30 && \
  restic backup /data && \
  /usr/local/bin/validate-restore-restic.sh /var/backups/sample.sha /tmp/restore-check latest && \
  /usr/local/bin/metrics.sh /tmp/restore-check /var/backups/metrics.csv && \
  python3 /usr/local/bin/train-validate.py /var/backups/metrics.csv

Tip: Keep metrics.csv under version control or sync it; the model improves with history.

Real-World Signals This Catches

  • Ransomware: comp_ratio spikes toward 1.0 and bin_ratio jumps as files become high-entropy. Extension diversity may also collapse if extensions are mass-changed.

  • Misconfiguration: total_files or total_bytes drop 20–40% after someone excludes a directory or a mount wasn’t present during backup.

  • Silent corruption: Sample restore checksum mismatches catch data drift, flaky storage, or damaged repo packs long before you need a real restore.

Conclusion and Next Steps (CTA)

Small, actionable layers beat grand promises:

  • Keep performing sample restores with checksums.

  • Log compact metrics per snapshot.

  • Let a simple model learn your normal and flag the weird.

Next steps:

  • Drop these scripts into your backup pipeline and run them nightly.

  • Tune thresholds in train-validate.py as your history grows.

  • Add notifications (email, Slack, Prometheus alerts) on non-zero exits.

  • Periodically do a full restore rehearsal on a spare host.

If you found this useful, wire it into one environment this week. Your future self (awake at 3 AM during an outage) will thank you.

Related posts