Posted on
• Artificial Intelligence

Artificial Intelligence Home Backup Systems

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Home Backup Systems (for Bash-first Linux users)

What if your NAS woke up encrypted and your “backup” dutifully synchronized the damage? That’s the nightmare. The opportunity: layer AI-style anomaly detection and intelligent guards around a rock‑solid, Bash-driven backup so it resists user error, ransomware, and silent data rot.

In this post you’ll:

  • Build a reliable, encrypted, deduplicated baseline with restic + rclone

  • Add an AI anomaly detector to spot suspicious backup deltas

  • Install a write‑storm watchdog to pause backups during a likely ransomware event

  • Automate quick restore drills so you can trust your backups

All examples are shell-first. Every tool includes apt, dnf, and zypper install commands.

Why AI for home backups?

  • Backups are only as good as what they capture. If a ransomware event or runaway script mutates thousands of files, a naïve backup may hurry to preserve the damage.

  • Most backup tools don’t “understand” what’s normal for your home dataset. An anomaly layer can spot weird patterns (huge spikes in changed files, bytes added, or duration) and halt or alert before bad data spreads.

  • With a tiny bit of machine learning on top of standard Linux tooling, you can get early-warning signals and safer automation—without buying a black-box appliance.

What you’ll build

1) A restic-based, encrypted backup to local and/or cloud targets (via rclone) 2) A Python-based anomaly detector (IsolationForest) that learns your normal backup patterns and flags spikes 3) An inotify-based write-rate watchdog that can freeze backups during suspicious write storms 4) A restore drill that randomly verifies files and checksums

1) Baseline: encrypted, deduplicated backups with restic + rclone

restic is fast, encrypted, deduplicated, and script-friendly. rclone gives you cloud targets.

Install:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y restic rclone
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y restic rclone
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y restic rclone

Optional: set up a cloud remote with rclone (S3, Backblaze B2, etc.):

rclone config

Example repo targets:

  • Local disk: /mnt/backup/restic

  • Cloud via rclone: rclone:myremote:home-restic

Create a password file and secure it:

mkdir -p ~/.config/restic
chmod 700 ~/.config/restic
echo "choose-a-long-unique-passphrase" > ~/.config/restic/password
chmod 600 ~/.config/restic/password

Initialize the repository:

export RESTIC_PASSWORD_FILE=$HOME/.config/restic/password
export RESTIC_REPOSITORY=/mnt/backup/restic   # or "rclone:myremote:home-restic"
restic init

Create an exclude file:

mkdir -p ~/.config/restic
cat > ~/.config/restic/excludes <<'EOF'
*.tmp
*.cache
node_modules/
.target/
.DS_Store
Thumbs.db
/lost+found
/mnt/*
/proc/*
/sys/*
/dev/*
/run/*
/var/tmp/*
EOF

A practical backup script (with JSON logging for AI analysis):

#!/usr/bin/env bash
set -euo pipefail

# --- config ---
export RESTIC_PASSWORD_FILE="${HOME}/.config/restic/password"
export RESTIC_REPOSITORY="${RESTIC_REPOSITORY:-/mnt/backup/restic}" # override via env if using rclone
BACKUP_SOURCES=(
  "$HOME/Documents"
  "$HOME/Pictures"
  "/etc"
)
EXCLUDES="${HOME}/.config/restic/excludes"
LOGDIR="/var/log/restic-ai"
FREEZE_FILE="/var/run/backup.freeze"
RETENTION_ARGS="--keep-daily 7 --keep-weekly 4 --keep-monthly 6"
HOSTTAG="$(hostname -s)"

# --- prep ---
sudo mkdir -p "${LOGDIR}"
sudo chown "$(id -u)":"$(id -g)" "${LOGDIR}"

if [[ -f "${FREEZE_FILE}" ]]; then
  echo "[backup] freeze file present (${FREEZE_FILE}); skipping backup."
  exit 0
fi

ts=$(date -u +%Y%m%dT%H%M%SZ)
logjson="${LOGDIR}/backup-${HOSTTAG}-${ts}.json"

# --- run backup (JSON) ---
restic backup \
  --json \
  --hostname "${HOSTTAG}" \
  --tag "auto" \
  --exclude-file "${EXCLUDES}" \
  "${BACKUP_SOURCES[@]}" | tee "${logjson}"

# --- AI anomaly check (optional; see section 2) ---
if command -v python3 >/dev/null 2>&1; then
  if python3 /usr/local/bin/ai_anomaly_guard.py --logdir "${LOGDIR}" ; then
    echo "[backup] anomaly guard: normal"
  else
    echo "[backup] anomaly guard: SUSPICIOUS; aborting retention to avoid propagating damage."
    exit 1
  fi
fi

# --- retention/prune ---
restic forget ${RETENTION_ARGS} --prune

Make it executable and test:

chmod +x ~/backup.sh
RESTIC_REPOSITORY=/mnt/backup/restic ~/backup.sh

Automate via cron (daily at 03:30):

crontab -e
# add:
30 3 * * * RESTIC_REPOSITORY=/mnt/backup/restic /home/youruser/backup.sh >> /var/log/backup.cron 2>&1

Or a systemd timer if you prefer (not shown to keep this post focused).

2) AI anomaly detection on backup deltas

We’ll train a simple IsolationForest on your backup history (from restic’s JSON summaries). It learns “normal” change patterns and flags spikes that could imply ransomware or accidental mass edits.

Install Python and pip:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y python3 python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y python3 python3-pip
  • openSUSE (zypper):
sudo zypper install -y python3 python3-pip

Install Python libs (user-local is fine):

pip3 install --user pandas scikit-learn

Create the anomaly script at /usr/local/bin/ai_anomaly_guard.py:

#!/usr/bin/env python3
import argparse, json, os, glob, sys
import pandas as pd
from sklearn.ensemble import IsolationForest

def extract_summary(json_path):
    # restic emits multi-line JSON; summary lines include "message_type":"summary"
    rows = []
    with open(json_path, 'r', encoding='utf-8', errors='ignore') as f:
        for line in f:
            line=line.strip()
            if not line:
                continue
            try:
                obj = json.loads(line)
            except Exception:
                continue
            if isinstance(obj, dict) and obj.get("message_type") == "summary":
                # Known fields (version dependent). Fallbacks if missing.
                s = obj.get("summary", obj)  # sometimes fields may be top-level
                files_new = s.get("files_new", 0)
                files_changed = s.get("files_changed", 0)
                files_unmodified = s.get("files_unmodified", 0)
                dirs_new = s.get("dirs_new", 0)
                data_added = s.get("data_added", s.get("bytes_added", 0))
                total_bytes_processed = s.get("total_bytes_processed", 0)
                duration = s.get("total_duration", s.get("duration", 0.0)) or 0.0
                rows.append({
                    "files_new": files_new,
                    "files_changed": files_changed,
                    "files_unmodified": files_unmodified,
                    "dirs_new": dirs_new,
                    "data_added": data_added,
                    "total_bytes_processed": total_bytes_processed,
                    "duration_s": duration,
                    "bps": (total_bytes_processed / duration) if duration else 0.0
                })
    return rows

def load_history(logdir):
    data = []
    for path in sorted(glob.glob(os.path.join(logdir, "backup-*.json"))):
        data.extend(extract_summary(path))
    if not data:
        return pd.DataFrame()
    return pd.DataFrame(data)

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--logdir", required=True)
    ap.add_argument("--min-samples", type=int, default=8, help="minimum history to train")
    ap.add_argument("--contamination", type=float, default=0.12, help="expected anomaly rate")
    args = ap.parse_args()

    df = load_history(args.logdir)
    if df.empty or len(df) < args.min_samples:
        # Not enough data to judge; allow backup to proceed.
        print("[ai] insufficient history; passing")
        sys.exit(0)

    features = df[["files_new","files_changed","data_added","total_bytes_processed","bps"]].fillna(0.0)
    clf = IsolationForest(
        n_estimators=200,
        contamination=args.contamination,
        random_state=42
    )
    clf.fit(features.iloc[:-1, :])
    pred = clf.predict(features.iloc[[-1], :])[0]  # 1 normal, -1 anomaly
    score = clf.decision_function(features.iloc[[-1], :])[0]

    last = df.iloc[-1].to_dict()
    msg = f"[ai] last backup: files_new={last['files_new']} changed={last['files_changed']} added={last['data_added']} bps={int(last['bps'])} score={score:.3f}"
    if pred == -1:
        print(msg + " -> ANOMALY")
        # Non-zero exit to signal backup script to stop retention/prune
        sys.exit(1)
    else:
        print(msg + " -> normal")
        sys.exit(0)

if __name__ == "__main__":
    main()

Make it executable:

sudo install -m 0755 ai_anomaly_guard.py /usr/local/bin/ai_anomaly_guard.py

Your backup.sh already calls this after each run. If it flags “SUSPICIOUS,” the script aborts before retention/prune, reducing the chance of overwriting good history with bad.

Tip: After a handful of normal backups (8–10), the detector becomes more useful.

3) Real-time write‑storm watchdog (pause backups during chaos)

If a process begins rewriting thousands of files quickly, let a lightweight watcher raise a freeze flag so your scheduled backup won’t run into the fire.

Install inotify-tools:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y inotify-tools
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y inotify-tools
  • openSUSE (zypper):
sudo zypper install -y inotify-tools

A simple watchdog:

#!/usr/bin/env bash
set -euo pipefail

WATCH_PATHS=("$HOME/Documents" "$HOME/Pictures")
EVENTS="modify,create,delete,move"
WINDOW_SEC=60
THRESHOLD=800           # tune for your dataset
FREEZE_FILE="/var/run/backup.freeze"

declare -a bucket
count=0
start=$(date +%s)

inotifywait -m -r -e ${EVENTS} --format '%e %w%f' "${WATCH_PATHS[@]}" | while read -r _; do
  now=$(date +%s)
  if (( now - start >= WINDOW_SEC )); then
    if (( count >= THRESHOLD )); then
      echo "[watchdog] high event rate (${count}/${WINDOW_SEC}s). Creating freeze file: ${FREEZE_FILE}"
      sudo touch "${FREEZE_FILE}"
    fi
    count=0
    start=$now
  fi
  ((count++))
done

  • Run it as a user service or tmux session.

  • Your backup.sh already checks for /var/run/backup.freeze and exits harmlessly.

  • Clear the freeze after investigating:

sudo rm -f /var/run/backup.freeze

This isn’t “AI,” but it’s a pragmatic guardrail that pairs well with anomaly detection.

4) Trust but verify: automatic restore drills

Backups you haven’t restored are just hopeful copies. This quick drill restores a random file from the latest snapshot and compares checksums.

#!/usr/bin/env bash
set -euo pipefail

export RESTIC_PASSWORD_FILE="${HOME}/.config/restic/password"
export RESTIC_REPOSITORY="${RESTIC_REPOSITORY:-/mnt/backup/restic}"

TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT

SNAP=$(restic snapshots --json | awk -F'"' '/"short_id"/{print $4}' | tail -n1)
[[ -z "${SNAP}" ]] && { echo "No snapshots found"; exit 1; }

# pick a random file path from the snapshot listing
FILE=$(restic ls "${SNAP}" --json | awk -F'"' '/"path":/ {print $4}' | shuf -n1)

[[ -z "${FILE}" ]] && { echo "No files in snapshot"; exit 1; }

echo "[verify] testing ${FILE} from snapshot ${SNAP}"
# compute original checksum if file still exists locally; otherwise skip source hash
if [[ -f "${FILE}" ]]; then
  SRC_HASH=$(sha256sum "${FILE}" | awk '{print $1}')
else
  SRC_HASH=""
fi

restic restore "${SNAP}" --target "${TMPDIR}" --include "${FILE}" >/dev/null
RESTORED="${TMPDIR}${FILE}"

if [[ -f "${RESTORED}" && -n "${SRC_HASH}" ]]; then
  DST_HASH=$(sha256sum "${RESTORED}" | awk '{print $1}')
  if [[ "${SRC_HASH}" == "${DST_HASH}" ]]; then
    echo "[verify] OK: hashes match"
    exit 0
  else
    echo "[verify] WARNING: hash mismatch"
    exit 2
  fi
else
  echo "[verify] Restored file present: $(test -f "${RESTORED}" && echo yes || echo no)."
  exit 0
fi

Schedule it weekly and alert on non-zero exit via your MTA or monitoring.

Real-world workflow

  • Daily at 03:30: backup.sh runs restic, logs JSON, calls AI guard. If normal, it prunes per retention.

  • Watchdog runs continuously. If a write-storm starts, it creates /var/run/backup.freeze so scheduled backups skip.

  • Weekly: restore drill runs a random-file check to keep you honest.

  • If the AI guard flags an anomaly, investigate before clearing the freeze or allowing the next run.

Tuning tips

  • Start conservative: lower the watchdog THRESHOLD only after observing your normal event rate.

  • IsolationForest contamination around 0.1–0.15 works well to catch spikes without too many false alarms. Adjust as you build history.

  • Keep your password file offsite or in a sealed envelope; without it, your backups are unrecoverable (by design).

Conclusion and next steps

You now have a Bash-first backup that’s:

  • Encrypted and deduplicated (restic)

  • Cloud-capable (rclone)

  • AI‑assisted to detect anomalies (IsolationForest)

  • Guarded against write storms (inotify)

  • Verified by routine restore drills

Next steps:

  • Add an offsite target (e.g., B2, S3, rsync.net via rclone).

  • Expand anomaly features (e.g., file extension entropy, per-directory deltas).

  • Integrate notifications (systemd-notify, desktop notify-send, or mail on failure).

  • Consider air‑gapped weekly snapshots for true last‑resort recovery.

If this helped, try implementing one guard this week—the anomaly detector or the watchdog—and schedule a restore drill. Your future self will thank you.

Related posts