Posted on
• Artificial Intelligence

Artificial Intelligence Linux Disaster Recovery

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Driven Linux Disaster Recovery: Bash-first, Ops-ready

It’s 03:12, the pager screams, and your logs look like static. Disk errors spike, SSH failures bloom, and your team is still rubbing sleep out of their eyes. Minutes matter. This is exactly where a small dose of practical AI, wrapped in battle-tested Bash, can turn “we think” into “we know” — and trigger the right recovery action automatically.

In this article, you’ll build a lightweight AI-assisted disaster recovery (DR) workflow that:

  • Baselines your system logs and detects anomalies in near real time.

  • Verifies backups automatically (with actual restore tests).

  • Watches storage health (SMART) and adapts backup behavior.

  • Runs fully on Linux using Bash + Python, no external cloud needed.

The goal isn’t hype. It’s signal over noise, fewer false alarms, faster restores.

Why AI here — and why now?

  • Logs and metrics have outgrown “grep + intuition.” Even small fleets produce enough noise to hide real failures. A tiny unsupervised model can spot weirdness faster than humans scanning screens.

  • Backups are only as good as your last proven restore. AI can score when to verify and what to test, then run the drills without waiting for a human.

  • Storage fails gradually, then suddenly. Early hints in SMART data can prompt you to snapshot, back up, and plan a replacement before data loss.

You don’t need a GPU cluster or a PhD — just Python, a few packages, and the Unix philosophy: do one thing well, then glue it together with Bash.

What you’ll build

  • A Bash script to extract features from logs/metrics.

  • A small Python model (Isolation Forest) to flag anomalies.

  • A DR wrapper to auto-verify backups (restic/borg) and test-restore.

  • A SMART health check that increases backup urgency on risk.

Prerequisites and installation

Below are minimal packages you’ll need. Pick the section for your distro.

  • Components:
    • Python 3, pip, venv
    • restic and/or borgbackup
    • jq, rsync
    • smartmontools
    • a mail utility (mailutils or mailx)
    • cron service

Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip \
  restic borgbackup rsync jq smartmontools \
  mailutils cron git

sudo systemctl enable --now cron

Fedora/RHEL (dnf)

sudo dnf install -y \
  python3 python3-virtualenv python3-pip \
  restic borgbackup rsync jq smartmontools \
  mailx cronie git

sudo systemctl enable --now crond

openSUSE (zypper)

sudo zypper refresh
sudo zypper install -y \
  python3 python3-virtualenv python3-pip \
  restic borgbackup rsync jq smartmontools \
  mailx cron git

sudo systemctl enable --now cron

Create a working directory and Python environment:

sudo mkdir -p /opt/ai-dr /var/lib/ai-dr /etc/ai-dr
sudo chmod 750 /opt/ai-dr /var/lib/ai-dr
sudo python3 -m venv /opt/ai-dr/venv
sudo /opt/ai-dr/venv/bin/pip install --upgrade pip
sudo /opt/ai-dr/venv/bin/pip install scikit-learn pandas numpy joblib

Set up a restic repository (local example; adapt for S3/SFTP/etc.):

sudo mkdir -p /backup/restic-repo
sudo mkdir -p /etc/restic
echo 'change_this_very_secret' | sudo tee /etc/restic/secret >/dev/null
sudo chmod 600 /etc/restic/secret

export RESTIC_REPOSITORY=/backup/restic-repo
export RESTIC_PASSWORD_FILE=/etc/restic/secret
restic init

If you use borg instead:

sudo mkdir -p /backup/borg-repo
borg init --encryption=repokey-blake2 /backup/borg-repo

Step 1: Collect signals with Bash (features from logs/metrics)

We’ll extract simple features from recent logs and system state. Start with journalctl (systemd). If your distro doesn’t use journald, adapt the grep source paths to /var/log/*.

Create /usr/local/sbin/log_features.sh:

#!/usr/bin/env bash
set -euo pipefail

OUT_DIR="/var/lib/ai-dr"
OUT_CSV="$OUT_DIR/features.csv"
WINDOW="5 minutes ago"  # sliding window
TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Use journalctl if available, else tail /var/log/syslog or /var/log/messages
if command -v journalctl >/dev/null 2>&1; then
  LOGSRC=$(journalctl --since "$WINDOW" -o short-iso 2>/dev/null || true)
else
  # Fallback: gather from common log files (adjust for your distro)
  LOGSRC=$( (tail -n 5000 /var/log/syslog 2>/dev/null; tail -n 5000 /var/log/messages 2>/dev/null) || true )
fi

# Basic counts
total_lines=$(printf "%s\n" "$LOGSRC" | wc -l || echo 0)
err_lines=$(printf "%s\n" "$LOGSRC" | grep -Ei 'error|failed|segfault|panic|i/o error|kernel oops|oom' | wc -l || echo 0)

# SSH auth failures (journal or auth.log)
if command -v journalctl >/dev/null 2>&1; then
  ssh_fail=$(journalctl -u ssh -u sshd --since "$WINDOW" 2>/dev/null | grep -E 'Failed password|Invalid user' | wc -l || echo 0)
else
  ssh_fail=$(grep -E 'Failed password|Invalid user' /var/log/auth.log 2>/dev/null | tail -n 5000 | wc -l || echo 0)
fi

# OOM kills
oom_kills=$(printf "%s\n" "$LOGSRC" | grep -E 'Out of memory|Killed process' | wc -l || echo 0)

# Disk I/O errors
io_err=$(printf "%s\n" "$LOGSRC" | grep -Ei 'I/O error|EXT4-fs error|Buffer I/O error' | wc -l || echo 0)

# Load average snapshot
load_1=$(cut -d ' ' -f1 /proc/loadavg)

# Header if new file
if [ ! -f "$OUT_CSV" ]; then
  echo "timestamp,total_lines,err_lines,ssh_fail,oom_kills,io_err,load_1" | sudo tee "$OUT_CSV" >/dev/null
  sudo chmod 640 "$OUT_CSV"
fi

echo "$TS,$total_lines,$err_lines,$ssh_fail,$oom_kills,$io_err,$load_1" | sudo tee -a "$OUT_CSV" >/dev/null

sudo install -m 0750 /usr/local/sbin/log_features.sh /usr/local/sbin/log_features.sh

Run it manually once to seed data:

sudo /usr/local/sbin/log_features.sh

Step 2: Train and score anomalies (tiny Python + Isolation Forest)

Create /opt/ai-dr/train_detect.py:

#!/usr/bin/env python3
import os, sys, json
import pandas as pd
from sklearn.ensemble import IsolationForest
from joblib import dump, load

FEATURES_CSV = "/var/lib/ai-dr/features.csv"
MODEL_PATH = "/var/lib/ai-dr/model.joblib"

def main():
    df = pd.read_csv(FEATURES_CSV)
    # Keep a reasonable window to adapt but not overfit
    df = df.tail(2000).reset_index(drop=True)

    feature_cols = ["total_lines","err_lines","ssh_fail","oom_kills","io_err","load_1"]
    X = df[feature_cols].fillna(0)

    # Train/update model when we have enough samples
    if len(X) < 50:
        print(json.dumps({"ready": False, "reason": "insufficient_data", "count": len(X)}))
        return

    clf = IsolationForest(
        n_estimators=200, contamination=0.02, random_state=42, n_jobs=-1
    )
    clf.fit(X[:-1])  # train on all but the latest row

    # Save model for reuse (optional)
    try:
        dump(clf, MODEL_PATH)
    except Exception:
        pass

    latest = X.iloc[[-1]]
    pred = clf.predict(latest)[0]           # -1 anomalous, 1 normal
    score = clf.decision_function(latest)[0]  # lower = more anomalous

    out = {
        "ready": True,
        "anomalous": (pred == -1),
        "score": float(score),
        "n_samples": int(len(X))
    }
    print(json.dumps(out))

if __name__ == "__main__":
    sys.exit(main())

sudo install -m 0750 /opt/ai-dr/train_detect.py /opt/ai-dr/train_detect.py

Test it:

sudo /usr/local/sbin/log_features.sh
sudo /opt/ai-dr/venv/bin/python /opt/ai-dr/train_detect.py

You should see JSON output with a score and anomaly flag after enough samples.

Step 3: Wire it to backups and real restore checks

Create an environment file for secrets and repo paths:

/etc/ai-dr/env:

# restic settings
export RESTIC_REPOSITORY=/backup/restic-repo
export RESTIC_PASSWORD_FILE=/etc/restic/secret

# what to back up and test-restore
export BACKUP_SRC="/etc /var/www /home"
export RESTORE_TEST_PATH="/var/tmp/ai-dr-restore"
export DR_ALERT_EMAIL="root@localhost"

sudo chmod 600 /etc/ai-dr/env

Create /usr/local/sbin/ai-dr.sh:

#!/usr/bin/env bash
set -euo pipefail

. /etc/ai-dr/env

VENV="/opt/ai-dr/venv"
FEATURES="/usr/local/sbin/log_features.sh"
PY="/opt/ai-dr/train_detect.py"

send_alert() {
  local subject="$1"
  local body="$2"
  if command -v mail >/dev/null 2>&1; then
    echo "$body" | mail -s "$subject" "$DR_ALERT_EMAIL"
  elif command -v mailx >/dev/null 2>&1; then
    echo "$body" | mailx -s "$subject" "$DR_ALERT_EMAIL"
  else
    logger -t ai-dr "ALERT: $subject - $body"
  fi
}

restic_backup() {
  . /etc/ai-dr/env
  restic backup $BACKUP_SRC --json | jq -C . || true
}

restic_verify_restore() {
  . /etc/ai-dr/env
  local tgt="${RESTORE_TEST_PATH}-$(date +%s)"
  mkdir -p "$tgt"
  # Restore a small but critical slice (adjust includes)
  restic restore latest --target "$tgt" --include /etc/hosts --include /etc/fstab
  # Basic integrity check
  if [ -f "$tgt/etc/hosts" ]; then
    logger -t ai-dr "Restore test OK at $tgt"
    echo "Restore test OK at $tgt"
  else
    send_alert "AI-DR restore test FAILED" "Could not find restored files in $tgt"
  fi
}

main() {
  sudo -n true >/dev/null 2>&1 || true

  $FEATURES
  local out
  out=$(/opt/ai-dr/venv/bin/python "$PY")
  echo "$out" | jq -C . || echo "$out"

  local ready
  ready=$(echo "$out" | jq -r '.ready // false' 2>/dev/null || echo "false")
  if [ "$ready" != "true" ]; then
    logger -t ai-dr "Model not ready yet"
    exit 0
  fi

  local anomalous
  anomalous=$(echo "$out" | jq -r '.anomalous' 2>/dev/null || echo "false")
  local score
  score=$(echo "$out" | jq -r '.score' 2>/dev/null || echo "0")

  if [ "$anomalous" = "true" ]; then
    send_alert "AI-DR anomaly detected (score=$score)" "Triggered backup verification and test restore."
    # 1) Run a fresh backup (optional but recommended during anomaly)
    restic_backup || true
    # 2) Verify repo health
    restic check --with-cache --read-data-subset=1/20 || send_alert "restic check FAILED" "Repo check failed"
    # 3) Perform a targeted test-restore
    restic_verify_restore
  fi
}

main "$@"

sudo install -m 0750 /usr/local/sbin/ai-dr.sh /usr/local/sbin/ai-dr.sh

Seed an initial backup and a first check:

sudo bash -lc '. /etc/ai-dr/env; restic backup $BACKUP_SRC'
sudo bash -lc '. /etc/ai-dr/env; restic check'

Optional: borg users can add a parallel verify block:

borg create --stats /backup/borg-repo::"$(hostname)-$(date +%F-%H%M)" $BACKUP_SRC
borg list /backup/borg-repo
borg extract --dry-run /backup/borg-repo::"latest" etc/hosts

Schedule via cron (runs every 5 minutes):

echo '*/5 * * * * root /usr/local/sbin/ai-dr.sh >>/var/log/ai-dr.log 2>&1' | sudo tee /etc/cron.d/ai-dr >/dev/null
sudo systemctl restart cron 2>/dev/null || sudo systemctl restart crond 2>/dev/null || true

Step 4: Watch storage health (SMART) and adapt behavior

SMART counters often drift before failure. If they tick up, back up now and plan a swap.

Create /usr/local/sbin/smart_guard.sh:

#!/usr/bin/env bash
set -euo pipefail
. /etc/ai-dr/env

BAD=0
REPORT=""

for d in /dev/sd? /dev/nvme?n1; do
  [ -e "$d" ] || continue
  out=$(smartctl -A "$d" 2>/dev/null || true)
  realloc=$(echo "$out" | awk '/Reallocated_Sector_Ct/ {print $10}' | tail -n1)
  pending=$(echo "$out" | awk '/Current_Pending_Sector/ {print $10}' | tail -n1)
  offline=$(echo "$out" | awk '/Offline_Uncorrectable/ {print $10}' | tail -n1)

  realloc=${realloc:-0}; pending=${pending:-0}; offline=${offline:-0}

  if [ "$realloc" -gt 0 ] || [ "$pending" -gt 0 ] || [ "$offline" -gt 0 ]; then
    BAD=1
    REPORT+="Device $d risk: realloc=$realloc pending=$pending offline=$offline\n"
  fi
done

if [ $BAD -eq 1 ]; then
  if command -v mail >/dev/null 2>&1; then
    echo -e "$REPORT" | mail -s "SMART risk detected" "$DR_ALERT_EMAIL"
  elif command -v mailx >/dev/null 2>&1; then
    echo -e "$REPORT" | mailx -s "SMART risk detected" "$DR_ALERT_EMAIL"
  fi
  # Run a priority backup
  restic backup $BACKUP_SRC --json | jq -C . || true
fi

sudo install -m 0750 /usr/local/sbin/smart_guard.sh /usr/local/sbin/smart_guard.sh
echo '15 * * * * root /usr/local/sbin/smart_guard.sh >>/var/log/ai-dr.log 2>&1' | sudo tee -a /etc/cron.d/ai-dr >/dev/null

Real-world-style walkthrough

  • 02:08 — Isolation Forest detects a spike in I/O errors and OOM kills against your normal 5-minute baseline.

  • The ai-dr.sh wrapper sends an alert, takes a fresh restic backup, runs restic check, and performs a targeted restore of /etc/hosts and /etc/fstab into a temp dir to prove the last snapshot is usable.

  • SMART guard at 03:00 flags Current_Pending_Sector > 0 on /dev/sdb, triggers another backup, and includes a clear report in the alert email.

  • Your on-call wakes to not only an alert but tangible proof that the restore path is alive. A disk swap is scheduled before business hours.

That is AI delivering clarity and buying you time.

Hardening tips

  • Protect secrets: keep RESTIC_PASSWORD_FILE mode 600, avoid exporting plain passwords in profiles.

  • Separate repos for hot and cold backups; store one offsite or immutable (e.g., object storage with write-once).

  • Prune and check regularly:

    restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check --read-data-subset=1/10

  • Test restores of representative data weekly. Automate AND randomly sample.

Conclusion and next steps (CTA)

You now have a lean, Linux-first disaster recovery loop:

  • Bash harvests signals.

  • A tiny AI model flags “weird.”

  • Backups are verified with real restore drills.

  • SMART risks accelerate your protection.

Next steps:

  • Expand feature extraction (network errors, service restarts, application logs).

  • Add host tags and centralize scores to prioritize fleet-wide drills.

  • Include a “full restore rehearsal” in a sandbox VM on a cadence.

  • If you use borg, mirror the verify/restore steps there as well.

Run a controlled DR drill this week. Time it. Document it. Then let this AI-assisted loop keep watch while you sleep.

Related posts