Posted on
• Artificial Intelligence

Artificial Intelligence for Linux Disaster Recovery Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence for Linux Disaster Recovery Automation

At 2:07 AM your pager explodes. Disk errors flood the logs, an app stops responding, and backups may or may not be current. In that moment, you don’t want to read a thousand lines of syslog—you want triage, answers, and a safe recovery path. This post shows how to bring practical AI into your Linux disaster recovery (DR) workflow to reduce MTTD/MTTR, turn noisy logs into signal, and automate the first, safest steps of recovery with guardrails.

We’ll cover why AI is a fit for DR, then give you ready-to-run scripts to:

  • Automate encrypted, verifiable backups

  • Detect anomalies in logs with a lightweight on-host model

  • Map anomalies to runbooks and fire controlled remediation

  • Drill and verify restores on a schedule

All examples are Bash-first with small, auditable Python helpers.

Why AI belongs in your DR toolbox

  • Volume and velocity: Modern systems emit far more logs/metrics than any human can parse in an outage. Anomaly detection prioritizes what changed, when, and where.

  • Patterns over rules: Failures rhyme, but don’t repeat exactly. Unsupervised models like Isolation Forest can flag “this looks new and bad” without a brittle ruleset.

  • Runbook acceleration: AI can classify symptoms and suggest the right recovery playbook faster than manual triage, while you keep humans in the approval loop.

  • Outcome focus: Faster detection and guided recovery reduce RTO/RPO while preserving consistency and audit trails.

Prerequisites and packages

Install these utilities on your DR host(s): restic (encrypted backups), jq (JSON parsing), Python 3 + venv/pip (for the lightweight AI), rsync, and S.M.A.R.T. tools.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y restic jq python3 python3-venv python3-pip smartmontools rsync cron
sudo systemctl enable --now cron

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y restic jq python3 python3-pip python3-virtualenv smartmontools rsync cronie
sudo systemctl enable --now crond

Note: On RHEL derivatives, you may need to enable EPEL to get restic.

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y restic jq python3 python3-pip python3-virtualenv smartmontools rsync cron
sudo systemctl enable --now cron

Create an isolated Python environment for the AI helper:

python3 -m venv /opt/ai-dr/venv
/opt/ai-dr/venv/bin/pip install --upgrade pip
/opt/ai-dr/venv/bin/pip install scikit-learn

1) Automate encrypted, verifiable backups with restic

Restic is fast, deduplicated, and encrypted by default. Configure it once and schedule it with systemd timers.

Create a restic environment file:

sudo install -d -m 750 /etc/restic
sudo tee /etc/restic/backup.env >/dev/null <<'EOF'
RESTIC_REPOSITORY=/var/backups/restic-repo
# Prefer using a password file with strict perms in production:
# RESTIC_PASSWORD_FILE=/etc/restic/secret
RESTIC_PASSWORD=ChangeThisStrongPassphrase
BACKUP_TAG=prod-$(hostname)
BACKUP_PATHS="/etc /var/lib /home"
EXCLUDES="--exclude-caches --exclude=/var/lib/docker/overlay2 --exclude=/var/tmp"
EOF
sudo chmod 640 /etc/restic/backup.env

Initialize the repository (local path example; swap for s3: or rest: URLs as needed):

sudo bash -c 'source /etc/restic/backup.env && restic init'

Backup script:

sudo tee /usr/local/bin/restic-backup.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source /etc/restic/backup.env
export RESTIC_REPOSITORY RESTIC_PASSWORD RESTIC_PASSWORD_FILE

restic backup \
  $EXCLUDES \
  --tag "$BACKUP_TAG" \
  --one-file-system \
  $BACKUP_PATHS

# Prune/forget policy: keep space sane
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
# Integrity check (lightweight)
restic check --read-data-subset=1/20
EOF
sudo chmod +x /usr/local/bin/restic-backup.sh

Systemd units:

sudo tee /etc/systemd/system/restic-backup.service >/dev/null <<'EOF'
[Unit]
Description=Restic Backup

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/backup.env
ExecStart=/usr/local/bin/restic-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
EOF

sudo tee /etc/systemd/system/restic-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Run Restic Backup hourly

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer

Test a restore dry-run:

# List snapshots
sudo bash -c 'source /etc/restic/backup.env && restic snapshots'

# Restore to a safe temp dir (example)
sudo mkdir -p /tmp/restore-test
sudo bash -c 'source /etc/restic/backup.env && restic restore latest --target /tmp/restore-test --include /etc/hosts'

2) Detect anomalies in logs with a tiny on-host model

We’ll stream logs, build a baseline on the fly, and flag anomalies using scikit-learn’s IsolationForest. No internet or heavyweight dependencies required.

Python anomaly watcher:

sudo tee /opt/ai-dr/ai_log_watch.py >/dev/null <<'EOF'
#!/usr/bin/env python3
import sys, json, time
from sklearn.ensemble import IsolationForest
from sklearn.feature_extraction.text import HashingVectorizer

# Parameters
BASELINE_LINES = 1000
N_FEATURES = 256
SEED = 42
CONTAMINATION = 0.01  # expected anomaly rate

vec = HashingVectorizer(n_features=N_FEATURES, alternate_sign=False, norm=None)
baseline_vecs = []
model = None

def now():
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())

for line in sys.stdin:
    line = line.rstrip("\n")
    if not line:
        continue

    X = vec.transform([line]).toarray()  # dense array for tree model

    if model is None:
        baseline_vecs.append(X[0])
        if len(baseline_vecs) >= BASELINE_LINES:
            model = IsolationForest(
                n_estimators=200,
                contamination=CONTAMINATION,
                random_state=SEED,
                n_jobs=1
            ).fit(baseline_vecs)
            # Emit a startup note
            sys.stderr.write(f"[ai-log] baseline built with {BASELINE_LINES} lines\n")
            sys.stderr.flush()
        continue

    score = -float(model.score_samples(X)[0])  # higher => more anomalous
    # Simple threshold: top ~1% anomalies (approx by contamination)
    is_anom = model.predict(X)[0] == -1

    event = {
        "ts": now(),
        "score": round(score, 6),
        "anomaly": bool(is_anom),
        "msg": line
    }
    print(json.dumps(event), flush=True)
EOF
sudo chmod +x /opt/ai-dr/ai_log_watch.py

Start streaming from journald or syslog:

# Systemd journals (kernel + services), minimal formatting
sudo journalctl -f -n 0 -o cat | /opt/ai-dr/venv/bin/python /opt/ai-dr/ai_log_watch.py | sudo tee -a /var/log/dr-ai.jsonl

Or for classic syslog:

sudo tail -Fn0 /var/log/syslog | /opt/ai-dr/venv/bin/python /opt/ai-dr/ai_log_watch.py | sudo tee -a /var/log/dr-ai.jsonl

Tip: run it as a service for persistence:

sudo tee /etc/systemd/system/ai-log-watch.service >/dev/null <<'EOF'
[Unit]
Description=AI Log Anomaly Watcher
After=network.target

[Service]
Type=simple
User=root
ExecStart=/bin/sh -c 'journalctl -f -n 0 -o cat | /opt/ai-dr/venv/bin/python /opt/ai-dr/ai_log_watch.py >> /var/log/dr-ai.jsonl'
Restart=always
RestartSec=2s

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ai-log-watch.service

3) Map anomalies to runbooks and automate safely

We’ll keep mappings in JSON and implement guarded actions in Bash. Only act on anomalies that match known patterns, and default to “notify + wait for approval.”

Runbook mappings:

sudo tee /etc/ai-dr/runbook.json >/dev/null <<'EOF'
[
  { "match": ["EXT4-fs error", "I/O error"], "action": "fsck_repair", "device": "/dev/sdb1" },
  { "match": ["Out of memory", "oom-killer"], "action": "restart_service", "service": "your-app.service" },
  { "match": ["connection refused", "database"], "action": "restart_service", "service": "postgresql" },
  { "match": ["Read-only file system"], "action": "remount_rw", "mount": "/" }
]
EOF

Action implementations:

sudo tee /usr/local/bin/dr-actions.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

approve() {
  # Require explicit approval unless DR_AUTO=1 is set (e.g., for staging)
  if [[ "${DR_AUTO:-0}" != "1" ]]; then
    echo "[dr] awaiting human approval (set DR_AUTO=1 to auto-approve)" >&2
    exit 2
  fi
}

case "${1:-}" in
  fsck_repair)
    approve
    dev="${2:-/dev/sdb1}"
    echo "[dr] attempting fsck (pre-flight: unmounted check) on $dev"
    # WARNING: Only safe if unmounted. Adjust for your environment.
    fsck -f -y "$dev"
    ;;
  restart_service)
    svc="${2:-}"
    [[ -n "$svc" ]] || { echo "[dr] missing service name"; exit 1; }
    echo "[dr] restarting $svc"
    systemctl restart "$svc"
    systemctl is-active --quiet "$svc" && echo "[dr] $svc is active" || { echo "[dr] $svc failed"; exit 1; }
    ;;
  remount_rw)
    approve
    mnt="${2:-/}"
    echo "[dr] remounting $mnt read-write"
    mount -o remount,rw "$mnt"
    ;;
  *)
    echo "[dr] unknown action: $1" >&2
    exit 1
    ;;
esac
EOF
sudo chmod +x /usr/local/bin/dr-actions.sh

Event handler (consume AI output, match, and act with guardrails):

sudo tee /usr/local/bin/dr-handle-events.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

RUNBOOK=/etc/ai-dr/runbook.json
THRESHOLD=0.5  # adjust based on your data
LOG_IN=/var/log/dr-ai.jsonl
LOG_OUT=/var/log/dr-actions.log

touch "$LOG_OUT"

tail -Fn0 "$LOG_IN" | while read -r line; do
  anom=$(echo "$line" | jq -r '.anomaly // false')
  score=$(echo "$line" | jq -r '.score // 0')
  msg=$(echo "$line" | jq -r '.msg // ""')

  [[ "$anom" == "true" && $(echo "$score>$THRESHOLD" | bc -l) -eq 1 ]] || continue

  echo "[dr] anomaly score=$score msg=$msg" | tee -a "$LOG_OUT"
  # Find first matching runbook entry
  idx=$(jq -r --arg m "$msg" '
    to_entries
    | map(select([.value.match[]?] | any(. as $k | ($m|test($k))))
    ) | .[0].key // -1' "$RUNBOOK")

  if [[ "$idx" -ge 0 ]]; then
    action=$(jq -r ".[$idx].action" "$RUNBOOK")
    param=$(jq -r ".[$idx] | if .action==\"restart_service\" then .service elif .action==\"fsck_repair\" then .device elif .action==\"remount_rw\" then .mount else empty end" "$RUNBOOK")
    echo "[dr] matched action=$action param=$param" | tee -a "$LOG_OUT"
    if /usr/local/bin/dr-actions.sh "$action" "$param"; then
      echo "[dr] action success" | tee -a "$LOG_OUT"
    else
      echo "[dr] action failed; escalate" | tee -a "$LOG_OUT"
    fi
  else
    echo "[dr] no runbook match; notify human" | tee -a "$LOG_OUT"
  fi
done
EOF
sudo chmod +x /usr/local/bin/dr-handle-events.sh

Run it (start in approval mode off; you can export DR_AUTO=1 for staging):

sudo env DR_AUTO=0 /usr/local/bin/dr-handle-events.sh

Notes:

  • Make fsck/repair actions environment-aware. Never fsck a mounted filesystem. Prefer boot-time checks, snapshots, or recovery instances.

  • For services, ensure unit files support graceful restart and health checks.

4) Drill: verify backups and the automation path

Regular, automated drills are where AI and DR become muscle memory.

Nightly restore verification (systemd):

sudo tee /usr/local/bin/restic-verify-restore.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source /etc/restic/backup.env
export RESTIC_REPOSITORY RESTIC_PASSWORD RESTIC_PASSWORD_FILE

TARGET=/tmp/restore-verify
rm -rf "$TARGET"
mkdir -p "$TARGET"

restic restore latest --target "$TARGET" --include /etc
# Simple integrity check on restored files
test -f "$TARGET/etc/hosts" && echo "[verify] restore looks sane"
# Cleanup
rm -rf "$TARGET"
EOF
sudo chmod +x /usr/local/bin/restic-verify-restore.sh

Timer:

sudo tee /etc/systemd/system/restic-verify.service >/dev/null <<'EOF'
[Unit]
Description=Restic Restore Verification

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/backup.env
ExecStart=/usr/local/bin/restic-verify-restore.sh
EOF

sudo tee /etc/systemd/system/restic-verify.timer >/dev/null <<'EOF'
[Unit]
Description=Nightly Restic Restore Verification

[Timer]
OnCalendar=03:15
Persistent=true
RandomizedDelaySec=600

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now restic-verify.timer

Optional: include S.M.A.R.T. signals in your pipeline to preempt disk-related incidents:

sudo smartctl -H /dev/sda || echo "[warn] S.M.A.R.T. reports issues on /dev/sda"

Real-world style example

  • Symptom: Kernel logs show intermittent I/O errors on /dev/sdb1.

  • The AI watcher flags the sudden burst as anomalous.

  • The handler matches “I/O error” and “EXT4-fs error” to a cautious runbook.

  • In production, DR_AUTO=0 causes it to log the suggested action and wait for approval (e.g., maintenance window).

  • You switch to a recovery instance, attach the volume, run fsck -y, and restore any damaged files from the latest restic snapshot.

  • Post-incident, you add those error patterns and the confirmed action to runbook.json for faster future response.

Sample anomaly line:

{"ts":"2026-07-07T01:14:08Z","score":0.812345,"anomaly":true,"msg":"kernel: EXT4-fs error (device sdb1): ext4_find_entry:1455: inode #123456: comm rsync: reading directory lblock 0"}

Practical tips and guardrails

  • Start narrow: one or two services, a small set of well-understood runbooks, and non-destructive actions first (restart, remount, failover).

  • Tune thresholds: Use the anomaly log to calibrate THRESHOLD and CONTAMINATION before enabling automation.

  • Separate staging vs production: Export DR_AUTO=1 in staging only; require human approval in production.

  • Audit everything: Keep dr-ai.jsonl and dr-actions.log for postmortems and compliance.

  • Keep backups independent: Store restic repositories off-host and off-zone where possible; verify restores frequently.

Conclusion and next steps

AI doesn’t replace your DR plan—it makes it faster, safer, and more consistent under pressure. With a few packages and small scripts, you can:

  • Back up and verify automatically

  • Convert log firehoses into prioritized signals

  • Trigger vetted runbooks with human-in-the-loop control

Your next step:

  • Deploy the backup + verify timers today

  • Turn on the AI log watcher in observe-only mode

  • Add two runbook mappings for your top failure modes

  • Schedule a quarterly recovery drill

When the next 2:07 AM page hits, you’ll have signal, suggestions, and a clear path to recovery.

Related posts