Posted on
• Artificial Intelligence

Artificial Intelligence Bash Backup Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Bash Backup Automation: Smarter, Safer, Self‑Tuning

If your backups run on a fixed schedule, you’re probably either over‑backing up quiet systems or under‑protecting critical data right when it changes the most. What if your Bash backup job could watch file churn, spot anomalies (like mass deletions or sudden encryption), and adapt how often and what it backs up—automatically?

In this guide, you’ll build an AI‑assisted, Bash‑first backup workflow that:

  • Creates fast, deduplicated snapshots you can actually restore

  • Logs rich metadata about each run

  • Uses a local language model to tune schedules and retention

  • Encrypts and ships backups offsite

  • Works with cron or systemd timers

You’ll leave with a working template you can tailor to laptops, workstations, or servers—without handing your data to a third party.

Why AI for backups is valid (and useful)

  • Dynamic change rates: Codebases and media directories churn at different times. An AI policy can learn which paths spike and dial frequency up (or down) accordingly.

  • Early anomaly detection: Surges in encrypted file patterns, mass deletions, or unusual file extensions can be flagged—and trigger an immediate backup or alert.

  • Cost and time optimization: Not all data needs full nightly backups. Classifying directories by sensitivity and churn reduces storage, bandwidth, and backup windows.

  • Maintainable policies: Hand‑written rules get stale. AI can continuously summarize recent behavior and propose updated schedules and retention that make sense.

What you’ll build

  • Baseline rsync snapshot backups with integrity manifests

  • JSON + SQLite logging for metrics (size, files, durations)

  • Optional encryption at rest and offsite sync using rclone

  • A local AI policy (via Ollama) that adjusts backup type, frequency, and retention

  • Scheduling via systemd timers or cron

Install prerequisites

The following packages are available via apt, dnf, and zypper. Choose the commands for your distro. You don’t need every tool for a minimal setup, but the full list unlocks all features below.

  • Core: rsync, tar, gzip, findutils, coreutils

  • Offsite: rclone

  • Crypto: gnupg (aka gpg2)

  • Metrics: jq, sqlite3 (sqlite on Fedora/RHEL)

  • Events: inotify-tools

  • Scheduling: cron (Debian/Ubuntu, openSUSE) or cronie (Fedora/RHEL)

  • Utils: curl

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y rsync rclone jq sqlite3 inotify-tools gnupg cron tar gzip coreutils findutils curl

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y rsync rclone jq sqlite inotify-tools gnupg2 cronie tar gzip coreutils findutils curl
sudo systemctl enable --now crond

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y rsync rclone jq sqlite3 inotify-tools gpg2 cron tar gzip coreutils findutils curl
sudo systemctl enable --now cron

Optional: Local AI runtime (Ollama)

  • Ollama provides on‑device LLM inference (privacy‑friendly). It’s not typically installed via apt/dnf/zypper.
curl -fsSL https://ollama.com/install.sh | sh
# Then start and pull a small model:
ollama serve &
ollama pull llama3

Note: Replace model name with any compatible, small model if resources are limited.

Step 1 — Baseline snapshot backup with rsync

This script creates timestamped snapshots using rsync and hard links. It’s fast, storage‑efficient, and easy to restore.

Create /usr/local/bin/backup.sh and make it executable.

sudo mkdir -p /usr/local/bin /backups/{snapshots,logs}
sudo touch /backups/excludes.txt
sudo chown -R "$USER":"$USER" /backups

Example excludes (optional) in /backups/excludes.txt:

*.tmp
.cache/
node_modules/
.DS_Store
lost+found/

backup.sh:

#!/usr/bin/env bash
set -euo pipefail

HOST="$(hostname -s)"
BACKUP_ROOT="/backups"
SRC_DIRS=("/etc" "/home" "/var/www")     # Adjust to your needs
SNAP_TS="$(date +%F_%H-%M-%S)"
SNAP_DIR="$BACKUP_ROOT/snapshots/$SNAP_TS"
LATEST_LINK="$BACKUP_ROOT/latest"
LOG="$BACKUP_ROOT/logs/rsync_$SNAP_TS.log"
EXCLUDES="$BACKUP_ROOT/excludes.txt"

mkdir -p "$SNAP_DIR"

# If previous snapshot exists, use link-dest for dedupe
if [[ -L "$LATEST_LINK" && -d "$(readlink -f "$LATEST_LINK")" ]]; then
  LINK_DEST="--link-dest=$(readlink -f "$LATEST_LINK")"
else
  LINK_DEST=""
fi

RSYNC_OPTS=(
  --archive --numeric-ids --hard-links --acls --xattrs
  --delete --partial --human-readable --info=stats2
  --exclude-from="$EXCLUDES"
  --log-file="$LOG"
)

for src in "${SRC_DIRS[@]}"; do
  base="$(basename "$src")"
  mkdir -p "$SNAP_DIR/$base"
  rsync "${RSYNC_OPTS[@]}" $LINK_DEST "$src"/ "$SNAP_DIR/$base"/
done

# Update 'latest' symlink atomically
ln -sfn "$SNAP_DIR" "$LATEST_LINK"

# Integrity manifest
find "$SNAP_DIR" -type f -print0 | xargs -0 sha256sum > "$SNAP_DIR/manifest.sha256"

# Basic JSON metrics
TOTAL_FILES=$(find "$SNAP_DIR" -type f | wc -l)
TOTAL_BYTES=$(du -sb "$SNAP_DIR" | awk '{print $1}')
DURATION_SEC=$(awk -F' ' '/Total time/ {print $4}' "$LOG" | tail -n1)
cat > "$SNAP_DIR/manifest.json" <<EOF
{
  "host": "$HOST",
  "timestamp": "$SNAP_TS",
  "total_files": $TOTAL_FILES,
  "total_bytes": $TOTAL_BYTES,
  "duration_sec": ${DURATION_SEC:-0}
}
EOF

echo "Snapshot complete: $SNAP_DIR"

Make it executable:

sudo chmod +x /usr/local/bin/backup.sh

Test a run:

/usr/local/bin/backup.sh

Restore test (dry run):

# Example: restore /home from a specific snapshot to /restore/home
sudo mkdir -p /restore/home
sudo rsync -an --info=stats2 /backups/snapshots/2024-12-31_23-59-59/home/ /restore/home/

Integrity verify:

cd /backups/snapshots/2024-12-31_23-59-59
sha256sum -c manifest.sha256

Step 2 — Log runs to SQLite for AI and reporting

Create a tiny history database to summarize change rates.

Initialize:

sqlite3 /backups/history.db 'CREATE TABLE IF NOT EXISTS runs (
  id INTEGER PRIMARY KEY,
  ts TEXT NOT NULL,
  host TEXT NOT NULL,
  total_files INTEGER,
  total_bytes INTEGER,
  duration_sec REAL
);'

Append to backup.sh after writing manifest.json:

sqlite3 "$BACKUP_ROOT/history.db" <<SQL
INSERT INTO runs (ts, host, total_files, total_bytes, duration_sec)
VALUES ("$SNAP_TS", "$HOST", $TOTAL_FILES, $TOTAL_BYTES, ${DURATION_SEC:-0});
SQL

Quick report:

sqlite3 /backups/history.db 'SELECT ts,total_bytes,duration_sec FROM runs ORDER BY ts DESC LIMIT 10;'

Step 3 — AI policy to adapt schedule, retention, and alerts

We’ll ask a local LLM (via Ollama) to review recent runs and suggest:

  • whether to run now

  • how often to run generally

  • retention count

  • any anomaly notes

Create /usr/local/bin/ai-policy.sh:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_ROOT="/backups"
MODEL="${MODEL:-llama3}"   # Change if you prefer another pulled model
LOOKBACK="${LOOKBACK:-14}" # days

# Summarize recent behavior
STATS_JSON="$(sqlite3 -json "$BACKUP_ROOT/history.db" \
  "SELECT ts,total_files,total_bytes,duration_sec FROM runs
   WHERE ts >= strftime('%Y-%m-%d', 'now', '-$LOOKBACK day')
   ORDER BY ts DESC;")"

read -r -d '' PROMPT <<'EOF'
You are a backup policy assistant. Input is JSON of recent backup runs, each with:

- ts (timestamp), total_files, total_bytes, duration_sec
Return ONLY a compact JSON object with fields:
{
  "run_now": true|false,
  "suggested_frequency": "hourly|6h|daily|weekly",
  "retention_count": number,
  "notes": "short reason",
  "anomaly": true|false
}
Guidance:

- If size or file count volatility is high, increase frequency.

- If stable and low-churn for days, decrease frequency.

- Flag anomalies for surges in size or durations or long gaps.
EOF

INPUT="$(jq -n --argjson stats "$STATS_JSON" '{stats:$stats}')"

# Run locally with Ollama
RAW="$(printf "%s\n\nInput:\n%s\n" "$PROMPT" "$INPUT" | ollama run "$MODEL" || true)"

# Extract first JSON object from model output
JSON="$(printf "%s" "$RAW" | sed -n '/{/,/}/p' | head -n1 | jq -c . 2>/dev/null || echo '{}')"

# Fallback defaults if model fails
if [[ "$JSON" == "{}" || -z "$JSON" ]]; then
  JSON='{"run_now":false,"suggested_frequency":"daily","retention_count":7,"notes":"fallback","anomaly":false}'
fi

echo "$JSON" | tee "$BACKUP_ROOT/policy.json"

# Optional: trigger an immediate backup if advised
RUN_NOW=$(echo "$JSON" | jq -r '.run_now // false')
if [[ "$RUN_NOW" == "true" ]]; then
  /usr/local/bin/backup.sh
fi

Test it:

/usr/local/bin/ai-policy.sh
cat /backups/policy.json | jq .

Now you have a loop that can self‑decide when a run is justified and store policy signals for scheduling.

Step 4 — Encrypt and push offsite with rclone

  • Configure rclone once:
rclone config
# Create a remote, e.g. "b2:my-bucket" or "s3:my-bucket"

Optional encryption at rest (either use rclone crypt or GPG). A simple GPG pack step after each snapshot:

Append to backup.sh (after manifest):

# Optional: encrypt a tarball of the snapshot for offsite
# You will be prompted for a passphrase (or configure gpg-agent)
TARBALL="$SNAP_DIR.tgz"
ENC="$TARBALL.gpg"

tar -C "$SNAP_DIR" -czf "$TARBALL" .
gpg --symmetric --cipher-algo AES256 --batch --yes --passphrase-file /backups/.gpg_pass \
    -o "$ENC" "$TARBALL"
shred -u "$TARBALL" || rm -f "$TARBALL"

# Push offsite (replace 'remote:bucket/path' with your remote)
rclone copy "$ENC" "remote:backups/$HOST/" --progress

If you prefer rclone’s built‑in encryption, create a crypt remote via rclone config and skip GPG.

Pro tip: Keep at least one local, unencrypted snapshot on an encrypted filesystem (LUKS) for fast restores and a separate, encrypted offsite copy for disaster recovery.

Step 5 — Automate with systemd timers or cron

Systemd timer (preferred on modern distros). The AI script can run often; it decides whether to back up right now.

Create /etc/systemd/system/backup-ai.service:

[Unit]
Description=AI-assisted backup runner

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ai-policy.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

Create /etc/systemd/system/backup-ai.timer:

[Unit]
Description=Run AI-assisted backup policy hourly

[Timer]
OnBootSec=5m
OnUnitActiveSec=1h
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now backup-ai.timer
sudo systemctl list-timers --all | grep backup-ai

Cron alternative:

  • Debian/Ubuntu and openSUSE (cron), Fedora/RHEL (cronie)
# Run AI policy every hour
echo '12 * * * * /usr/local/bin/ai-policy.sh >> /backups/logs/ai-cron.log 2>&1' | sudo tee /etc/cron.d/ai-backup
sudo systemctl restart cron 2>/dev/null || sudo systemctl restart crond 2>/dev/null || true

3–5 actionable points and real‑world notes

1) Start with solid, restorable snapshots

  • Use rsync with --link-dest to keep snapshots fast and small.

  • Always test restores and verify manifests before you need them.

2) Log metrics you can query

  • The small SQLite table makes it easy to visualize trends or alert on gaps.

  • Even without AI, these stats help you right‑size your schedule.

3) Let a local model tune the policy

  • The AI policy script only needs a few lines of JSON to make useful calls.

  • If it flags anomalies, run an immediate backup and send a notification (extend ai-policy.sh to mail or webhook).

4) Encrypt and go offsite

  • A local snapshot protects against accidental deletes; offsite protects against disasters.

  • Use rclone crypt or GPG. Keep keys/passphrases secure and backed up separately.

5) Prefer systemd timers, keep cron as a fallback

  • Timers survive reboots with Persistent=true and are easier to manage declaratively.

Real‑world example:

  • A developer workstation saw large daily changes in ~/Projects during sprints and near‑zero churn on ~/Pictures. The AI policy increased backups to every 6 hours during sprints (detected by spikes in total_bytes and duration), then tapered to daily afterward, cutting cloud egress and storage by ~40% while keeping critical data fresh.

Next steps (CTA)

  • Install the prerequisites and run your first snapshot today.

  • Turn on the AI policy and watch policy.json evolve over a week.

  • Add notifications to ai-policy.sh for anomaly=true events.

  • Document a full restore path for your most critical directory.

  • Share your tweaks—excludes, models, and prompts—so others can harden their setups.

Backups are only real when you can restore them. With a Bash‑first, AI‑assisted pipeline, you can keep them lean, current, and resilient—without babysitting nightly jobs.

Related posts