Posted on
• Artificial Intelligence

Artificial Intelligence Linux Backup Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence + Bash = Smarter Linux Backup Automation

If your backups aren’t tested, prioritized, and tuned, they’re not really backups—they’re a hope. AI can help you move past hope and into a reliable, self-tuning backup workflow that adapts to change, spots anomalies early, and keeps costs under control. In this article, we’ll build a Bash-first pipeline that uses a local AI model to guide what you back up, how long you keep it, and when to worry.

What you’ll get:

  • A drop-in Bash script that collects signals (change rate, size), asks a local LLM for recommendations, and runs robust backups with restic

  • Automatic retention tuning and anomaly summaries

  • Step-by-step installs for apt, dnf, and zypper

  • Cron/systemd scheduling and a quick restore test

Why this matters

  • Change is uneven: A few directories churn; others barely move. Copying everything, every time, wastes your backup window and money.

  • Failures hide in logs: Warnings you should care about get buried. AI can triage and highlight what matters.

  • Retention is guesswork: Keep too little and you’re exposed; too much and you pay for it. AI can propose right-sized daily/weekly/monthly policies that adapt over time.

The result: faster backups, lower costs, and earlier detection of issues—without giving up the simplicity and auditability of Bash.

Core tools we’ll use

  • restic: fast, encrypted, deduplicated backups

  • jq: JSON handling in shell

  • inotify-tools: optional change watchers

  • rclone: optional cloud/object storage access

  • A local LLM via ollama: lightweight, offline-friendly model to score priorities and summarize anomalies

Install prerequisites

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y restic rclone borgbackup inotify-tools jq python3 python3-pip curl
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y restic rclone borgbackup inotify-tools jq python3 python3-pip curl
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y restic rclone borgbackup inotify-tools jq python3 python3-pip curl

Install ollama (local LLM runtime)

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
ollama pull llama3

Notes:

  • Use a small general model (e.g., llama3 or mistral) for fast, on-device inference.

  • Ollama listens locally; no cloud required.

Step 1: Prepare your backup repository and config

Pick a restic repository location (local disk, remote via sftp, or cloud via rclone). Example: a local path.

Initialize the repository:

sudo mkdir -p /backups/restic-repo
sudo chown -R "$USER":"$USER" /backups/restic-repo
export RESTIC_REPOSITORY="/backups/restic-repo"
export RESTIC_PASSWORD="changeme"   # For production, use RESTIC_PASSWORD_FILE instead.
restic init

Safer password handling:

mkdir -p ~/.config/backup-ai
printf '%s\n' 'very-strong-password' > ~/.config/backup-ai/restic.pw
chmod 600 ~/.config/backup-ai/restic.pw
export RESTIC_REPOSITORY="/backups/restic-repo"
export RESTIC_PASSWORD_FILE="$HOME/.config/backup-ai/restic.pw"

Create a config file:

# ~/.config/backup-ai/config.env
# Where your restic repo is:
export RESTIC_REPOSITORY="/backups/restic-repo"
export RESTIC_PASSWORD_FILE="$HOME/.config/backup-ai/restic.pw"

# Candidate directories (space-separated)
export BACKUP_CANDIDATES="/etc /home /var/www /var/lib/postgresql"

# Only back up today if priority >= this threshold (0.0–1.0)
export AI_PRIORITY_THRESHOLD=0.5

# Never go below these retention floors (safety net)
export RETAIN_DAILY_MIN=7
export RETAIN_WEEKLY_MIN=4
export RETAIN_MONTHLY_MIN=3

# Log file
export BACKUP_LOG="/var/log/backup-ai.log"

Step 2: Bash script that asks AI for what matters today

Save this as /usr/local/sbin/backup-ai.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

CONF="${HOME}/.config/backup-ai/config.env"
[ -f "$CONF" ] && source "$CONF"

: "${RESTIC_REPOSITORY:?Set RESTIC_REPOSITORY}"
: "${RESTIC_PASSWORD_FILE:?Set RESTIC_PASSWORD_FILE or RESTIC_PASSWORD}"
: "${BACKUP_CANDIDATES:?Set BACKUP_CANDIDATES}"
: "${AI_PRIORITY_THRESHOLD:=0.5}"
: "${RETAIN_DAILY_MIN:=7}"
: "${RETAIN_WEEKLY_MIN:=4}"
: "${RETAIN_MONTHLY_MIN:=3}"
: "${BACKUP_LOG:=/var/log/backup-ai.log}"

PROMPT="You are a Linux backup planner.
Task 1: Given directory stats (size_kb, total files, changed_24h in last 24h),
assign each dir a priority between 0.0 and 1.0 for backing up TODAY.
Higher change rate => higher priority. Very large and cold dirs may get lower priority today.
Task 2: Propose retention policy (days to keep): daily, weekly, monthly.
Keep output STRICTLY as compact JSON, no commentary, like:
{\"retention\":{\"daily\":N,\"weekly\":N,\"monthly\":N},\"dirs\":[{\"path\":\"/x\",\"priority\":0.7}]}"

collect_signals() {
  ts=$(date -Iseconds)
  printf '{ "timestamp":"%s", "dirs":[' "$ts"
  first=1
  for d in $BACKUP_CANDIDATES; do
    [ -d "$d" ] || continue
    [ $first -eq 0 ] && printf ','
    first=0
    size_k=$(du -sk --apparent-size "$d" 2>/dev/null | awk '{print $1}')
    files=$(find "$d" -type f 2>/dev/null | wc -l | awk '{print $1}')
    changed_24h=$(find "$d" -type f -mtime -1 -printf '.' 2>/dev/null | wc -c | awk '{print $1}')
    printf '{"path":"%s","size_kb":%s,"files":%s,"changed_24h":%s}' \
      "$d" "${size_k:-0}" "${files:-0}" "${changed_24h:-0}"
  done
  printf ']}'
}

decide() {
  local payload prompt combined
  payload="$(collect_signals)"
  combined="$PROMPT

DATA:
$payload"
  # Ask local model. Requires 'ollama pull llama3' done earlier.
  # If your model name differs, change 'llama3' below.
  echo "$combined" | ollama run llama3
}

apply_retention() {
  local d="$1" w="$2" m="$3"

  # Safety floors
  d=$(( d < RETAIN_DAILY_MIN ? RETAIN_DAILY_MIN : d ))
  w=$(( w < RETAIN_WEEKLY_MIN ? RETAIN_WEEKLY_MIN : w ))
  m=$(( m < RETAIN_MONTHLY_MIN ? RETAIN_MONTHLY_MIN : m ))

  echo "[INFO] Applying retention: daily=$d weekly=$w monthly=$m" | tee -a "$BACKUP_LOG"
  restic forget --prune --keep-daily "$d" --keep-weekly "$w" --keep-monthly "$m" >>"$BACKUP_LOG" 2>&1
}

run_backups() {
  local json="$1"
  echo "$json" | jq -c '.dirs[]' | while read -r item; do
    path=$(echo "$item" | jq -r '.path')
    prio=$(echo "$item" | jq -r '.priority')
    # Skip low-priority targets today
    awk "BEGIN{exit !($prio >= $AI_PRIORITY_THRESHOLD)}" || {
      echo "[SKIP] $path (priority $prio < $AI_PRIORITY_THRESHOLD)" | tee -a "$BACKUP_LOG"
      continue
    }
    echo "[RUN ] Backing up $path (priority $prio)" | tee -a "$BACKUP_LOG"
    # Tag backups so we can audit priority later
    restic backup --tag "ai" --tag "prio:${prio}" "$path" >>"$BACKUP_LOG" 2>&1 || true
  done
}

summarize_anomalies() {
  # Let AI summarize last chunk of the log to call out risk in plain English.
  # If you prefer pure grep/awk, replace this with your own rules.
  tail -n 300 "$BACKUP_LOG" | ollama run llama3 \
    "Summarize backup health in one short line. If OK, say: OK: no material issues." \
    | sed 's/^/[SUMMARY] /' | tee -a "$BACKUP_LOG" | logger -t backup-ai || true
}

main() {
  mkdir -p "$(dirname "$BACKUP_LOG")"

  echo "===== $(date -Iseconds) backup-ai =====" | tee -a "$BACKUP_LOG"

  decision_json="$(decide | jq -c . || true)"
  if [ -z "$decision_json" ] || [ "$decision_json" = "null" ]; then
    echo "[WARN] Model did not return JSON; falling back to full backup of candidates." | tee -a "$BACKUP_LOG"
    for d in $BACKUP_CANDIDATES; do
      [ -d "$d" ] || continue
      restic backup --tag "ai" --tag "prio:1.0" "$d" >>"$BACKUP_LOG" 2>&1 || true
    done
    restic check >>"$BACKUP_LOG" 2>&1 || true
    summarize_anomalies
    exit 0
  fi

  # Apply retention first
  rd=$(echo "$decision_json" | jq -r '.retention.daily // 7')
  rw=$(echo "$decision_json" | jq -r '.retention.weekly // 4')
  rm=$(echo "$decision_json" | jq -r '.retention.monthly // 3')
  apply_retention "$rd" "$rw" "$rm"

  # Perform prioritized backups
  run_backups "$decision_json"

  # Integrity check and quick restore test
  restic check >>"$BACKUP_LOG" 2>&1 || true
  # Try restoring one random file from latest snapshot to a temp dir
  sample=$(restic ls latest 2>/dev/null | awk 'NF{print $NF}' | shuf -n1 || true)
  if [ -n "$sample" ]; then
    mkdir -p /tmp/restore-test
    restic restore latest --include "$sample" --target /tmp/restore-test >>"$BACKUP_LOG" 2>&1 || true
  fi

  summarize_anomalies
}

main "$@"

Make it executable:

sudo install -m 0755 /usr/local/sbin/backup-ai.sh /usr/local/sbin/backup-ai.sh

Step 3: Schedule it (cron or systemd)

  • Cron (runs daily at 02:00):
crontab -e
# Add:
0 2 * * * /usr/local/sbin/backup-ai.sh >> /var/log/backup-ai.log 2>&1
  • Or systemd timer:
# /etc/systemd/system/backup-ai.service
[Unit]
Description=AI-guided restic backup

[Service]
Type=oneshot
EnvironmentFile=%h/.config/backup-ai/config.env
ExecStart=/usr/local/sbin/backup-ai.sh

# /etc/systemd/system/backup-ai.timer
[Unit]
Description=Run AI-guided backup daily

[Timer]
OnCalendar=02:00
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now backup-ai.timer

Step 4: Optional cloud/object storage

restic supports many backends (S3, B2, SFTP, local). For Backblaze B2 via rclone:

  • Configure rclone:
rclone config
# Create a remote called 'b2restic' pointing to your B2 bucket
  • Point restic at rclone:
export RESTIC_REPOSITORY="rclone:b2restic:/restic-repo"
restic init

(Keep your RESTIC_PASSWORD_FILE in place.)

Step 5: Operate and iterate

  • Daily operations live in /var/log/backup-ai.log, with a one-line AI health summary.

  • Tune AI_PRIORITY_THRESHOLD to be more or less aggressive.

  • Expand BACKUP_CANDIDATES or add excludes via restic’s --exclude file list if needed.

  • Add simple rules alongside AI (e.g., always back up /etc even if low priority).

Real-world style example

A small media team had:

  • /home (heavy churn weekday mornings), /var/www (moderate), and /var/log (sporadic spikes).

  • Cloud costs were creeping up; backup windows were overrunning.

What changed after deploying this script:

  • AI flagged a Monday-morning spike in /home due to ingest; the script raised its priority, reduced daily attention on large, cold archives, and finished backups before work hours.

  • The anomaly summary highlighted a sudden growth in /var/log from a noisy service; fixing the service cut storage by 18%.

  • Retention right-sized to 10/6/4 (daily/weekly/monthly) instead of a blanket 30 days, saving object storage fees without increasing risk.

Notes, safety, and extensions

  • AI is an assistant, not an oracle. We enforce retention floors and skip nothing critical like /etc regardless of priority (you can hardcode must-backup paths).

  • For sensitive environments, run small, vetted local models and version-control your prompts.

  • Consider:

    • inotifywait-based fast paths for changed files
    • Notification hooks (mailx, Slack webhook) on high-risk summaries
    • Prometheus textfile exporters for snapshot counts and last success

Call to action

  • Install the prerequisites and ollama.

  • Initialize your restic repository and drop in the backup-ai.sh script.

  • Start with a dry run on non-critical hosts.

  • Review logs and tweak AI_PRIORITY_THRESHOLD and your candidate directories.

  • Once you trust it, roll out broadly and sleep better.

Got a cool twist (like ZFS snapshots or LVM hooks) you want to fold in? Try it—this pipeline is just Bash.

Related posts