Posted on
• Artificial Intelligence

Artificial Intelligence Snapshot Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Snapshot Automation (with Bash): Smarter, Leaner, Safer Backups

Ever needed to roll back a bad update only to find your snapshot rotation pruned the one you needed? Or burned disk space on snapshots you never used? There’s a better way: let AI decide when to snapshot and what to keep.

In this post, you’ll wire up a small, auditable Bash tool that:

  • Collects simple signals (updates pending, change rate, disk free space)

  • Asks an AI policy (local or cloud) whether to take a snapshot

  • Creates Btrfs or LVM snapshots with metadata

  • Prunes old snapshots based on AI-recommended retention

You’ll get commands for apt, dnf, and zypper; a ready‑to‑drop script; and a systemd timer to automate it.

Why AI for snapshots?

  • Not all hours are equal. Change rates, patch days, and deploy windows vary. A static “snapshot every hour” wastes space and often misses the most critical events.

  • Context matters. AI can weigh multiple signals—pending updates, disk headroom, CPU load, recent file churn—and decide if a snapshot is warranted.

  • Better retention. Instead of blind FIFO deletion, you can keep snapshots that AI labels “important” for longer and prune the rest sooner.

The outcome: fewer snapshots, better coverage, and clearer labels for when it matters.

What we’ll build

  • A single Bash script ai-snapshot.sh that:

    • Supports Btrfs subvolume snapshots and LVM snapshots
    • Gathers metrics
    • Calls an AI policy (optional; falls back to heuristics)
    • Creates snapshots with human-friendly labels
    • Prunes by age (and can be extended to importance)

  • A systemd timer (or cron) for unattended runs.

Prerequisites and installation

Install baseline tools and at least one snapshot backend (Btrfs or LVM). Use your distro’s package manager:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y btrfs-progs lvm2 curl jq inotify-tools sysstat

  • Fedora/RHEL/CentOS/Alma/Rocky (dnf):

    sudo dnf install -y btrfs-progs lvm2 curl jq inotify-tools sysstat

  • openSUSE/SLE (zypper):

    sudo zypper refresh
sudo zypper install -y btrfsprogs lvm2 curl jq inotify-tools sysstat

Notes:

  • You only need one snapshot backend (Btrfs or LVM) for this tutorial. ZFS is possible but packaging varies by distro; not covered here.

  • For cron scheduling (optional), install a cron daemon if not present:

    • apt: sudo apt install -y cron
    • dnf: sudo dnf install -y cronie
    • zypper: sudo zypper install -y cron (or cronie on some variants)

Optional: local AI with Ollama (if you don’t want to use a cloud API)

curl -fsSL https://ollama.com/install.sh | sh
# Run once to pull a small model, e.g.:
ollama pull llama3.1:8b

The script: AI-driven snapshot decisions

Save the following as /usr/local/sbin/ai-snapshot.sh and make it executable:

sudo install -m 0755 /dev/stdin /usr/local/sbin/ai-snapshot.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# ai-snapshot.sh
# Create and prune snapshots based on simple system metrics + optional AI policy.
# Supports: BTRFS subvolume snapshots and LVM snapshots.
#
# Required privileges: root
#
# Configuration via environment variables:
#  BACKEND           : "btrfs" | "lvm" (required)
#  TARGET_PATH       : path to observe for changes (defaults to /)
#
#  BTRFS_SUBVOL      : absolute path of btrfs subvolume to snapshot (if BACKEND=btrfs)
#  BTRFS_SNAPDIR     : directory (on same btrfs FS) to store readonly snapshots
#
#  LVM_LV_PATH       : e.g. /dev/vgname/lvname (if BACKEND=lvm)
#  LVM_SNAP_SIZE     : e.g. 5G (snapshot COW size)
#
#  AI_PROVIDER       : "none" (default) | "ollama" | "openai"
#  AI_LABEL          : optional hint label for snapshots (e.g. "pre-update")
#
#  # Ollama:
#  OLLAMA_HOST       : default http://127.0.0.1:11434
#  OLLAMA_MODEL      : e.g. llama3.1:8b
#
#  # OpenAI:
#  OPENAI_API_KEY    : your API key
#  OPENAI_MODEL      : e.g. gpt-4o-mini (or any JSON-capable chat model)
#
#  RETENTION_DAYS_DEFAULT : fallback retention in days (default 14)
#
# Usage (examples):
#  sudo BACKEND=btrfs BTRFS_SUBVOL=/ BTRFS_SNAPDIR=/@snapshots /usr/local/sbin/ai-snapshot.sh
#  sudo BACKEND=lvm LVM_LV_PATH=/dev/vg0/root LVM_SNAP_SIZE=5G /usr/local/sbin/ai-snapshot.sh
#
# Safety: Test on non-production first.

umask 027

# Defaults
TARGET_PATH="${TARGET_PATH:-/}"
AI_PROVIDER="${AI_PROVIDER:-none}"
RETENTION_DAYS_DEFAULT="${RETENTION_DAYS_DEFAULT:-14}"

STATE_DIR="/var/lib/ai-snapshot"
META_DIR="$STATE_DIR/meta"
LOCKFILE="/run/lock/ai-snapshot.lock"

mkdir -p "$META_DIR"

exec 9>"$LOCKFILE" || true
if ! flock -n 9; then
  echo "[ai-snapshot] Another run is in progress; exiting."
  exit 0
fi

require_root() {
  if [[ "${EUID:-$UID}" -ne 0 ]]; then
    echo "This script must be run as root."
    exit 1
  fi
}

log() { printf "[%s] %s\n" "$(date +'%F %T')" "$*"; }

slugify() {
  tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+|-+$//g' | cut -c1-40
}

detect_distro() {
  if command -v apt-get >/dev/null 2>&1; then echo "apt"
  elif command -v dnf >/dev/null 2>&1; then echo "dnf"
  elif command -v zypper >/dev/null 2>&1; then echo "zypper"
  else echo "unknown"
  fi
}

count_pending_updates() {
  case "$(detect_distro)" in
    apt)
      # Count upgradable packages (simulation)
      apt-get -s upgrade 2>/dev/null | grep -cE '^Inst ' || true
      ;;
    dnf)
      # Return 0 if none, else count lines with "package.arch"
      dnf -q check-update 2>/dev/null | grep -E '^[[:alnum:]][^[:space:]]+\.[[:alnum:]_-]+' -c || true
      ;;
    zypper)
      # Skip headers, count remaining rows
      zypper -q lu 2>/dev/null | awk 'NR>2 && $1 != "" {print}' | wc -l
      ;;
    *)
      echo 0
      ;;
  esac
}

get_loadavg() {
  awk '{print $1}' /proc/loadavg
}

get_free_pct() {
  # 100 - Used% from df on TARGET_PATH mount
  df -P "$TARGET_PATH" | awk 'NR==2 {print 100 - $5}'
}

last_run_ts_file="$STATE_DIR/last_run.ts"
get_changed_files() {
  local since
  if [[ -f "$last_run_ts_file" ]]; then
    since="$(cat "$last_run_ts_file")"
  else
    since=$(( $(date +%s) - 24*3600 )) # default: last 24h
  fi
  # Approximate: files modified since "since"
  find "$TARGET_PATH" -xdev -type f -newermt "@$since" 2>/dev/null | wc -l
}

ai_decide() {
  # Inputs: metrics JSON on stdin.
  # Output: JSON: {"decision":true|false,"reason":"...","retention_days":N,"label":"..."}
  local metrics prompt body
  metrics="$(cat)"
  local base='{"decision":false,"reason":"heuristic fallback","retention_days":'"$RETENTION_DAYS_DEFAULT"',"label":""}'

  if [[ "$AI_PROVIDER" == "none" ]]; then
    # Heuristic: snapshot if updates pending OR many changes and enough free space
    local updates changed free load
    updates="$(jq -r '.updates' <<<"$metrics")"
    changed="$(jq -r '.changed_files' <<<"$metrics")"
    free="$(jq -r '.free_pct' <<<"$metrics")"
    load="$(jq -r '.loadavg' <<<"$metrics")"

    if { [[ "$updates" -ge 1 ]] || [[ "$changed" -ge 1000 ]]; } && (( $(printf '%.0f' "$free") >= 20 )); then
      jq -n --arg reason "heuristic: updates_or_churn_with_space" --arg label "${AI_LABEL:-}" \
         --argjson retention $RETENTION_DAYS_DEFAULT \
         '{decision:true, reason:$reason, retention_days:$retention, label:$label}'
    else
      jq -n --arg reason "heuristic: low_value_window" --arg label "${AI_LABEL:-}" \
         --argjson retention $RETENTION_DAYS_DEFAULT \
         '{decision:false, reason:$reason, retention_days:$retention, label:$label}'
    fi
    return
  fi

  prompt=$(cat <<'P'
You are a system reliability assistant. Decide if we should take a filesystem snapshot now.
Return strict JSON with keys: decision (boolean), reason (short string), retention_days (integer 1..60), label (short kebab-case string).
Prefer snapshots when updates are pending, file churn is high, or before typical maintenance windows. Be conservative if disk free < 15%.
P
)

  body=$(jq -n \
    --arg p "$prompt" \
    --argjson m "$metrics" \
    '{prompt:($p+"\nMetrics:\n"+($m|tojson)+"\nDecide now and return only JSON."), stream:false}')

  case "$AI_PROVIDER" in
    ollama)
      local host="${OLLAMA_HOST:-http://127.0.0.1:11434}"
      local model="${OLLAMA_MODEL:?Set OLLAMA_MODEL}"
      # Ollama generate returns {"response": "..."} when stream=false
      local resp
      resp=$(curl -sfS "$host/api/generate" \
        -H "Content-Type: application/json" \
        -d "$(jq -n --arg model "$model" --arg prompt "$(jq -r '.prompt' <<<"$body")" --argjson stream false '{model:$model, prompt:$prompt, stream:$stream}')" )
      # Extract JSON object from response
      local json
      json=$(jq -r '.response' <<<"$resp" | awk 'BEGIN{RS="}"} {print $0"}"}' | tail -n1)
      jq -er . >/dev/null 2>&1 <<<"$json" || json="$base"
      jq -n --argjson j "$json" '$j'
      ;;
    openai)
      local model="${OPENAI_MODEL:?Set OPENAI_MODEL}"
      : "${OPENAI_API_KEY:?Set OPENAI_API_KEY}"
      local resp json
      resp=$(curl -sfS https://api.openai.com/v1/chat/completions \
        -H "Authorization: Bearer $OPENAI_API_KEY" \
        -H "Content-Type: application/json" \
        -d "$(jq -n \
              --arg model "$model" \
              --arg sys "You output only JSON." \
              --arg user "$(jq -r '.prompt' <<<"$body")" \
              '{model:$model, response_format:{type:"json_object"}, messages:[{role:"system",content:$sys},{role:"user",content:$user}]}')" )
      json=$(jq -r '.choices[0].message.content' <<<"$resp")
      jq -er . >/dev/null 2>&1 <<<"$json" || json="$base"
      jq -n --argjson j "$json" '$j'
      ;;
    *)
      jq -n "$base"
      ;;
  esac
}

create_btrfs_snapshot() {
  local src="$1" destdir="$2" name="$3"
  [[ -d "$destdir" ]] || mkdir -p "$destdir"
  btrfs subvolume snapshot -r "$src" "$destdir/$name"
  echo "$destdir/$name"
}

create_lvm_snapshot() {
  local lv_path="$1" snap_size="$2" snap_name="$3"
  lvcreate -s -n "$snap_name" -L "$snap_size" "$lv_path" >/dev/null
  echo "/dev/$(lvs --noheadings -o vg_name "$lv_path" | awk '{$1=$1;print}')/$snap_name"
}

prune_btrfs() {
  local snapdir="$1" before_ts="$2"
  find "$snapdir" -mindepth 1 -maxdepth 1 -type d -printf "%f\n" | while read -r s; do
    local meta="$META_DIR/$s.json"
    if [[ -f "$meta" ]]; then
      local exp
      exp="$(jq -r '.expire_at' "$meta")"
      if [[ -n "$exp" && "$exp" -le "$before_ts" ]]; then
        log "Pruning Btrfs snapshot $s"
        btrfs subvolume delete "$snapdir/$s" || true
        rm -f "$meta"
      fi
    fi
  done
}

prune_lvm() {
  local lv_base="$1" before_ts="$2"
  # Expect snapshot names to start with "snap-"
  lvs --noheadings -o lv_name "$lv_base" | awk '{$1=$1;print}' | grep -E '^snap-' || true | while read -r s; do
    local meta="$META_DIR/$s.json"
    if [[ -f "$meta" ]]; then
      local exp
      exp="$(jq -r '.expire_at' "$meta")"
      if [[ -n "$exp" && "$exp" -le "$before_ts" ]]; then
        log "Pruning LVM snapshot $s"
        lvremove -f "$(dirname "$lv_base")/$s" || true
        rm -f "$meta"
      fi
    fi
  done
}

main() {
  require_root

  # Gather metrics
  local updates changed free load metrics now
  updates="$(count_pending_updates)"
  changed="$(get_changed_files)"
  free="$(get_free_pct)"
  load="$(get_loadavg)"
  now="$(date +%s)"
  printf "%s\n" "$now" > "$last_run_ts_file"

  metrics=$(jq -n --arg backend "${BACKEND:-}" \
                 --arg target "$TARGET_PATH" \
                 --arg label "${AI_LABEL:-}" \
                 --arg hostname "$(hostname -s)" \
                 --arg time "$(date -Is)" \
                 --argjson updates "$updates" \
                 --argjson changed "$changed" \
                 --argjson free "$free" \
                 --argjson load "$load" \
                 '{time:$time,host:$hostname,backend:$backend,target:$target,updates:$updates,changed_files:$changed,free_pct:$free,loadavg:$load,label:$label}')

  log "Metrics: $(jq -c . <<<"$metrics")"

  local decision_json
  decision_json="$(ai_decide <<<"$metrics")"
  log "AI decision: $(jq -c . <<<"$decision_json")"

  local do_snap reason retention labelslug
  do_snap="$(jq -r '.decision' <<<"$decision_json")"
  reason="$(jq -r '.reason' <<<"$decision_json")"
  retention="$(jq -r '.retention_days' <<<"$decision_json")"
  labelslug="$(jq -r '.label' <<<"$decision_json" | slugify)"

  if [[ "$do_snap" != "true" ]]; then
    log "Skipping snapshot: $reason"
    exit 0
  fi

  local ts name meta_path expire_at
  ts="$(date +'%Y%m%d_%H%M%S')"
  name="snap-$ts"
  [[ -n "$labelslug" ]] && name="${name}-${labelslug}"

  expire_at=$(( now + retention*24*3600 ))
  meta_path="$META_DIR/$name.json"

  case "${BACKEND:-}" in
    btrfs)
      : "${BTRFS_SUBVOL:?Set BTRFS_SUBVOL}"
      : "${BTRFS_SNAPDIR:?Set BTRFS_SNAPDIR}"
      log "Creating Btrfs snapshot $name from $BTRFS_SUBVOL -> $BTRFS_SNAPDIR"
      snap_path="$(create_btrfs_snapshot "$BTRFS_SUBVOL" "$BTRFS_SNAPDIR" "$name")"
      jq -n --arg name "$name" --arg type "btrfs" --arg src "$BTRFS_SUBVOL" --arg path "$snap_path" \
            --arg reason "$reason" --arg time "$(date -Is)" \
            --argjson retention "$retention" --argjson expire_at "$expire_at" \
            --argjson metrics "$metrics" \
            '{name:$name,backend:$type,src:$src,path:$path,reason:$reason,time:$time,retention_days:$retention,expire_at:$expire_at,metrics:$metrics}' > "$meta_path"
      prune_btrfs "$BTRFS_SNAPDIR" "$now"
      ;;
    lvm)
      : "${LVM_LV_PATH:?Set LVM_LV_PATH}"
      : "${LVM_SNAP_SIZE:?Set LVM_SNAP_SIZE}"
      log "Creating LVM snapshot $name from $LVM_LV_PATH size $LVM_SNAP_SIZE"
      snap_dev="$(create_lvm_snapshot "$LVM_LV_PATH" "$LVM_SNAP_SIZE" "$name")"
      jq -n --arg name "$name" --arg type "lvm" --arg src "$LVM_LV_PATH" --arg dev "$snap_dev" \
            --arg reason "$reason" --arg time "$(date -Is)" \
            --argjson retention "$retention" --argjson expire_at "$expire_at" \
            --argjson metrics "$metrics" \
            '{name:$name,backend:$type,src:$src,device:$dev,reason:$reason,time:$time,retention_days:$retention,expire_at:$expire_at,metrics:$metrics}' > "$meta_path"
      prune_lvm "$LVM_LV_PATH" "$now"
      ;;
    *)
      echo "Set BACKEND=btrfs or BACKEND=lvm"
      exit 2
      ;;
  esac

  log "Snapshot $name created. Metadata: $meta_path (expires $(date -d "@$expire_at" +%F))"
}

main "$@"
EOF

Make it executable:

sudo chmod +x /usr/local/sbin/ai-snapshot.sh

Configure and run

Pick your backend and run a test.

  • Btrfs example:

    sudo mkdir -p /@snapshots
sudo BACKEND=btrfs \
   BTRFS_SUBVOL=/ \
   BTRFS_SNAPDIR=/@snapshots \
   AI_PROVIDER=none \
   /usr/local/sbin/ai-snapshot.sh

  • LVM example:

    sudo BACKEND=lvm \
   LVM_LV_PATH=/dev/vg0/root \
   LVM_SNAP_SIZE=5G \
   AI_PROVIDER=none \
   /usr/local/sbin/ai-snapshot.sh

Add AI policy if you like:

  • Local (Ollama):

    export AI_PROVIDER=ollama
export OLLAMA_MODEL=llama3.1:8b
# Optional label hint:
export AI_LABEL=pre-update
sudo -E BACKEND=btrfs BTRFS_SUBVOL=/ BTRFS_SNAPDIR=/@snapshots /usr/local/sbin/ai-snapshot.sh

  • Cloud (OpenAI):

    export AI_PROVIDER=openai
export OPENAI_MODEL=gpt-4o-mini
export OPENAI_API_KEY=sk-REDACTED
sudo -E BACKEND=lvm LVM_LV_PATH=/dev/vg0/root LVM_SNAP_SIZE=5G /usr/local/sbin/ai-snapshot.sh

Tip: Set environment variables in /etc/default/ai-snapshot or a systemd drop-in to avoid long commands.

Automate with systemd (recommended)

Create a service:

sudo tee /etc/systemd/system/ai-snapshot.service >/dev/null <<'EOF'
[Unit]
Description=AI-driven snapshot
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=-/etc/default/ai-snapshot
ExecStart=/usr/local/sbin/ai-snapshot.sh
EOF

Create a timer (hourly, persistent):

sudo tee /etc/systemd/system/ai-snapshot.timer >/dev/null <<'EOF'
[Unit]
Description=Run AI snapshot hourly

[Timer]
OnBootSec=5m
OnUnitActiveSec=1h
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-snapshot.timer
systemctl list-timers | grep ai-snapshot

Example /etc/default/ai-snapshot:

BACKEND=btrfs
BTRFS_SUBVOL=/
BTRFS_SNAPDIR=/@snapshots
AI_PROVIDER=ollama
OLLAMA_MODEL=llama3.1:8b
RETENTION_DAYS_DEFAULT=14
AI_LABEL=hourly

Cron alternative:

echo '0 * * * * root /usr/local/sbin/ai-snapshot.sh >> /var/log/ai-snapshot.log 2>&1' | sudo tee /etc/cron.d/ai-snapshot

3–5 actionable tips and real-world examples

1) Start with heuristics, then add AI
Run with AI_PROVIDER=none for a week. Review metrics and snapshot frequency in /var/lib/ai-snapshot/meta/. Flip to AI when you know your baseline.

2) Align snapshots to change windows
Set AI_LABEL=pre-update and run before patching: sudo AI_PROVIDER=none AI_LABEL=pre-update BACKEND=btrfs BTRFS_SUBVOL=/ BTRFS_SNAPDIR=/@snapshots /usr/local/sbin/ai-snapshot.sh sudo apt update && sudo apt -y full-upgrade Real-world: teams snapshot just before maintenance windows to guarantee a clean rollback point.

3) Tune pruning by role
For databases on LVM, increase RETENTION_DAYS_DEFAULT or let AI suggest longer retention when churn is high. For stateless dev desktops, dial retention down to save space.

4) Label snapshots for clarity
The script writes JSON metadata with reason and label. Use labels like pre-update, pre-deploy, high-churn for faster incident triage.

5) Keep it auditable and small
Everything is Bash + standard tools. Keep prompts and policy simple; your SRE on-call can read and reason about it in minutes.

Safety notes

  • Always test on non-production volumes first.

  • Ensure Btrfs snapshots live on the same filesystem as the source subvolume.

  • LVM snapshots consume COW space; if they fill, they become invalid. Size them generously for busy windows.

  • Store API keys securely (systemd EnvironmentFile with correct permissions, or a secret manager).

  • Network hiccups? The script falls back to heuristics if AI is unavailable.

Conclusion and next steps

You now have a compact, extensible way to take snapshots when they matter and keep them as long as they’re valuable. Start with the heuristic mode, observe behavior, then enable a local or cloud AI policy for smarter decisions.

Next steps:

  • Put the script into systemd and run it hourly.

  • Try Ollama for local decisions.

  • Add role-specific prompts (e.g., “database node,” “CI runner,” “developer workstation”).

  • Extend pruning to consider AI “importance” scores, not just time.

If this improved your backups, share it with your team—and let me know which signals your AI found most predictive!

Related posts