Posted on
• Artificial Intelligence

Artificial Intelligence ZFS Administration

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence ZFS Administration: Smarter Bash for Safer Pools

Your ZFS pool is healthy—until it isn’t. Disks quietly accumulate reallocated sectors, write latencies creep up, and free space dwindles faster than you expect. What if an assistant could continuously read your ZFS and S.M.A.R.T. telemetry, forecast capacity crunches, and suggest safe, shell-ready actions during incidents? In this post, we’ll build an AI-assisted ZFS admin toolkit using familiar Bash, a little Python, and a local LLM you can run offline.

Value in a nutshell:

  • Catch issues earlier with trend forecasts (e.g., “pool will hit 80% in 12.3 days”).

  • Triage incidents faster with AI-generated checklists and shell commands.

  • Generate snapshot retention policies that match data churn and space constraints.

  • Keep everything auditable, reviewable, and under your control.

Why AI + ZFS now?

  • ZFS is rich in signals (zpool status/iostat, zpool events, S.M.A.R.T., dmesg). Humans miss patterns; machines don’t.

  • Local LLMs can summarize complex state and produce actionable, non-destructive steps—without sending data off-host.

  • Light-weight forecasting gives you time to add vdevs, rebalance, or prune snapshots before performance degrades.

What we’ll build

1) A telemetry collector that logs pool and device health in JSON lines. 2) A simple capacity-forecast tool that estimates time to 80% full for each pool. 3) An AI-powered triage command that digests system outputs and prints a clean, shell-ready plan. 4) A snapshot policy assistant that proposes retention tiers (hourly/daily/weekly/monthly) tailored to your datasets.

All scripts are plain Bash (plus one tiny Python file), designed to be reviewed and adapted.

Prerequisites and installation

Note: This guide assumes you already have ZFS installed and your pools imported. We’ll install supporting tools only.

Base tools:

  • jq, curl

  • smartmontools, nvme-cli

  • sysstat (for iostat)

  • bc

  • Python 3 with NumPy/Pandas/Scikit-learn (for capacity forecasting)

  • Optional: Ollama for running a local LLM

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jq curl smartmontools nvme-cli sysstat bc python3 python3-pip
# Optional: install Python scientific stack from distro
sudo apt install -y python3-numpy python3-pandas python3-sklearn
# Or via pip (user install):
python3 -m pip install --user numpy pandas scikit-learn

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y jq curl smartmontools nvme-cli sysstat bc python3 python3-pip
# Optional: Python scientific stack from distro
sudo dnf install -y python3-numpy python3-pandas python3-scikit-learn
# Or via pip (user install):
python3 -m pip install --user numpy pandas scikit-learn

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y jq curl smartmontools nvme-cli sysstat bc python3 python3-pip
# Optional: Python scientific stack from distro
sudo zypper install -y python3-numpy python3-pandas python3-scikit-learn
# Or via pip (user install):
python3 -m pip install --user numpy pandas scikit-learn

Optional: Install Ollama (local LLM runtime; works offline once model is downloaded)

curl -fsSL https://ollama.com/install.sh | sh
# Pull a compact, capable model (adjust to taste)
ollama pull llama3:8b

1) Collect ZFS telemetry to JSON (cron-friendly)

Create /usr/local/sbin/zfs-ai-collect.sh:

#!/usr/bin/env bash
set -euo pipefail

OUT_DIR=${OUT_DIR:-/var/log/zfs-ai}
mkdir -p "$OUT_DIR"
chmod 750 "$OUT_DIR"

timestamp() { date -u +%s; }

collect_pool() {
  local pool="$1" ts size alloc free cap frag health
  ts=$(timestamp)
  read -r size alloc free cap frag health < <(zpool list -Hp -o size,allocated,free,capacity,fragmentation,health "$pool")

  # Grab aggregate ops and bandwidth (best-effort, version-dependent)
  read_ops=0 write_ops=0 read_bps=0 write_bps=0
  mapfile -t io < <(zpool iostat -Hpvy 1 1 "$pool" 2>/dev/null | awk 'NR==1, /capacity/ {next} /^[^ \t]/ {root=$1} $1==root {ro=$3; wo=$4; rb=$5; wb=$6; print ro,wo,rb,wb; exit}')
  if [[ ${#io[@]} -gt 0 ]]; then
     read read_ops write_ops read_bps write_bps <<<"${io[0]}"
  fi

  # Devices in the pool (for quick S.M.A.R.T. health summaries)
  mapfile -t devs < <(zpool status -P "$pool" | awk '/^\t(\/dev|scsi|nvme)/{gsub("\t",""); print $1}' | sort -u)
  smart_bad=0
  for d in "${devs[@]}"; do
     [[ -e "$d" ]] || continue
     if smartctl -H -n standby "$d" >/tmp/smh 2>&1; then
        grep -q "PASSED" /tmp/smh || smart_bad=$((smart_bad+1))
     fi
  done

  jq -nc --arg ts "$(timestamp)" \
         --arg pool "$pool" \
         --argjson size "$size" \
         --argjson alloc "$alloc" \
         --argjson free "$free" \
         --arg cap "$cap" \
         --arg frag "$frag" \
         --arg health "$health" \
         --argjson ro "$read_ops" \
         --argjson wo "$write_ops" \
         --argjson rb "$read_bps" \
         --argjson wb "$write_bps" \
         --argjson smart_bad "$smart_bad" \
    '$ARGS.named' >> "$OUT_DIR/metrics.jsonl"
}

main() {
  mapfile -t pools < <(zpool list -H -o name 2>/dev/null || true)
  for p in "${pools[@]}"; do
    collect_pool "$p"
  done
}
main

Make it executable:

sudo install -m 0750 zfs-ai-collect.sh /usr/local/sbin/zfs-ai-collect.sh

Add a cron job to run every 10 minutes:

sudo tee /etc/cron.d/zfs-ai <<'CRON'
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
OUT_DIR=/var/log/zfs-ai
*/10 * * * * root /usr/local/sbin/zfs-ai-collect.sh
CRON

Check logs:

tail -n 3 /var/log/zfs-ai/metrics.jsonl | jq .

2) Forecast time-to-80% capacity (simple regression)

Create /usr/local/sbin/zfs-ai-forecast.py:

#!/usr/bin/env python3
import sys, json, time, math
from collections import defaultdict

data = defaultdict(list)
for line in sys.stdin:
    try:
        rec = json.loads(line)
        pool = rec["pool"]
        ts = int(rec["ts"])
        size = float(rec["size"])
        alloc = float(rec["alloc"])
        data[pool].append((ts, size, alloc))
    except Exception:
        continue

def eta_to_target(tseries, target):
    if len(tseries) < 5:
        return None
    xs = [t for t,_,_ in tseries]
    ys = [a for _,_,a in tseries]
    x0 = xs[0]
    xs = [x - x0 for x in xs]
    n = len(xs)
    meanx = sum(xs)/n
    meany = sum(ys)/n
    num = sum((xs[i]-meanx)*(ys[i]-meany) for i in range(n))
    den = sum((xs[i]-meanx)**2 for i in range(n))
    if den == 0: return None
    slope = num/den  # bytes/sec
    if slope <= 0: return math.inf
    y0 = ys[-1]
    return (target - y0)/slope

for pool, points in data.items():
    points.sort()
    size = points[-1][1]
    used = points[-1][2]
    target = 0.80 * size
    secs = eta_to_target(points, target)
    if secs is None:
        print(f"{pool}: not enough data (need >= 5 points).")
        continue
    if secs == math.inf:
        print(f"{pool}: usage stable/decreasing; no projected 80% fill.")
        continue
    days = secs / 86400
    pct = 100 * used / size
    print(f"{pool}: {pct:.1f}% used now; projected 80% in {days:.1f} days.")

Install and run:

sudo install -m 0755 zfs-ai-forecast.py /usr/local/sbin/zfs-ai-forecast.py
cat /var/log/zfs-ai/metrics.jsonl | /usr/local/sbin/zfs-ai-forecast.py

Tip: Pipe this into your alerting if the forecast drops below a threshold (e.g., <14 days).

3) AI-assisted incident triage (local LLM via Ollama)

Create /usr/local/sbin/zfs-ai-triage.sh:

#!/usr/bin/env bash
set -euo pipefail
MODEL="${OLLAMA_MODEL:-llama3:8b}"
PROMPT_FILE=$(mktemp)
cleanup(){ rm -f "$PROMPT_FILE"; }
trap cleanup EXIT

gather() {
  echo "=== zpool status ==="
  zpool status -x || true
  echo
  echo "=== zpool list ==="
  zpool list -v || true
  echo
  echo "=== zfs list (top 20 by used) ==="
  zfs list -t filesystem -o name,used,avail,referenced,compressratio -S used | head -n 21 || true
  echo
  echo "=== recent zpool events (last 50) ==="
  zpool events -v -T u 2>/dev/null | tail -n 200 || true
  echo
  echo "=== smartctl summaries ==="
  for d in $(zpool status -P | awk '/^\t(\/dev|scsi|nvme)/{gsub("\t",""); print $1}' | sort -u); do
    echo "# $d"
    smartctl -H -A -n standby "$d" 2>&1 | egrep -i 'overall|reallocated|pending|crc|media_errors|wear|percentage_used|temperature' || true
    echo
  done
  echo "=== dmesg (zfs,sd,nvme errors, 200 lines) ==="
  dmesg -T | egrep -i 'ZFS|pool|sd .*error|nvme.*err|checksum|I/O error' | tail -n 200 || true
}

cat > "$PROMPT_FILE" <<'P'
You are an expert SRE specializing in ZFS on Linux. Analyze the following system outputs.

- Identify likely root causes.

- Classify risk: low/medium/high, and whether immediate action is needed.

- List the next 3–7 safe, non-destructive commands to run for confirmation.

- If replacement/resilver is needed, show the exact zpool/zfs commands, but mark them as "REQUIRES CONFIRMATION".

- Output concise bullet points and shell-ready commands only.

Context:
P

gather >> "$PROMPT_FILE"

if command -v ollama >/dev/null 2>&1; then
  echo "Using Ollama model: $MODEL" >&2
  ollama run "$MODEL" < "$PROMPT_FILE"
else
  echo "No local LLM found (ollama). Install Ollama or set up your own inference endpoint." >&2
  exit 1
fi

Use it when you see trouble:

sudo OLLAMA_MODEL=llama3:8b /usr/local/sbin/zfs-ai-triage.sh | less -R

You’ll get a short list of probable causes, risk levels, and safe commands (e.g., inspect a suspect vdev, scrub planning, or resilver checklist). Always review before acting.

4) Snapshot policy assistant (recommendations you can apply)

Create /usr/local/sbin/zfs-ai-snapshot-policy.sh:

#!/usr/bin/env bash
set -euo pipefail
TOP="${1:-}"
[ -z "$TOP" ] && { echo "Usage: $0 <pool|dataset>"; exit 1; }

report=$(zfs list -r -o name,used,avail,referenced,compressratio,logicalused "$TOP" | sed '1d' | head -n 100)

read -r -d '' SYSTEM <<'SYS' || true
You are assisting with ZFS snapshot policy planning.
Constraints:

- Prioritize datasets with high change rate (approximate by logicalused/used) and low available space.

- Propose a tiered policy like: hourly (N), daily (N), weekly (N), monthly (N).

- Keep total snapshot space under ~30% of used unless ample free space exists.

- Output for each dataset: "dataset: hourly=X daily=Y weekly=Z monthly=W".
SYS

PROMPT="$SYSTEM

Here is current state (columns: name used avail referenced compressratio logicalused):
$report

Produce recommendations for the top 20 datasets only. Keep it short."

if command -v ollama >/dev/null 2>&1; then
  ollama run "${OLLAMA_MODEL:-llama3:8b}" <<< "$PROMPT"
else
  echo "Ollama not found; printing a safe default for top 20:"
  echo "$report" | awk 'NR<=20 {print $1": hourly=24 daily=14 weekly=8 monthly=6"}'
fi

Example: apply an hourly snapshot for 24 hours to one dataset (review first!)

DATASET=tank/home
# Create snapshot with ISO-like tag
zfs snapshot -r "$DATASET@auto-$(date -u +%Y%m%d-%H%MZ)"
# Prune hourly snapshots older than 24h for that dataset (non-recursive example)
zfs list -t snapshot -o name -s creation | grep "^$DATASET@auto-" | \
  awk -v cutoff="$(date -u -d '24 hours ago' +%s)" '
    {
      split($0, a, "@auto-");
      gsub("Z","",a[2]); gsub("-","",a[2]); # naive parse
    }
  ' # implement your preferred pruning logic or use tools like sanoid

For production-grade snapshot orchestration and pruning, consider purpose-built tools (e.g., sanoid/syncoid) and use the AI output as planning input.

Real-world-style example

  • Telemetry shows pool “tank” at 73.2% with rising write_ops/read_ops and 1 device with S.M.A.R.T. not PASSED.

  • Forecast says: “tank: 73.2% used now; projected 80% in 9.8 days.”

  • Triage assistant outputs:

    • Likely cause: one device with pending sectors; vdev experiencing retries; medium risk; plan a scrub and device replacement ASAP.
    • Safe checks:
    • zpool status -v tank
    • smartctl -x /dev/sdX
    • zpool iostat -v tank 2 10
    • Replacement (REQUIRES CONFIRMATION):
    • zpool offline tank /dev/sdX
    • replace drive physically
    • zpool replace tank /dev/sdX
    • zpool status tank (monitor resilver)

Meanwhile, snapshot policy assistant proposes tightening hourly snapshots (higher churn datasets) and trimming weekly retention—buying space while you schedule replacement and capacity expansion.

Safety checklist

  • Never run destructive commands automatically. Require human review or an explicit ALLOW flag.

  • Test scripts on a non-production host or a VM with a throwaway pool.

  • Keep your LLM local (Ollama) if logs or hostnames are sensitive.

  • Version-control your prompts and outputs for traceability.

  • Complement with monitoring/alerting (Prometheus node_exporter + textfile collector, or simple cron + mail).

Conclusion and next steps

AI won’t replace your ZFS expertise—but it will make it faster to see trends, triage incidents, and plan safe actions. Start by:

  • Scheduling the telemetry collector (done).

  • Running the forecast weekly; alert when T–80% < 14 days.

  • Using triage during scrubs, checksum errors, or rising latencies.

  • Letting the snapshot assistant propose retention tiers, then implementing with your preferred tooling.

Next steps:

  • Export metrics to Prometheus (textfile) and graph forecasts in Grafana.

  • Add arcstats and per-dataset change tracking for richer signals.

  • Extend the triage prompt with site-specific runbooks.

If you found this useful, wire it into your cron/alert flow this week. Your future self (and your pools) will thank you.

Related posts