Posted on
• Artificial Intelligence

Artificial Intelligence Disk Usage Analysis

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Disk Usage Analysis with Bash: Turn “Where Did My Space Go?” Into Action

Woke up to a full disk and a downed service? Traditional tools like du, df, and ncdu are great at showing “what’s big,” but they don’t explain “why it’s big,” “what’s safe to delete,” or “what looks abnormal today.” This post shows how to add a thin layer of AI and anomaly detection to your existing Bash workflow so you can prioritize, explain, and act on disk usage—faster and more safely.

You’ll get:

  • A fast, scriptable way to snapshot disk usage

  • An LLM-powered summary that suggests safe cleanups

  • A small Python anomaly detector to catch runaways

  • Real-world validation steps with classic Linux tools

  • Automation tips so your disk doesn’t surprise you again

Note: Everything below is CLI-first, Linux-friendly, and easy to roll back if you don’t like it.

Why AI for Disk Usage?

  • Scale and complexity: Modern systems generate logs, caches, containers, images, and build artifacts across multiple mount points. Finding the true culprit is noisy.

  • Explainability: Knowing a directory is large is different from understanding if it’s a cache, a log backlog, or a build artifact—and which commands are safe for cleanup.

  • Prioritization: AI can quickly categorize space by “type” and “risk,” while anomaly detection flags unusual spikes you might otherwise miss.

  • Speed to action: Combine du output with an LLM and get command-ready suggestions you can validate with ncdu or fdupes.

Prerequisites and Installation

Install the basics you’ll use below. Pick the command for your distro.

  • Debian/Ubuntu (apt):
sudo apt-get update
sudo apt-get install -y findutils coreutils ncdu fdupes python3 python3-pip python3-venv curl
  • Fedora/RHEL (dnf):
sudo dnf install -y findutils coreutils ncdu fdupes python3 python3-pip python3-virtualenv curl
  • openSUSE (zypper):
sudo zypper install -y findutils coreutils ncdu fdupes python3 python3-pip python3-virtualenv curl

Optional (LLM runner): Ollama (runs local open-source LLMs)

curl -fsSL https://ollama.com/install.sh | sh
# Pull a small, capable model (choose one)
ollama pull llama3:latest
# or
ollama pull mistral:latest

Tip: On some RHEL systems, you may need EPEL for ncdu/fdupes.

Step 1: Snapshot Disk Usage Fast

Start with a quick, repeatable snapshot. This gives you:

  • Top directories by size (fast triage)

  • Per-file TSV for deeper analysis/anomaly detection

Create scan.sh:

#!/usr/bin/env bash
set -euo pipefail

ROOT="${1:-/}"         # Path to scan, e.g., /, /var, /home
DEPTH="${2:-4}"        # du depth
OUT="${3:-scan.tsv}"   # Output base name

# 1) Directory-level sizes (fast triage)
sudo du -B1 -x --max-depth="$DEPTH" "$ROOT" 2>/dev/null | sort -nr > "$OUT"

# 2) Per-file TSV: size<TAB>mtime_epoch<TAB>path (for anomaly detection)
TSV="${OUT%.tsv}.files.tsv"
sudo find "$ROOT" -xdev -type f -printf '%s\t%T@\t%p\n' 2>/dev/null > "$TSV"

echo "Wrote:"
echo "  - $OUT         (directory sizes)"
echo "  - $TSV         (per-file TSV: size<TAB>mtime<TAB>path)"

Usage examples:

chmod +x scan.sh
./scan.sh /var 4 var-scan.tsv
./scan.sh /home 3 home-scan.tsv

Notes:

  • -xdev keeps the scan on the same filesystem (safer, faster).

  • Adjust DEPTH to trade speed vs. granularity.

Step 2: Summarize and Suggest Cleanups with an LLM

Use a small, local model to categorize big directories and propose safe cleanup commands. This is not a blind “rm” bot; it’s a brainstorming assistant you validate with classic tools.

Create ai_summarize.sh:

#!/usr/bin/env bash
set -euo pipefail

ROOT="${1:-/}"
TOP="${2:-50}"
MODEL="${MODEL:-llama3:latest}"  # or mistral:latest

# Prepare a "top N" list of heavy directories
TMP="/tmp/top_dirs.$$.tsv"
du -B1 -x --max-depth=3 "$ROOT" 2>/dev/null | sort -nr | head -n "$TOP" > "$TMP"

PROMPT=$(cat <<'EOF'
You are a Linux storage assistant. Given a list of directories with sizes in bytes, propose safe cleanup actions.
For each path, include:

- category (logs, cache, build, media, VM/container, dupes, unknown)

- why it is large (short explanation)

- example cleanup command(s) (default to read-only commands; include rm only for obvious cache/tmp)

- risk level (low/med/high)

- verification tip (a command to confirm before deleting)
EOF
)

# Feed the context into the LLM
ollama run "$MODEL" <<EOF
$PROMPT

Root: $ROOT
Top directories (bytes path):
$(awk '{printf "%12d  %s\n",$1,$2}' "$TMP")
EOF

Usage:

chmod +x ai_summarize.sh
./ai_summarize.sh /var 40

What you’ll get: short, categorized suggestions like “/var/log/journal — logs — suggest journalctl --vacuum-time=7d (low risk)” and validation tips before deleting anything.

Step 3: Detect Size Anomalies with a Tiny ML Script (No Data Science Degree Required)

When today’s layout looks like yesterday’s +10%, you’re fine. When a log or cache jumps 5× overnight, you want to know.

Create a virtual environment (optional but recommended):

  • apt (Debian/Ubuntu):
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn
  • dnf (Fedora/RHEL):
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn
  • zypper (openSUSE):
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install pandas scikit-learn

For a single-snapshot sanity check using z-scores (no history required), this lightweight script is enough:

Create anomaly_disk.py:

#!/usr/bin/env python3
import sys, math, statistics, os
from collections import defaultdict

if len(sys.argv) < 2:
    print("Usage: anomaly_disk.py scan.files.tsv [depth]", file=sys.stderr)
    sys.exit(1)

path = sys.argv[1]
depth = int(sys.argv[2]) if len(sys.argv) > 2 else 3

sizes = defaultdict(int)
with open(path, 'r', errors='ignore') as f:
    for line in f:
        try:
            size_s, mtime_s, p = line.rstrip('\n').split('\t', 2)
            size = int(size_s)
        except ValueError:
            continue
        parts = [seg for seg in p.split(os.sep) if seg]
        key = '/' + '/'.join(parts[:depth]) if parts else '/'
        sizes[key] += size

vals = [math.log10(v + 1) for v in sizes.values()]
mu = statistics.mean(vals)
sd = statistics.pstdev(vals) or 1.0

scored = [(((math.log10(v + 1) - mu) / sd), v, k) for k, v in sizes.items()]
scored.sort(reverse=True)

print("zscore  bytes        path")
for z, v, k in scored[:30]:
    flag = "ANOMALY" if z > 2.5 else ""
    print(f"{z:5.2f}  {v:12d}  {k} {flag}")

Usage:

chmod +x anomaly_disk.py
./anomaly_disk.py var-scan.files.tsv 3

Interpretation:

  • zscore > 2.5: unusually large relative to peers at the same depth.

  • Investigate with ncdu or ls before acting.

Want multi-day trend detection? Save daily TSV snapshots and diff totals per path. You can extend the script to read multiple days and flag sharp increases.

Step 4: Validate and Act (Real-world Workflow)

Use AI as a guide, then verify and act with classic tools.

  • Inspect a suspicious path interactively:
sudo ncdu /var
sudo ncdu /home/youruser
  • Find and review duplicate files before deletion:
fdupes -r -S ~/Downloads
# To interactively delete duplicates (dangerous; read prompts carefully):
# fdupes -rdN ~/Downloads
  • Common safe cleanups (confirm first!):
# Systemd journal (keep 7 days)
sudo journalctl --vacuum-time=7d

# apt caches (Debian/Ubuntu)
sudo apt-get clean
sudo apt-get autoclean

# dnf caches (Fedora/RHEL)
sudo dnf clean all

# zypper caches (openSUSE)
sudo zypper clean

# Docker artifacts (if you use Docker)
docker system df
docker system prune -f
docker image prune -a -f

Tip: For apps creating their own caches, target only obvious cache dirs (e.g., node_modules in temporary build trees, pip caches, browser caches) after verifying they’re not part of active deployments.

Step 5: Automate the Boring Stuff

Run scans on a schedule and review anomalies proactively.

  • Save scripts:
sudo install -m 0755 scan.sh /usr/local/bin/scan.sh
sudo install -m 0755 anomaly_disk.py /usr/local/bin/anomaly_disk.py
sudo install -m 0755 ai_summarize.sh /usr/local/bin/ai_summarize.sh
  • Create a log directory:
sudo mkdir -p /var/log/diskwatch
  • Cron example (scan /var and /home nightly):
sudo crontab -e

Add:

# m h dom mon dow user  command
5  3 * * * root /usr/local/bin/scan.sh /var 4 /var/log/diskwatch/var-$(date +\%F).tsv
15 3 * * * root /usr/local/bin/anomaly_disk.py /var/log/diskwatch/var-$(date +\%F).files.tsv 3 > /var/log/diskwatch/var-anomalies-$(date +\%F).txt
25 3 * * * root /usr/local/bin/scan.sh /home 3 /var/log/diskwatch/home-$(date +\%F).tsv
35 3 * * * root /usr/local/bin/anomaly_disk.py /var/log/diskwatch/home-$(date +\%F).files.tsv 3 > /var/log/diskwatch/home-anomalies-$(date +\%F).txt
  • Optional: Generate an LLM summary weekly for a human-friendly review:
45 3 * * 1 root /usr/local/bin/ai_summarize.sh / 50 > /var/log/diskwatch/summary-$(date +\%F).txt

Example Outcomes You Can Expect

  • “/var/log/journal grew 5× since yesterday” → vacuum to 7 days, confirm services are healthy.

  • “/home/dev/builds/…/node_modules is 12 GB” → clean old build artifacts, keep the active one.

  • “/var/lib/docker overlay2 30 GB” → prune unused images/volumes after verifying containers.

  • “~/Downloads duplicates” → de-duplicate with fdupes.

The AI summary can quickly bucket paths into logs, caches, builds, media, or containers and propose only low-risk, read-first commands, leaving irreversible steps for explicit, informed action.

Conclusion and Call to Action

Disk space incidents rarely come from nowhere—they come from unmonitored growth. By combining Bash-native snapshots with a small LLM and a simple anomaly detector, you get:

  • Faster triage

  • Clearer explanations

  • Safer, validated cleanups

  • Less 3 a.m. firefighting

Your next steps: 1) Install prerequisites and save the three scripts. 2) Run a manual scan today on /var and /home. 3) Generate the AI summary and sanity-check with ncdu. 4) Add the cron entries so tomorrow’s surprises become tomorrow’s insights.

Have improvements or a favorite cleanup pattern? Turn it into a prompt snippet or a shell function and share it with your team. Your disks (and future self) will thank you.

Related posts