Posted on
• Artificial Intelligence

AI-Based Disk Usage Analysis

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Based Disk Usage Analysis: Turn du Noise Into Actionable Insights

Ever had a server go read-only because the disk filled up overnight? You SSH in, run du -sh /*, eyeball some big directories, and start deleting caches or old logs. It works—until it doesn’t. What if you could turn raw disk usage into clear, prioritized insights and safe, reviewable cleanup steps in minutes?

This article shows how to blend classic Linux tooling (du, find, ncdu, jq) with AI to:

  • Summarize disk usage into human-friendly categories

  • Detect anomalies between snapshots (what grew, where, and why)

  • Generate a dry-run cleanup plan you can safely review

All from Bash, locally or via an API.

Why AI for Disk Usage?

  • Context over raw numbers: du and ncdu are great, but they don’t explain growth patterns, root causes, or which actions are safest for your environment.

  • Pattern recognition: AI can recognize common culprits (journals, container layers, build artifacts, caches) and suggest safe, targeted cleanups.

  • Privacy options: You can run a local LLM (e.g., via Ollama) to keep paths and file names private, or use a cloud API if that fits your workflow.

  • Repeatability: With snapshots and AI summaries, you create a lightweight, explainable “storage health” workflow.

What We’ll Build

1) A compact JSON snapshot of your disk usage using standard tools
2) An AI-generated summary explaining where your space goes
3) A diff workflow to catch weekly/monthly anomalies
4) A dry-run cleanup plan as a shell script (you review before executing)

Prerequisites and Installation

We’ll use:

  • ncdu (fast directory scanner)

  • jq (JSON processing)

  • curl (API calls if using a cloud LLM)

  • Ollama (optional, for running a local LLM)

Install these with your package manager:

  • Debian/Ubuntu (apt)
sudo apt update
sudo apt install -y ncdu jq curl
  • Fedora/RHEL/CentOS Stream (dnf)
sudo dnf install -y ncdu jq curl
  • openSUSE/SLE (zypper)
sudo zypper refresh
sudo zypper install -y ncdu jq curl

Optional: Install Ollama (local LLM)

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama || true
ollama pull llama3:8b

Validate:

ollama run llama3:8b "Hello from my Linux shell"

If you prefer a cloud LLM (e.g., OpenAI), you only need curl and jq, plus an API key:

export OPENAI_API_KEY="sk-..."

Note: If using a cloud LLM, you are sending file paths and sizes to a third party. For sensitive environments, use a local model via Ollama.

Step 1: Create a Disk Snapshot (JSON)

We’ll capture the top directories by size at a modest depth. This is usually enough to identify culprits without producing giant outputs.

TARGET=/
OUTDIR=~/ai-disk-snapshots
mkdir -p "$OUTDIR"

# Top 1000 directories by size (bytes), depth up to 3. Adjust as needed.
du -x -B1 -d 3 "$TARGET" 2>/dev/null | sort -nr | head -n 1000 \
| jq -R -s '
  split("\n")[:-1]
  | map( (split("\t")) as $f | {bytes: ($f[0]|tonumber), path: $f[1]} )
' > "$OUTDIR/$(date -u +%Y%m%dT%H%M%SZ)-du.json"

ls -l "$OUTDIR"

Tip: Use -x to stay on the same filesystem, and -d 3 to limit noise.

Optional deep-dive: ncdu’s interactive UI can help you browse:

sudo ncdu -x /

Step 2: Summarize Disk Usage with AI

Use a local LLM (Ollama) or a cloud API to convert raw JSON into a short, actionable report.

Local (Ollama):

SNAP=$(ls -1 "$OUTDIR"/*-du.json | tail -n1)

PROMPT=$(cat <<'P'
You are a Linux storage SRE. You will receive a JSON array of items {bytes, path}.

- Identify the biggest consumers by category (e.g., logs, containers, caches, build artifacts, user homes).

- Explain likely root causes and risks.

- Suggest 3–5 safe, reviewable cleanup actions with exact commands.

- Keep it concise and practical.
JSON follows:
P
)
printf "%s\n%s\n" "$PROMPT" "$(cat "$SNAP")" | ollama run llama3:8b

Cloud API (OpenAI example):

SNAP=$(ls -1 "$OUTDIR"/*-du.json | tail -n1)
export OPENAI_API_KEY="sk-..."

jq -Rs --arg sys "You are a Linux storage SRE. Summarize the JSON array of {bytes,path} into prioritized findings and 3–5 safe cleanup actions with exact commands." \
  '{model:"gpt-4o-mini",temperature:0.2,
    messages:[{role:"system",content:$sys},{role:"user",content:.}]}' < "$SNAP" \
| curl -s https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
  -H "Content-Type: application/json" \
  -d @- \
| jq -r '.choices[0].message.content'

Step 3: Detect Anomalies with Snapshots (What Grew and Where)

Take periodic snapshots (e.g., weekly) and compare them. Here’s a simple JSON diff by path:

# Pick an earlier baseline and the most recent snapshot
BASE=$(ls -1 "$OUTDIR"/*-du.json | head -n1)
NEW=$(ls -1 "$OUTDIR"/*-du.json | tail -n1)

jq -s '
  def index_by_path(a):
    a | sort_by(.path) | group_by(.path) | map({key:.[0].path, val:.[0].bytes}) | from_entries;

  (index_by_path(.[0])) as $b |
  (index_by_path(.[1])) as $n |
  ($b + $n | keys | map({
    path: .,
    base: ($b[.] // 0),
    new:  ($n[.] // 0),
    delta: (($n[.] // 0) - ($b[.] // 0))
  }) | sort_by(-.delta)) as $d |
  {deltas: $d}
' "$BASE" "$NEW" > "$OUTDIR/diff.json"

jq '.deltas[:20]' "$OUTDIR/diff.json"

Ask the AI to explain the growth and prioritize actions:

Local (Ollama):

printf "%s\n%s\n" \
  "You are a Linux storage SRE. The JSON shows per-path base/new/delta sizes in bytes. Identify the top growth drivers, explain likely causes, and propose 3–5 safe cleanup steps. JSON:" \
  "$(cat "$OUTDIR/diff.json")" \
| ollama run llama3:8b

Cloud API (OpenAI example):

jq -Rs --arg sys "You are a Linux storage SRE. The input is a JSON diff with {path,base,new,delta}. Summarize growth drivers and suggest safe, reviewable cleanup steps." \
  '{model:"gpt-4o-mini",temperature:0.2,
    messages:[{role:"system",content:$sys},{role:"user",content:.}]}' < "$OUTDIR/diff.json" \
| curl -s https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
  -H "Content-Type: application/json" \
  -d @- | jq -r '.choices[0].message.content'

Step 4: Auto-Generate a Safe, Dry-Run Cleanup Plan

Ask the model to output a reviewable script that only echoes commands. You can then un-comment or run selectively.

PLAN="$OUTDIR/cleanup-plan-$(date -u +%Y%m%dT%H%M%SZ).sh"

printf "%s\n%s\n" \
'Generate a reviewable Bash cleanup plan for Linux based on this disk diff JSON. 
Rules:

- Output only a shell script.

- Use set -euo pipefail.

- Use dry-run: prefix destructive commands with echo.

- Prefer safe, official tools: 
  * journalctl --vacuum-time= or --vacuum-size=
  * package manager cache cleans (apt, dnf, zypper)
  * docker system prune (with filters), podman equivalents
  * npm/yarn/pip/conda cache clean

- Never suggest rm -rf on critical system dirs.

- Include comments explaining each step.
JSON:' \
"$(cat "$OUTDIR/diff.json")" \
| ollama run llama3:8b > "$PLAN"

chmod +x "$PLAN"
sed -n '1,120p' "$PLAN"

Review the plan, then remove “echo” on the lines you trust. You can also copy/paste individual commands instead of running the whole plan.

Real-World Cleanup Cheatsheet (Safe, Common Targets)

  • systemd journal logs
sudo journalctl --disk-usage
sudo journalctl --vacuum-time=7d
# or
sudo journalctl --vacuum-size=1G
  • Package manager caches
# Debian/Ubuntu
sudo apt-get clean

# Fedora/RHEL/CentOS Stream
sudo dnf clean all

# openSUSE/SLE
sudo zypper clean --all
  • Docker/Podman artifacts (dry-run with echo; add filters)
docker system df
echo docker system prune --all --volumes --filter 'until=168h'
# For Podman:
podman system df
echo podman system prune --all --volumes --filter 'until=168h'
  • Language/package caches
pip cache dir
pip cache purge

npm cache verify
npm cache clean --force

yarn cache dir
yarn cache clean

conda clean -a -y
  • Large logs or temp files (verify first!)
sudo du -ah /var/log | sort -hr | head
# Rotate/truncate safely via logrotate or application settings.

Tips for Production Use

  • Privacy first: Prefer local models (Ollama) in environments where path names or project names are sensitive.

  • Scope carefully: Use du -x and --max-depth to reduce noise. Run targeted scans on /var, /home, /opt, /srv, and container storage.

  • Version your snapshots: Keep a rolling window (e.g., last 12) so you can see trends.

  • Review before action: Keep the AI-generated cleanup plan as a dry run. Human review is non-negotiable.

Conclusion and Next Steps

You don’t need a full observability suite to get smart about disk usage. With a few shell commands and an LLM, you can go from raw “du noise” to prioritized insights and safe actions.

Next steps:

  • Install the prerequisites (ncdu, jq, curl, optional Ollama)

  • Take your first snapshot and generate a summary

  • Set a weekly reminder to snapshot again and review the diff

  • Keep the cleanup plan dry-run, and apply changes selectively

When disk pressure strikes again, you’ll spend minutes—not hours—getting to root cause and acting safely.

Related posts