Posted on
• Artificial Intelligence

AI-Powered File Classification

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered File Classification with Bash: Turn Chaos into Structure

Ever opened your Downloads folder and felt a mix of dread and resignation? PDFs, invoices, logs, photos, archives—thousands of files with cryptic names and no structure. Traditional tools like file, grep, and MIME types help, but they can’t tell you if a PDF is an invoice, a resume, or a research paper.

What if your shell could? With a few battle-tested CLI tools and an LLM (local or cloud), you can auto-extract content, classify files by meaning, tag them, and move them into a clean, auditable structure—all from Bash.

This guide shows you how.

Why AI-powered classification is worth it

  • Content awareness beats guesses: Filenames and extensions lie; content doesn’t. AI can read snippets and classify semantically.

  • Reproducible workflows: Scriptable, explainable, and adaptable—no manual dragging and dropping.

  • Privacy-first options: Run models locally with ollama or use an OpenAI-compatible API if you prefer hosted models.

  • Faster retrieval and compliance: Tags like “finance”, “HR”, “legal”, “code”, “photos” get your files in the right bins—plus optional sensitivity labels.

What we’ll build

A Bash-based pipeline that: 1. Extracts text and metadata from common file types (PDF, DOCX, text, images via OCR). 2. Feeds a compact “file summary” to an LLM with a strict JSON schema. 3. Parses the model’s response with jq. 4. Applies actions: move to category folders, set xattrs, and print a CSV log.

You get two backends:

  • Local: ollama with a small, fast model.

  • Cloud: any OpenAI-compatible chat API via curl.

Prerequisites and installation

We’ll install only widely-available packages. Commands for apt, dnf, and zypper are provided.

Core tools we’ll use:

  • jq (JSON parsing)

  • file (MIME detection)

  • ripgrep (optional keyword fallback)

  • exiftool (metadata; package names vary)

  • pdftotext (from poppler)

  • tesseract-ocr (optional OCR for images)

  • docx2txt (for .docx)

  • attr (for xattrs, optional)

  • curl (for API calls)

Install them with:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y jq ripgrep file libimage-exiftool-perl poppler-utils tesseract-ocr docx2txt attr curl
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y jq ripgrep file perl-Image-ExifTool poppler-utils tesseract docx2txt attr curl
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y jq ripgrep file perl-Image-ExifTool poppler-tools tesseract docx2txt attr curl

Optional: local LLM with ollama (works on most modern distros):

curl -fsSL https://ollama.com/install.sh | sh
# Optionally enable at boot (if installed as a service)
sudo systemctl enable --now ollama 2>/dev/null || true
# Pull a small general model
ollama pull llama3:8b

Cloud option (OpenAI-compatible):

  • Set your key and endpoint (OpenAI example):
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://api.openai.com/v1"
export OPENAI_MODEL="gpt-4o-mini"
  • Or for an OpenAI-compatible self-hosted endpoint, set OPENAI_BASE_URL accordingly.

The script: ai_classify.sh

What it does:

  • Extracts a compact text snippet + metadata from each file.

  • Prompts an LLM to return strict JSON:

    • top_category: one-of preset categories or “other”
    • secondary_tags: array of keywords
    • sensitivity: one of public/internal/confidential
    • suggested_name: a clean filename stem
    • confidence: 0.0–1.0

  • Moves the file into ./classified/<top_category>/.

  • Optionally sets xattrs with tags (if attr is installed).

  • Writes a CSV log to classification_log.csv.

Save this as ai_classify.sh and make it executable: chmod +x ai_classify.sh.

#!/usr/bin/env bash
set -euo pipefail

# Configuration
ALLOWED_CATEGORIES=("finance" "legal" "hr" "code" "logs" "media" "photos" "audio" "video" "personal" "work" "education" "research" "receipts" "invoices" "contracts" "medical" "travel" "other")
MAX_BYTES=5000
OUTDIR="${OUTDIR:-classified}"
DRY_RUN=0
BACKEND="${AI_BACKEND:-auto}"     # auto|ollama|openai
OLLAMA_MODEL="${OLLAMA_MODEL:-llama3:8b}"
OPENAI_MODEL="${OPENAI_MODEL:-${OPENAI_MODEL:-gpt-4o-mini}}"
OPENAI_BASE_URL="${OPENAI_BASE_URL:-https://api.openai.com/v1}"
LOGFILE="${LOGFILE:-classification_log.csv}"

usage() {
  echo "Usage: $0 [-n] DIRECTORY"
  echo "  -n   dry run (no moves, just print results)"
  exit 1
}

while getopts ":n" opt; do
  case "$opt" in
    n) DRY_RUN=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND-1))

[ $# -eq 1 ] || usage
ROOT="$1"
[ -d "$ROOT" ] || { echo "Not a directory: $ROOT" >&2; exit 2; }

need() { command -v "$1" >/dev/null 2>&1 || { echo "Missing dependency: $1" >&2; exit 3; }; }
need file
need jq

have() { command -v "$1" >/dev/null 2>&1; }

# Backend autodetect
detect_backend() {
  if [ "$BACKEND" != "auto" ]; then
    echo "$BACKEND"
    return
  fi
  if have ollama; then
    echo "ollama"
  elif [ -n "${OPENAI_API_KEY:-}" ]; then
    echo "openai"
  else
    echo "none"
  fi
}
BACKEND="$(detect_backend)"
if [ "$BACKEND" = "none" ]; then
  echo "No AI backend available. Install ollama or set OPENAI_API_KEY." >&2
  exit 4
fi

json_list_categories() {
  printf '['
  local first=1
  for c in "${ALLOWED_CATEGORIES[@]}"; do
    [ $first -eq 1 ] || printf ','
    printf '"%s"' "$c"
    first=0
  done
  printf ']'
}

extract_text() {
  # Outputs up to MAX_BYTES of representative text to stdout
  local f="$1"
  local mime
  mime="$(file -b --mime-type -- "$f" || true)"
  case "$mime" in
    text/*|application/json|application/xml)
      head -c "$MAX_BYTES" -- "$f" 2>/dev/null || true
      ;;
    application/pdf)
      if have pdftotext; then
        pdftotext -q "$f" - -layout 2>/dev/null | head -c "$MAX_BYTES" || true
      fi
      ;;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document)
      if have docx2txt; then
        docx2txt "$f" - 2>/dev/null | head -c "$MAX_BYTES" || true
      fi
      ;;
    image/*)
      if have tesseract; then
        tesseract "$f" stdout 2>/dev/null | head -c "$MAX_BYTES" || true
      fi
      ;;
    *)
      # Try simple text read in case it's actually text
      head -c "$MAX_BYTES" -- "$f" 2>/dev/null || true
      ;;
  esac
}

file_summary_json() {
  # Emits a compact JSON summary for LLM
  local f="$1"
  local name base ext size mime sha1 meta ex
  name="$(basename -- "$f")"
  base="${name%.*}"
  ext="${name##*.}"
  size="$(stat -c %s -- "$f" 2>/dev/null || stat -f %z -- "$f")"
  mime="$(file -b --mime-type -- "$f" || true)"
  sha1="$(sha1sum -- "$f" 2>/dev/null | awk '{print $1}')"
  meta=""
  if have exiftool; then
    # Grab a few helpful metadata fields if present (silently ignore errors)
    meta="$(exiftool -s -s -s -Title -Subject -Keywords -Author -Creator -CreateDate -DocumentName -- "$f" 2>/dev/null | head -n 20 | tr '\n' ';' | sed 's/"/\\"/g')"
  fi
  ex="$(extract_text "$f" | tr -d '\000' | head -c "$MAX_BYTES" | sed 's/"/\\"/g')"

  printf '{"name":"%s","ext":"%s","size":%s,"mime":"%s","sha1":"%s","metadata_hint":"%s","content_snippet":"%s"}' \
    "$name" "$ext" "${size:-0}" "$mime" "$sha1" "${meta:-}" "${ex:-}"
}

build_prompt() {
  local summary="$1"
  local categories
  categories="$(json_list_categories)"
  cat <<EOF
You are an expert file classifier. Return ONLY strict minified JSON with keys:
{"top_category": string in $categories,
 "secondary_tags": array of 1-6 short lowercase tags,
 "sensitivity": one of ["public","internal","confidential"],
 "suggested_name": short kebab-case stem without extension,
 "confidence": number 0.0-1.0}

Rules:

- Base decisions on provided content snippet, metadata_hint, name, and mime.

- Prefer exact categories from the list; use "other" if none fit.

- Keep JSON on one line. No code fences, no commentary.

File summary:
$summary
EOF
}

classify_with_ollama() {
  local prompt="$1"
  ollama run "$OLLAMA_MODEL" -p "$prompt" 2>/dev/null
}

classify_with_openai() {
  local prompt="$1"
  curl -sS "$OPENAI_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-Type: application/json" \
    -d @- <<JSON
{
  "model": "${OPENAI_MODEL}",
  "temperature": 0.2,
  "response_format": { "type": "json_object" },
  "messages": [
    {"role":"system","content":"You are a careful, deterministic file classifier."},
    {"role":"user","content": $(jq -Rs . <<< "$prompt")}
  ]
}
JSON
}

ensure_log_header() {
  if [ ! -f "$LOGFILE" ]; then
    echo "timestamp,sha1,src_path,top_category,secondary_tags,sensitivity,suggested_name,confidence" > "$LOGFILE"
  fi
}

classify_file() {
  local f="$1"
  local summary prompt raw out json_line top tags sens name conf catdir stem ext newname dest
  summary="$(file_summary_json "$f")"
  prompt="$(build_prompt "$summary")"

  case "$BACKEND" in
    ollama)
      raw="$(classify_with_ollama "$prompt" || true)"
      json_line="$(echo "$raw" | tr -d '\n' | sed -E 's/^.*(\{.*\}).*$/\1/')" # best-effort
      ;;
    openai)
      raw="$(classify_with_openai "$prompt" || true)"
      json_line="$(echo "$raw" | jq -rc '.choices[0].message.content' 2>/dev/null || true)"
      ;;
  esac

  # Validate JSON
  out="$(echo "$json_line" | jq -rc '.' 2>/dev/null || true)"
  if [ -z "$out" ]; then
    echo "WARN: Failed to parse JSON for $f; raw:" >&2
    echo "$raw" >&2
    return 1
  fi

  top="$(echo "$out" | jq -r '.top_category')"
  tags="$(echo "$out" | jq -c '.secondary_tags')"
  sens="$(echo "$out" | jq -r '.sensitivity')"
  name="$(echo "$out" | jq -r '.suggested_name')"
  conf="$(echo "$out" | jq -r '.confidence')"

  # Prepare destination
  ext="${f##*.}"
  stem="${name:-$(basename "$f" | sed 's/\.[^.]*$//')}"
  [ -n "$top" ] || top="other"
  catdir="$OUTDIR/$top"
  mkdir -p -- "$catdir"

  newname="${stem}.${ext}"
  dest="$catdir/$newname"
  # Avoid overwrite
  if [ -e "$dest" ]; then
    dest="${catdir}/${stem}-$(date +%s).${ext}"
  fi

  # xattrs for tags if available
  if have setfattr; then
    : # no-op
  elif have attr; then
    : # attr pkg provides setfattr
  fi
  if have setfattr; then
    setfattr -n user.tags -v "$(echo "$tags" | tr -d '[]"' | tr ',' ' ')" -- "$f" 2>/dev/null || true
    setfattr -n user.sensitivity -v "$sens" -- "$f" 2>/dev/null || true
  fi

  if [ "$DRY_RUN" -eq 1 ]; then
    echo "[DRY] $f -> $dest | cat=$top tags=$tags sens=$sens conf=$conf"
  else
    mv -n -- "$f" "$dest"
    echo "Moved: $f -> $dest (cat=$top, sens=$sens, conf=$conf)"
  fi

  # Log CSV
  ensure_log_header
  local sha1
  sha1="$(sha1sum -- "$dest" 2>/dev/null | awk '{print $1}')"
  printf '%s,%s,"%s",%s,"%s",%s,%s,%s\n' \
    "$(date -Is)" "$sha1" "$dest" "$top" "$(echo "$tags" | tr -d '"')" "$sens" "$stem" "$conf" \
    >> "$LOGFILE"
}

export -f classify_file file_summary_json extract_text build_prompt classify_with_ollama classify_with_openai

# Walk the directory
find "$ROOT" -type f ! -path "$OUTDIR/*" -print0 | while IFS= read -r -d '' f; do
  # Skip obviously binary archives to save tokens; you can remove this if needed.
  case "$f" in
    *.zip|*.tar|*.gz|*.bz2|*.xz|*.7z) echo "Skip archive: $f"; continue ;;
  esac
  classify_file "$f" || true
done

Usage examples:

  • Dry run first:
./ai_classify.sh -n ~/Downloads
  • Then commit to moves:
./ai_classify.sh ~/Downloads

Result:

  • Files land in ./classified/<category>/.

  • A classification_log.csv captures every decision.

  • If attr is installed, tags/sensitivity are stored in xattrs.

Actionable steps and real-world examples

1) Start with local-only privacy

  • Install ollama, pull llama3:8b.

  • Run the script with AI_BACKEND=ollama:

AI_BACKEND=ollama OLLAMA_MODEL=llama3:8b ./ai_classify.sh -n ~/Downloads

  • Verify categories and tags, then rerun without -n.

  • Great for laptops handling confidential documents.

2) Add OCR and DOCX for broader coverage

  • Ensure tesseract-ocr and docx2txt are installed.

  • The script will extract text from scans and Word docs automatically.

  • Example: scanned receipts now land under receipts with tags like ["flight","hotel"].

3) Use a cloud model for higher accuracy

  • Set:
export OPENAI_API_KEY="sk-..."
export OPENAI_MODEL="gpt-4o-mini"
AI_BACKEND=openai ./ai_classify.sh -n ~/inbox
  • Cloud models often provide better classification on messy content (e.g., invoices vs. quotes).

4) Enforce your taxonomy

  • Update the ALLOWED_CATEGORIES list in the script to match your org (e.g., ["sops","tickets","design","sre","compliance"]).

  • The model will choose “other” if nothing fits, keeping your tree clean.

5) Audit-friendly logs and xattrs

  • The CSV log becomes your audit trail.

  • With attr, tags persist with the file; tools like getfattr -d file.pdf show them.

  • Combine with ripgrep for rule-based fallbacks:

rg -n --json 'invoice|total due|VAT' classified/finance

Real-world wins:

  • Finance: PDFs named “scan0001.pdf” get classified as invoices or receipts with vendor tags.

  • Engineering: .log and .txt get split into logs vs code with service names as tags.

  • HR/Legal: Contracts and resumes surface under legal / hr with sensitivity “confidential”.

  • Photos: OCR catches embedded text (“boarding pass”, “ID”) and routes to travel or personal.

Tips and extensions

  • Rate limits: For large folders, throttle with find ... | xargs -0 -n 10 -P 2 ./ai_classify.sh by adapting the script into a function or batch.

  • Cost control: Use MAX_BYTES to keep prompts small; prefer local models for huge archives.

  • Safer moves: Run with -n first, then on the log inspect outliers (low confidence).

  • Renames only: Comment out mv and just use xattrs + logs if you don’t want to move files.

  • Additional extractors: Add handlers for .eml (use ripmime), spreadsheets (ssconvert or in2csv), or audio (pipe transcripts from Whisper if installed).

Conclusion and next steps

You don’t need a heavyweight DMS to tame your filesystem. A careful mix of standard Linux tooling and an LLM backend can classify, tag, and file your documents semantically—all from Bash.

Next steps:

  • Install the prerequisites for your distro (apt/dnf/zypper above).

  • Choose your backend: local ollama or an OpenAI-compatible API.

  • Run the script with -n, adjust categories, then organize for real.

Got a unique taxonomy or tricky file types? Extend the extractors and category list—then let your shell do the heavy lifting.

Related posts