Posted on
• Artificial Intelligence

Automatically Tag Files Using Artificial Intelligence

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Automatically Tag Files Using Artificial Intelligence (from Bash)

Ever stared at a giant folder full of PDFs, photos, and code and thought, “I’ll organize this later”? Later never comes. Tags do. In this post, you’ll build a simple, scriptable pipeline to automatically tag files with AI—directly from Bash—so you can find what you need when you need it.

We’ll cover why it’s worth doing, how to run it with either a local model (Ollama) or a cloud API, and give you a drop-in script you can adapt. You’ll also get distro-specific installation commands for everything we use.

Why auto-tagging is worth it

  • Findability: Tags transform chaotic folders into search-friendly datasets (“research”, “invoice-2024”, “golang”, “photography”).

  • Cross-format: PDFs, notes, code, and images can share contextual tags, making multi-format research trivial.

  • Reproducible: A scriptable tagging pipeline lets you re-run tagging for new or changed files.

  • Local or cloud: Run a small local LLM for privacy, or a cloud LLM for higher accuracy—same Bash interface.

What you’ll build

  • A Bash script that:

    • Extracts text/metadata from files (PDFs, text/code, common image EXIF).
    • Sends a concise summary to an AI model and asks for 3–5 relevant tags in JSON.
    • Applies tags using:
    • Extended attributes (xattr via setfattr), or
    • “Tag folders” (symlink-based tag directories).

  • Works with:

    • OpenAI-compatible HTTP API (via curl).
    • Local models via Ollama.

You can extend it to other file types or customize prompts later.

Prerequisites and installation

We’ll use jq, ripgrep, poppler (for PDFs), attr (for xattrs), and optionally exiftool for images.

Install these on your distro:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y jq ripgrep poppler-utils attr libimage-exiftool-perl

  • Fedora/RHEL/Rocky (dnf):

    sudo dnf install -y jq ripgrep poppler-utils attr perl-Image-ExifTool

  • openSUSE Leap/Tumbleweed (zypper):

    sudo zypper install -y jq ripgrep poppler-tools attr perl-Image-ExifTool

Optional (for parallel speed-ups later):

  • apt:

    sudo apt install -y parallel

  • dnf:

    sudo dnf install -y parallel

  • zypper:

    sudo zypper install -y parallel

Choose your AI backend

  • Cloud (OpenAI-compatible):

    • Set environment variables:
    export OPENAI_API_KEY="sk-..."
export AI_PROVIDER="openai"
export AI_MODEL="gpt-4o-mini"   # or any compatible chat/completions model

  • Local (Ollama):

    • Install Ollama:
    curl -fsSL https://ollama.com/install.sh | sh
    • Pull a model (example):
    ollama pull llama3.1:8b
    • Set environment variables:
    export AI_PROVIDER="ollama"
export AI_MODEL="llama3.1:8b"

Tip: You can also use other OpenAI-compatible servers by setting a custom base:

export AI_API_BASE="https://your-openai-compatible-endpoint/v1"

The script: auto-tag.sh

Save this as auto-tag.sh and make it executable (chmod +x auto-tag.sh). It supports extended-attributes or tag-folder style tagging and aims to extract a compact, privacy-aware snippet to minimize tokens.

#!/usr/bin/env bash
set -euo pipefail

# Auto Tag Files Using AI
# Dependencies: bash, jq, ripgrep, poppler-utils/poppler-tools, attr, (optional) exiftool, curl, ollama (optional)

# Usage:
#   ./auto-tag.sh /path/to/dir --extensions "pdf,md,txt,jpg,jpeg,png" --backend xattr
#   ./auto-tag.sh /path/to/dir --extensions "pdf,md,txt" --backend tagsdir --tagsdir "$HOME/Tags"
#
# Env:
#   AI_PROVIDER: openai|ollama (default: openai if OPENAI_API_KEY is set, else ollama)
#   AI_MODEL: model name (e.g., gpt-4o-mini, llama3.1:8b)
#   OPENAI_API_KEY: required if AI_PROVIDER=openai
#   AI_API_BASE: OpenAI-compatible base URL (default: https://api.openai.com/v1)
#
# Notes:
# - Extended attributes require the 'attr' package. Tags saved under user.tags (comma-separated).
# - Tag folders create symlinks: $TAGS_DIR/<tag>/<unique-symlink-to-file>

# Defaults
DIR=""
EXTENSIONS="pdf,md,txt,org,rst,log,markdown,json,yml,yaml,py,sh,js,ts,java,go,rs,c,cpp,html,css,xml,jpg,jpeg,png,tif,tiff,heic"
BACKEND="xattr"  # xattr | tagsdir
TAGS_DIR="${HOME}/Tags"
NUM_TAGS=5
DRY_RUN=0
MAX_CHARS=40000

# Simple arg parse
while [[ $# -gt 0 ]]; do
  case "$1" in
    --extensions) EXTENSIONS="$2"; shift 2;;
    --backend)    BACKEND="$2"; shift 2;;
    --tagsdir)    TAGS_DIR="$2"; shift 2;;
    --num-tags)   NUM_TAGS="$2"; shift 2;;
    --dry-run)    DRY_RUN=1; shift 1;;
    -h|--help)
      echo "Usage: $0 DIR [--extensions 'pdf,md,txt'] [--backend xattr|tagsdir] [--tagsdir PATH] [--num-tags N] [--dry-run]"
      exit 0
      ;;
    *) DIR="$1"; shift 1;;
  esac
done

if [[ -z "${DIR}" ]]; then
  echo "Error: missing DIR. Try: $0 /path/to/dir" >&2
  exit 1
fi

if [[ ! -d "$DIR" ]]; then
  echo "Error: DIR not found: $DIR" >&2
  exit 1
fi

IFS=, read -r -a EXT_ARR <<<"$EXTENSIONS"

# Detect provider if not set
AI_PROVIDER="${AI_PROVIDER:-}"
if [[ -z "${AI_PROVIDER}" ]]; then
  if [[ -n "${OPENAI_API_KEY:-}" ]]; then
    AI_PROVIDER="openai"
  else
    AI_PROVIDER="ollama"
  fi
fi

AI_MODEL="${AI_MODEL:-gpt-4o-mini}"
AI_API_BASE="${AI_API_BASE:-https://api.openai.com/v1}"

# Ensure tags dir if using tagsdir
if [[ "$BACKEND" == "tagsdir" ]]; then
  mkdir -p "$TAGS_DIR"
fi

# Helpers
lower_ext() {
  local f="$1"
  local e="${f##*.}"
  echo "${e,,}"
}

is_wanted_ext() {
  local e="$1"
  for want in "${EXT_ARR[@]}"; do
    if [[ "$e" == "${want,,}" ]]; then return 0; fi
  done
  return 1
}

extract_text() {
  # Emits a concise text summary for the file to stdout
  local f="$1"
  local e
  e="$(lower_ext "$f")"
  case "$e" in
    pdf)
      if command -v pdftotext >/dev/null 2>&1; then
        pdftotext -q "$f" - 2>/dev/null | head -c "$MAX_CHARS"
      fi
      ;;
    md|markdown|txt|org|rst|log|json|yml|yaml|py|sh|js|ts|java|go|rs|c|h|hpp|cpp|html|css|xml)
      # Prefer first ~1200 lines or MAX_CHARS, whichever comes first
      sed -n '1,1200p' "$f" | head -c "$MAX_CHARS"
      ;;
    jpg|jpeg|png|tif|tiff|heic)
      if command -v exiftool >/dev/null 2>&1; then
        exiftool -s -s -s \
          -Title -Description -Subject -Keywords -Make -Model -Software -CreateDate -GPSLatitude -GPSLongitude \
          "$f" 2>/dev/null
      fi
      ;;
    *)
      # Unknown/binary types: skip
      ;;
  esac
}

ai_json_tags() {
  # Input: prompt content via stdin. Output: one tag per line to stdout.
  local content
  content="$(cat)"

  local sys_prompt="You are a file-tagging assistant. Based on filename and an excerpt or metadata, return a compact JSON object with a single top-level key 'tags' containing ${NUM_TAGS} highly relevant, short, lowercase, hyphenated tags (no punctuation, no spaces except hyphens). Example: {\"tags\":[\"project-planning\",\"golang\",\"invoice-2024\"]}. Only output valid JSON."

  if [[ "$AI_PROVIDER" == "openai" ]]; then
    if [[ -z "${OPENAI_API_KEY:-}" ]]; then
      echo "Error: OPENAI_API_KEY not set" >&2
      return 1
    fi
    local payload
    payload="$(jq -n --arg model "$AI_MODEL" --arg sys "$sys_prompt" --arg usr "$content" \
      '{model:$model, messages:[{role:"system",content:$sys},{role:"user",content:$usr}], temperature:0.2}')"

    curl -sS "${AI_API_BASE}/chat/completions" \
      -H "Authorization: Bearer ${OPENAI_API_KEY}" \
      -H "Content-Type: application/json" \
      -d "$payload" \
    | jq -r '.choices[0].message.content' \
    | jq -r 'try fromjson | .tags // empty | .[]' 2>/dev/null

  elif [[ "$AI_PROVIDER" == "ollama" ]]; then
    # Assumes ollama is running and 'AI_MODEL' is available
    if ! command -v ollama >/dev/null 2>&1; then
      echo "Error: ollama not found. Install and/or set AI_PROVIDER=openai." >&2
      return 1
    fi
    # We prompt the model to return JSON only.
    local full_prompt="${sys_prompt}\n\nCONTENT:\n${content}\n\nReturn only JSON."
    # ollama run prints plain text by default (we expect JSON as text)
    ollama run "$AI_MODEL" "$full_prompt" \
    | jq -r 'try fromjson | .tags // empty | .[]' 2>/dev/null
  else
    echo "Error: unknown AI_PROVIDER: $AI_PROVIDER" >&2
    return 1
  fi
}

apply_tags_xattr() {
  local f="$1"; shift
  local tags_csv
  tags_csv="$(IFS=, ; echo "$*")"
  if [[ "$DRY_RUN" -eq 1 ]]; then
    echo "[DRY] setfattr user.tags='$tags_csv'  --  $f"
    return 0
  fi
  # store combined tags as comma-separated value
  setfattr -n user.tags -v "$tags_csv" -- "$f" 2>/dev/null || true
}

apply_tags_tagsdir() {
  local f="$1"; shift
  local base rel hash
  base="$(basename "$f")"
  # Hash to ensure unique symlink names across identical basenames
  hash="$(printf '%s' "$f" | sha256sum | cut -c1-12)"
  for t in "$@"; do
    local d="${TAGS_DIR}/${t}"
    mkdir -p "$d"
    local link="${d}/${base}.${hash}"
    if [[ "$DRY_RUN" -eq 1 ]]; then
      echo "[DRY] ln -s \"$f\" \"$link\""
    else
      ln -sf "$f" "$link"
    fi
  done
}

process_file() {
  local f="$1"
  local e
  e="$(lower_ext "$f")"
  if ! is_wanted_ext "$e"; then
    return 0
  fi

  # Build context for prompt
  local size mtime excerpt
  size="$(stat -c%s "$f" 2>/dev/null || stat -f%z "$f")"
  mtime="$(date -d "@$(stat -c%Y "$f" 2>/dev/null || stat -f%m "$f")" +'%Y-%m-%d %H:%M:%S' 2>/dev/null || true)"
  excerpt="$(extract_text "$f" | tr -d '\000' | head -c "$MAX_CHARS")"

  # If we have no excerpt/metadata at all, skip
  if [[ -z "${excerpt// }" ]]; then
    echo "Skip (no extractable text/metadata): $f"
    return 0
  fi

  # Create final content passed to AI
  local content
  content=$(cat <<EOF
Filename: $(basename "$f")
Path: $f
Extension: $e
Size: $size bytes
Modified: $mtime

Excerpt/Metadata:
${excerpt}
EOF
)

  mapfile -t tags < <(printf '%s' "$content" | ai_json_tags | sed 's/[^a-z0-9\-]//g' | sed 's/--*/-/g' | sed 's/^-//; s/-$//' | awk 'length>0' | head -n "$NUM_TAGS" | sort -u)

  if [[ "${#tags[@]}" -eq 0 ]]; then
    echo "No tags generated for: $f"
    return 0
  fi

  echo "Tags for $f => ${tags[*]}"

  if [[ "$BACKEND" == "xattr" ]]; then
    apply_tags_xattr "$f" "${tags[@]}"
  else
    apply_tags_tagsdir "$f" "${tags[@]}"
  fi
}

export -f lower_ext is_wanted_ext extract_text ai_json_tags apply_tags_xattr apply_tags_tagsdir process_file
export AI_PROVIDER AI_MODEL AI_API_BASE OPENAI_API_KEY BACKEND TAGS_DIR NUM_TAGS DRY_RUN MAX_CHARS
export EXTENSIONS

# Find files with ripgrep (fast, respects .gitignore unless -uu)
rg -uu --files "$DIR" \
  | while IFS= read -r f; do
      process_file "$f"
    done

How to run it (examples)

1) Tag a research folder (PDFs + Markdown) with extended attributes:

export AI_PROVIDER=openai
export OPENAI_API_KEY="sk-..."          # your API key
export AI_MODEL="gpt-4o-mini"
./auto-tag.sh ~/Research --extensions "pdf,md,markdown,txt" --backend xattr

Now user.tags on each file contains a comma-separated list. You can query with:

getfattr -n user.tags --absolute-names ~/Research/some.pdf

2) Tag a photo archive locally (no cloud) using tag folders:

export AI_PROVIDER=ollama
export AI_MODEL="llama3.1:8b"
./auto-tag.sh ~/Photos --extensions "jpg,jpeg,png" --backend tagsdir --tagsdir "$HOME/Tags"

You’ll get directories like ~/Tags/travel/ populated with symlinks to matching photos.

3) Dry-run first (recommended):

./auto-tag.sh ~/Docs --extensions "pdf,txt,md" --backend xattr --dry-run

4) Speed it up with GNU parallel (optional, advanced):

rg -uu --files ~/Docs | parallel -P 4 ./auto-tag.sh {}

Note: If you parallelize, wrap AI calls with basic rate limiting or caching to avoid hitting API limits.

Real-world use cases

  • Research library: Tag PDFs as “machine-learning”, “bayesian-inference”, “paper-2021” to unify academic papers and notes across formats.

  • Photo management: Use EXIF + AI to infer “landscape”, “night-shot”, “travel-japan”, even if you never added manual keywords.

  • Codebase indexing: Tag files “kubernetes-yaml”, “golang”, “api-client”, “db-migration” to make grep searches more targeted.

Tips, privacy, and maintenance

  • Privacy: For sensitive files, prefer local models (Ollama) or a self-hosted OpenAI-compatible server. Trim excerpts aggressively.

  • Cost and rate limits: Limit MAX_CHARS, set NUM_TAGS to 3–5, and batch over time (cron).

  • Caching: Store a sidecar file (e.g., .tags.ai.json) keyed by file path + mtime + size to skip reprocessing unchanged files.

  • Consistency: Constrain the model with clear instructions—lowercase, hyphenated, 3–5 tags—to keep your tag vocabulary clean.

  • Recovery: If tag folders feel messy, you can delete and regenerate; extended attributes can be cleared with:

    setfattr -x user.tags /path/to/file

Conclusion and next steps

You now have a practical, distro-friendly Bash workflow to auto-tag files with AI. It’s fast, scriptable, and flexible—run locally with Ollama for privacy, or use a cloud API for more power. From here:

  • Start small: run a dry-run on a single folder.

  • Tune: adjust the prompt, file types, and MAX_CHARS.

  • Scale: add caching and a cron job for new/modified files.

  • Integrate: wire tags into your search/indexing tools or file manager.

If this saved you hours of manual tagging, share it with a teammate—or extend the script and open-source your version. Happy tagging!

Related posts