Posted on
• Artificial Intelligence

Artificial Intelligence Desktop Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Desktop Automation with Bash: Turn Natural Language into Clicks and Keystrokes

Ever felt you spend more time clicking menus and typing the same things than actually solving problems? Imagine telling your Linux desktop, “Open Firefox, search for ‘xdotool manual’, and bookmark the first result,” and having it just happen. That’s the promise of AI-powered desktop automation—gluing lightweight Linux tools to an AI “planner” that translates your intent into deterministic actions.

This article explains why AI desktop automation is worth your time, what building blocks you need, and gives you 3–5 concrete Bash examples to start driving your desktop with an AI co-pilot—locally or via an API—plus OCR for screen-aware clicks.

Note: Most examples target X11 (xdotool/wmctrl). On Wayland, see the Wayland note below.

Why AI desktop automation is a valid (and powerful) approach

  • It turns intent into actions: LLMs can map “what you want” into command sequences. Bash executes them quickly and repeatably.

  • It’s composable: Unix tools do one thing well (window focus, keystrokes, OCR, HTTP). AI just orchestrates them.

  • Privacy-flexible: Use local models (e.g., Ollama) for sensitive tasks; use a cloud API when you need better reasoning.

  • It scales from quick wins to RPA: Automate one repetitive 10-second task today. Chain them into robust routines tomorrow.

Prerequisites: Install the toolbox

These utilities let Bash see and drive your desktop, plus talk to AI and do OCR.

  • xdotool: keyboard/mouse automation (X11)

  • wmctrl: window focus and manipulation (X11)

  • scrot: screenshots

  • tesseract + English data: OCR with bounding boxes

  • ImageMagick: image preprocessing for OCR (optional but useful)

  • jq + curl: JSON + HTTP for AI calls

  • inotify-tools: watch folders to auto-trigger scripts

  • xclip: clipboard integration

Install with your package manager:

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y xdotool wmctrl scrot tesseract-ocr tesseract-ocr-eng imagemagick jq curl inotify-tools xclip

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y xdotool wmctrl scrot tesseract tesseract-langpack-eng ImageMagick jq curl inotify-tools xclip

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y xdotool wmctrl scrot tesseract ImageMagick jq curl inotify-tools xclip
# If OCR lacks English data, try:
# sudo zypper install tesseract-ocr-traineddata-english
# If not found, discover your repo’s name with:
# zypper se -i -v tesseract | grep -i data

Optional: a local LLM with Ollama (works across distros):

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1

Wayland note:

  • xdotool/wmctrl primarily work under X11. On Wayland (GNOME/KDE defaults), try compositor-specific tools (e.g., wtype/ydotool/swaymsg/wlrctl), or log into an X11 session for these recipes. Package names vary by distro.

1) Warm-up: Drive apps with xdotool + wmctrl

Let’s focus Firefox, jump to the address bar, search for a query, and hit Enter—all from Bash.

#!/usr/bin/env bash
set -euo pipefail

QUERY="${*:-xdotool site:man7.org}"

# Activate a Firefox window; adjust title/class if needed
if wmctrl -x -a "firefox.Firefox"; then
  :
else
  # Launch if not running
  firefox & disown
  # Wait for the window to appear
  sleep 2
fi

# Ensure a Firefox window is focused
wmctrl -x -a "firefox.Firefox"

# Send keystrokes: Ctrl+L to focus URL bar, type query, press Return
xdotool key --clearmodifiers ctrl+l
xdotool type --delay 5 --clearmodifiers "$QUERY"
xdotool key --clearmodifiers Return

Save as firefox_search.sh, chmod +x firefox_search.sh, then:

./firefox_search.sh "xdotool windowactivate example"

Tips:

  • Use xdotool search --name "Title" to target specific windows.

  • Add small delays (--delay 5 or sleep 0.2) for flaky apps.

  • Prefer deterministic selectors (window class with wmctrl -lx) over titles when possible.

2) Make it “smart”: Use an LLM to plan actions, then execute them

Here we’ll ask an LLM to turn an instruction into a list of shell commands (xdotool/wmctrl), parse with jq, and run safely.

Local model with Ollama:

#!/usr/bin/env bash
set -euo pipefail

INSTRUCTION="${*:-Open Firefox, search for \"bash cheat sheet\", and bookmark the page.}"

read -r -d '' SYSTEM <<'EOF'
You are a planner that outputs only JSON. Given a user instruction about desktop tasks,
return: {"steps": ["command1", "command2", ...]}
Rules:

- Use wmctrl to focus apps, xdotool for keys/typing.

- Be conservative, deterministic, and avoid destructive actions.

- Use short pauses via 'sleep 0.2' if needed between steps.

- Only output JSON. No explanations.
EOF

PAYLOAD=$(jq -n --arg sys "$SYSTEM" --arg user "$INSTRUCTION" '{
  model: "llama3.1",
  prompt: ($sys + "\n\nUser: " + $user + "\nJSON:"),
  stream: false
}')

JSON=$(curl -s http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "$PAYLOAD" | jq -r '.response')
echo "LLM raw:" >&2
echo "$JSON" >&2

mapfile -t STEPS < <(printf '%s' "$JSON" | jq -r '.steps[]' 2>/dev/null || true)

if [ "${#STEPS[@]}" -eq 0 ]; then
  echo "No steps returned. Full response:" >&2
  printf '%s\n' "$JSON" >&2
  exit 1
fi

echo "Executing steps:"
for cmd in "${STEPS[@]}"; do
  echo "+ $cmd"
  bash -lc "$cmd"
done

Cloud API alternative (example outline for OpenAI’s Chat Completions):

# Requires: export OPENAI_API_KEY=...
MODEL=${MODEL:-"gpt-4o-mini"}
JSON=$(curl -s https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg m "$MODEL" --arg sys "$SYSTEM" --arg u "$INSTRUCTION" '{
        model: $m,
        response_format: { type:"json_object" },
        messages: [{role:"system", content:$sys}, {role:"user", content:$u}]
      }')" \
  | jq -r '.choices[0].message.content')

Safety ideas:

  • Run in “dry-run” mode first: print commands, ask Y/n.

  • Allow-list: reject commands not matching ^(xdotool|wmctrl|sleep|echo)\b.

  • Keep explicit scopes: only operate on specific windows/classes.

3) See your screen: OCR-driven clicks with Tesseract

Want to click a button when you only know the text on it? Screenshot, OCR to TSV (with coordinates), find the text, then move and click.

#!/usr/bin/env bash
set -euo pipefail

TARGET_TEXT="${1:-Settings}"
TMP_IMG="$(mktemp --suffix=.png)"

# Fullscreen screenshot
scrot "$TMP_IMG"

# Optional preprocessing for better OCR:
# convert "$TMP_IMG" -colorspace Gray -threshold 60% "$TMP_IMG"

# Tesseract to TSV (needs English trained data installed)
TSV=$(tesseract "$TMP_IMG" stdout -l eng tsv 2>/dev/null || true)

# Find the closest match (case-insensitive). Adjust matching as needed.
# TSV columns: level page block par line word left top width height conf text
read -r L T W H <<<"$(awk -v pat="$(echo "$TARGET_TEXT" | tr '[:upper:]' '[:lower:]')" -F'\t' '
  BEGIN{IGNORECASE=1}
  NR>1 && $0 ~ pat && $10 != "" {
    # choose the first match with reasonable confidence
    if ($9+0 >= 60) { print $7, $8, $9, $10; exit }
  }' <<< "$TSV")"

if [ -z "${L:-}" ]; then
  echo "Could not locate \"$TARGET_TEXT\" on screen."
  rm -f "$TMP_IMG"
  exit 1
fi

CX=$((L + W/2))
CY=$((T + H/2))
echo "Clicking at $CX,$CY for \"$TARGET_TEXT\""
xdotool mousemove "$CX" "$CY" click 1

rm -f "$TMP_IMG"

Usage:

./click_by_text.sh "Settings"

Notes:

  • OCR is imperfect—preprocess with ImageMagick (grayscale/threshold) to improve results.

  • On HiDPI or multiple monitors, coordinates should still align on X11; Wayland varies by compositor.

  • If detection is noisy, use stricter matching, or filter by region (crop before OCR).

4) Automate on events: OCR screenshots to clipboard (inotify)

Automatically OCR new screenshots and copy the text to your clipboard.

#!/usr/bin/env bash
set -euo pipefail

WATCH_DIR="${1:-$HOME/Pictures/Screenshots}"

echo "Watching $WATCH_DIR for new images..."
inotifywait -m -e close_write --format '%w%f' "$WATCH_DIR" | while read -r FILE; do
  case "$FILE" in
    *.png|*.jpg|*.jpeg)
      echo "OCR -> $FILE"
      TEXT="$(tesseract "$FILE" stdout -l eng 2>/dev/null || true)"
      if [ -n "$TEXT" ]; then
        printf '%s' "$TEXT" | xclip -selection clipboard
        echo "Copied OCR text to clipboard."
      else
        echo "No text found."
      fi
      ;;
  esac
done

Run it once and forget it:

./watch_ocr.sh  # or ./watch_ocr.sh /path/to/screenshots

Real-world patterns you can build next

  • Email triage: Focus mail client window, filter unread, expand first conversation, archive or label based on an LLM summary.

  • Cloud console tasks: Use OCR + planner to navigate web UIs without brittle coordinates.

  • Project dashboards: One command to open multiple apps/URLs, split terminals, and prepare a session.

Wayland and security notes

  • Wayland: xdotool/wmctrl are X11-centric. On Wayland, choose compositor-appropriate tools (e.g., wtype/ydotool/swaymsg). Many distros offer an “Xorg” session if you need full automation quickly.

  • Safety: Be careful sending screenshots or text to cloud APIs. Prefer local LLMs for sensitive data. Use allow-lists and dry-runs when executing AI-generated commands. Keep automation scoped to specific windows/classes.

Conclusion and next steps (CTA)

Pick a single repetitive task you do daily. Script it with xdotool/wmctrl. Then add AI as a planner for the ambiguous part (“what should I click/type next?”). Layer in OCR when you need to target on-screen text. Keep iterating until your desktop feels like a teammate, not a timesink.

Your next step:

  • Install the toolbox (see commands above).

  • Choose one task and implement either:

    • A direct script with xdotool/wmctrl, or
    • The AI planner script to translate your instruction into steps.

  • Share your best trick or pain point—you’ll likely inspire the next great snippet.

Happy automating!

Related posts