Posted on
• Artificial Intelligence

Artificial Intelligence Bash Error Handling

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Bash Error Handling: make your shell scripts fail smart

Ever had a cron job quietly die at 2 a.m. and leave you guessing? Bash is the glue of Linux automation, but its default error behavior can be vague, silent, or both. The fix isn’t just “add set -e”; it’s about making failures observable, actionable, and—optionally—AI-assisted.

In this guide you’ll:

  • Turn on strict, predictable error behavior

  • Capture rich context when things fail

  • Auto-retry the right way (with backoff)

  • Log errors in machine-readable form

  • Optionally ask a local AI for on-the-spot triage suggestions

By the end, your scripts will not only fail loudly—they’ll fail intelligently.

Why this matters

  • Bash still runs builds, deploys, and data jobs across CI/CD and servers.

  • Transient failures (network hiccups, rate limits) are normal; scripts must recover.

  • Root-causing production failures from a single line of stderr is painful.

  • Generative AI can summarize and hypothesize issues faster than a human at 3 a.m.

Good error handling reduces MTTR, improves reliability, and makes your future self grateful.

Prerequisites: tools you’ll need

We’ll use jq for JSON logging, ShellCheck for linting, Bats for tests, and curl for demos.

  • Debian/Ubuntu (apt):

    • sudo apt update && sudo apt install -y jq curl shellcheck bats

  • Fedora/RHEL/CentOS Stream (dnf):

    • sudo dnf install -y jq curl ShellCheck bats

  • openSUSE (zypper):

    • sudo zypper refresh && sudo zypper install jq curl ShellCheck bats

Optional local AI (Ollama):

  • curl -fsSL https://ollama.com/install.sh | sh

  • ollama pull llama3

Note: Ollama installs via its script (no apt/dnf/zypper package at this time).

Core patterns (copy-paste ready)

1) Turn on predictable failure with “strict mode”

#!/usr/bin/env bash
# ai-bash-template.sh
set -Eeuo pipefail
IFS=$'\n\t'

LOG_FILE=${LOG_FILE:-/tmp/ai-bash-errors.log}

cleanup() {
  # Clean up temp files, sockets, locks, etc.
  :
}

ai_suggest() {
  # Optional: require AI_SUGGEST=1 and a local model via Ollama
  [[ "${AI_SUGGEST:-0}" -eq 1 ]] || return 0
  command -v ollama >/dev/null 2>&1 || { echo "AI: Ollama not found, skipping." >&2; return 0; }
  local payload="${1:-}"
  # Avoid leaking secrets: redact before sending to any model.
  local prompt="You are a senior SRE. Analyze this Bash error event and suggest likely root causes and next steps, be concise:\n\n${payload}"
  echo -e "$prompt" | ollama run llama3 2>/dev/null | sed 's/^/AI: /'
}

on_error() {
  local exit_code=$?
  local line=${BASH_LINENO[0]:-0}
  local cmd=${BASH_COMMAND}
  local func=${FUNCNAME[1]:-MAIN}
  local src=${BASH_SOURCE[1]:-$0}
  local ts
  ts=$(date -Is)

  # Build a JSON error event
  local json
  json=$(jq -nc \
    --arg ts "$ts" \
    --arg script "$src" \
    --arg func "$func" \
    --arg line "${line}" \
    --arg cmd "$cmd" \
    --arg status "${exit_code}" \
    '{ts:$ts, script:$script, func:$func, line:($line|tonumber), cmd:$cmd, status:($status|tonumber)}')

  echo "$json" >> "$LOG_FILE"
  echo "ERROR: '$cmd' exited with $exit_code at $func:$line ($src) [ts=$ts]" >&2

  ai_suggest "$json" || true
}

trap cleanup EXIT
trap on_error ERR
trap 'echo "Interrupted"; exit 130' INT TERM

Why it works:

  • set -Eeuo pipefail ensures failures propagate and unset vars explode loudly.

  • trap on_error ERR captures failing commands with context (function, line, last command).

  • jq ensures logs are structured for machines and humans.

2) Retry transient failures with exponential backoff

retry() {
  # Usage: retry <max_attempts> <base_seconds> <command...>
  local -i max=${1:?max_attempts}; shift
  local -i base=${1:?base_delay}; shift
  local -i attempt=1

  until "$@"; do
    local rc=$?
    if (( attempt >= max )); then
      echo "Retry: giving up after $attempt attempts (last rc=$rc)" >&2
      return "$rc"
    fi
    local -i sleep_for=$(( base * 2 ** (attempt - 1) ))
    echo "Retry: attempt $attempt failed (rc=$rc), sleeping ${sleep_for}s..." >&2
    sleep "$sleep_for"
    ((attempt++))
  done
}

Why it works:

  • Backoff avoids hammering flaky endpoints.

  • Returns the real exit code for traps and callers.

3) Emit machine-readable logs for easy search and alerts

You already saw JSON logging in on_error. Send it to files, journald, or a log forwarder. With jq installed you can pretty-print or filter:

  • Show last 5 errors:

    • tail -n 5 /tmp/ai-bash-errors.log | jq

  • Filter by command:

    • jq 'select(.cmd|test("curl"))' /tmp/ai-bash-errors.log

Structured logs enable tooling: grep/jq, dashboards, or alerting on .status != 0.

4) Real-world example: robust JSON fetch

This example pulls a JSON API with retries, validates it, and benefits from our traps and logs.

main() {
  local url=${1:-"https://httpbin.org/json"}
  local out=${2:-"/tmp/demo.json"}

  echo "Fetching $url ..."
  retry 4 2 curl -fsS "$url" -o "$out"

  # Validate JSON structure
  jq -e . "$out" >/dev/null

  echo "OK: wrote valid JSON to $out"
}

main "$@"

  • On transient network issues it retries with backoff.

  • If curl or jq fail, on_error logs a JSON error event and prints a human-friendly message.

  • Set AI_SUGGEST=1 and have Ollama installed to see suggested next steps:

    • AI_SUGGEST=1 ./ai-bash-template.sh

5) Lint and test your error handling

  • Lint with ShellCheck (prevents many foot-guns):

    • shellcheck ai-bash-template.sh

  • Minimal tests with Bats (focus on error paths and retry behavior):

#!/usr/bin/env bats

# bats file: test/error_handling.bats
load 'test_helper/bats-support/load'
load 'test_helper/bats-assert/load'

@test "retry succeeds when command eventually works" {
  run bash -c '
    source ./ai-bash-template.sh
    i=0
    flaky(){ ((i++<2)) && return 1 || return 0; }
    retry 5 1 flaky
  '
  [ "$status" -eq 0 ]
}

@test "retry returns failure when max attempts exceeded" {
  run bash -c '
    source ./ai-bash-template.sh
    always_fail(){ return 7; }
    retry 3 1 always_fail
  '
  [ "$status" -eq 7 ]
}

Run:

  • bats test/error_handling.bats

Install if missing:

  • Debian/Ubuntu: sudo apt install -y shellcheck bats

  • Fedora/RHEL/CentOS Stream: sudo dnf install -y ShellCheck bats

  • openSUSE: sudo zypper install ShellCheck bats

Pro tips

  • Redact secrets in logs and AI prompts. Don’t ever send tokens or PII to a model.

  • Prefer [[ ... ]] over [ ... ] for safer Bash conditionals.

  • In pipelines that must not mask failures, rely on set -o pipefail or check each stage.

  • Use timeout for untrusted/external calls: timeout 30s curl ...

  • Keep traps tiny and reliable; avoid heavy work inside ERR handlers.

Conclusion and next steps

Your Bash scripts can fail smart: 1) Paste the template into a real script. 2) Install jq, ShellCheck, and Bats using apt/dnf/zypper. 3) Add retries around flaky operations. 4) Start logging JSON errors to a file and inspect with jq. 5) Optionally enable AI triage with Ollama for quick hypotheses.

Small improvements compound. Refactor one script today, wire in strict mode and error traps, and watch your mean time to recovery shrink.

Related posts