Posted on
• Artificial Intelligence

Artificial Intelligence Bash Logging and Error Handling

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI‑Powered Bash Logging and Error Handling: Turn 2 A.M. Incidents into Actionable Insights

Ever been paged for a broken Bash job and found a single line of cryptic output to work with? Traditional logs tell you what failed, not why. By combining robust Bash patterns with a lightweight dose of AI, you can capture better context, summarize root causes, and propose fixes automatically—before you even open your laptop.

This article shows you how to:

  • Build rock‑solid logging and error handling in Bash

  • Emit structured, machine‑readable logs

  • Automatically summarize failures with an AI endpoint

  • Add retries and send lightweight alerts

You’ll get copy‑pasteable snippets and distro‑specific install commands.

Why this matters

  • Bash is everywhere: CI pipelines, cron jobs, container entrypoints, maintenance tasks.

  • But defaults are brittle: silent errors, swallowed failures in pipelines, and unstructured logs.

  • Adding structure (JSON logs, traps, error context) makes problems observable.

  • Adding AI on top turns “noise” into “next steps,” proposing likely root causes and fixes, and cutting time-to-resolution.

Prerequisites and installation

We’ll use:

  • curl for HTTP calls (AI, webhooks)

  • jq for JSON building/parsing

  • logger (from util-linux) for syslog/journald integration

Install with your package manager:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y curl jq util-linux

  • Fedora/RHEL/CentOS (dnf):

    sudo dnf install -y curl jq util-linux

  • openSUSE (zypper):

    sudo zypper refresh
sudo zypper install -y curl jq util-linux

Note: journald/systemd are typically present by default on modern distros.

1) Build a solid Bash foundation: strict mode, traps, and a single log function

Start every serious script with strict error handling and a consistent logger. This alone catches a large class of “works on my machine” issues.

#!/usr/bin/env bash
set -Eeuo pipefail

# Configuration (override via environment variables)
LOG_FILE=${LOG_FILE:-"$PWD/myjob.log"}
LOG_TAG=${LOG_TAG:-myjob}

# Optional AI + alerting
AI_ENDPOINT=${AI_ENDPOINT:-https://api.openai.com/v1/responses}
AI_MODEL=${AI_MODEL:-gpt-4.1-mini}
AI_API_KEY=${AI_API_KEY:-""}
SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL:-""}

mkdir -p "$(dirname "$LOG_FILE")"

ts() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

log() {
  local level=${1:-INFO}; shift || true
  local msg="$*"
  # Console + file + syslog (best-effort)
  printf "%s [%s] %s\n" "$(ts)" "$level" "$msg" \
    | tee -a "$LOG_FILE" \
    | logger -t "$LOG_TAG" -p "user.${level,,}" 2>/dev/null || true
}

  • set -Eeuo pipefail enables early, explicit failure (including pipeline errors).

  • logger ships with util-linux; it forwards to syslog/journald so operations teams can use journalctl/centralized logging.

  • LOG_FILE defaults to the current directory to avoid permission headaches.

2) Emit structured, machine‑readable logs (JSON)

Structured logs make it easy to query, alert, and feed AI. Add a JSON logger that captures command, cwd, pid, and exit codes.

log_json() {
  local level=${1:-INFO}; shift || true
  local message="$1"; shift || true
  local exit_code="${1:-0}"

  jq -c -n \
    --arg t "$(ts)" \
    --arg level "$level" \
    --arg msg "$message" \
    --arg cmd "${BASH_COMMAND:-}" \
    --arg cwd "$PWD" \
    --arg script "${BASH_SOURCE[0]}" \
    --arg pid "$$" \
    --arg exit "$exit_code" \
    '{time:$t, level:$level, message:$msg,
      context:{cmd:$cmd, cwd:$cwd, script:$script, pid:($pid|tonumber), exit:($exit|tonumber)}}' \
    | tee -a "$LOG_FILE" \
    | logger -t "$LOG_TAG" 2>/dev/null || true
}

Query examples:

  • Recent errors in file logs:

    jq -r 'select(.level=="ERROR") | .time + " " + .message' myjob.log

  • From journald by tag:

    journalctl -t myjob -o json-pretty | jq -r '.MESSAGE'

3) Wire in AI summaries on failure (with redaction and retries)

On error, trap the failure, capture context, and ask an AI endpoint to propose likely root causes and next steps. Keep it opt‑in via an API key and redact sensitive values.

redact() {
  # Best-effort scrubbing; extend with your org’s patterns
  sed -E \
    -e 's/(API[_-]?KEY|TOKEN|PASSWORD)=\S+/\1=***REDACTED***/g' \
    -e 's/(Authorization: Bearer) [A-Za-z0-9._-]+/\1 ***REDACTED***/g'
}

retry() {
  local max=${1:-3}; shift
  local delay=${1:-1}; shift
  local n=1
  until "$@"; do
    local rc=$?
    if (( n >= max )); then
      log ERROR "Command failed after $n attempts (rc=$rc): $*"
      return $rc
    fi
    log WARN "Attempt $n failed (rc=$rc). Retrying in ${delay}s: $*"
    sleep "$delay"
    delay=$(( delay * 2 ))
    n=$(( n + 1 ))
  done
}

ai_summarize() {
  local err_msg="$(printf "%s" "$1" | redact)"
  local context="$(printf "%s" "$2" | redact)"

  [[ -n "$AI_API_KEY" ]] || { log WARN "AI_API_KEY not set; skipping AI summary."; return 0; }

  local prompt="You are a Linux/Bash SRE assistant. Summarize a probable root cause and propose 1–3 concrete next actions.
Error:
$err_msg

Recent log tail:
$context"

  local payload
  payload=$(jq -n --arg model "$AI_MODEL" --arg input "$prompt" '{model:$model, input:$input, temperature:0.2}')

  local resp summary
  resp=$(curl -sS -H "Authorization: Bearer $AI_API_KEY" -H "Content-Type: application/json" \
              -d "$payload" "$AI_ENDPOINT" || true)
  summary=$(jq -r 'try .output_text // .choices[0].message.content // empty' <<<"$resp")

  if [[ -n "$summary" ]]; then
    log ERROR "AI summary: $summary"
    printf "%s\n" "$summary" >> "${LOG_FILE%.log}.ai.log" || true
    if [[ -n "$SLACK_WEBHOOK_URL" ]]; then
      jq -n --arg text "*$LOG_TAG* AI summary:\n$summary" '{text:$text}' \
        | curl -sS -X POST -H "Content-type: application/json" --data @- "$SLACK_WEBHOOK_URL" >/dev/null || true
    fi
  else
    log WARN "No AI summary parsed. Raw response: $resp"
  fi
}

on_err() {
  local rc=$?
  local cmd=${BASH_COMMAND:-}
  local err="Command: $cmd | Exit: $rc"
  local taillog
  taillog=$(tail -n 50 "$LOG_FILE" 2>/dev/null || true)
  log_json ERROR "Trap ERR" "$rc"
  ai_summarize "$err" "$taillog"
  exit "$rc"
}

trap on_err ERR
trap 'log INFO "Completed with exit code $?";' EXIT

How it works:

  • Any uncaught error triggers on_err (thanks to set -Eeuo pipefail).

  • We append structured context and tail the log.

  • We call an AI endpoint with a minimal, focused prompt.

  • If available, we also post the summary to a Slack Incoming Webhook.

Security note:

  • Do not send secrets to external services. The redact function is a guardrail; adapt it to your environment and prefer private/self‑hosted models for sensitive workloads.

4) A realistic main() with retries and a forced failure to test

Drop this in the same script to see the flow end-to-end.

main() {
  log INFO "Starting job..."
  log_json INFO "Structured block begins"

  # Example of a flaky call with retries
  retry 3 1 curl -fsS https://example.invalid/healthz

  log INFO "Doing a thing that might fail..."
  false  # Intentionally fail to test traps, summaries, and alerts
}

main "$@"

Run it:

bash ./myjob.sh

You should see:

  • Human‑readable logs in myjob.log

  • Structured JSON entries piped to journald (journalctl -t myjob)

  • An AI summary appended to myjob.ai.log (if AI_API_KEY is set)

Configure environment for AI and Slack:

export AI_API_KEY="your_api_key_here"
export AI_ENDPOINT="https://api.openai.com/v1/responses"   # or your self-hosted compatible endpoint
export AI_MODEL="gpt-4.1-mini"                             # or another available model
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"

Operational tips and real‑world patterns

  • Make logs discoverable:

    • File logs: predictable paths (e.g., /var/log/myteam/myjob.log)
    • Syslog/journald: consistent LOG_TAG and use journalctl -t myjob -o short-iso

  • Test your traps:

    • Simulate errors with false or a failing curl. Confirm AI summaries appear.

  • Keep prompts short and specific:

    • Feed only the last N lines of context. Long prompts are slower and cost more.

  • Prefer self‑hosted endpoints for sensitive data:

    • Point AI_ENDPOINT to an internal, OpenAI‑compatible gateway if needed.

  • CI/CD integration:

    • Export AI_API_KEY as a secret, run the same script in pipelines, and surface AI summaries as job annotations.

Troubleshooting

  • logger not found:

    • Install util-linux (see package commands above).

  • jq not found:

    • Install jq (see package commands above).

  • Permission denied writing logs:

    • Change LOG_FILE to a writable path (default uses $PWD).

  • No AI summary:

    • Check AI_API_KEY, AI_ENDPOINT reachability, and model name.
    • Inspect raw response in logs when parsing fails.

Conclusion and next steps

With a few functions and sane defaults, you’ve upgraded Bash from “best effort” to “observable and explainable.” The combination of strict mode, structured logs, and AI summaries makes root cause analysis faster and on‑call far less painful.

Your next steps: 1) Copy the template into your most brittle scripts. 2) Install curl, jq, and util-linux with your package manager. 3) Set AI_API_KEY and test the failure path. 4) Wire the script into cron or systemd and watch journalctl -t myjob. 5) Iterate on redaction rules and prompts to fit your environment.

If this helped, share the template with your team and standardize it as your org’s Bash skeleton.

Related posts