Posted on
• Artificial Intelligence

Artificial Intelligence Alert Fatigue Reduction

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Cut Through the Noise: Bash + AI to Reduce Alert Fatigue

You’re on call. Your phone won’t stop buzzing. Pages on “disk almost full” that have been “almost” for weeks, flaky health checks, and the same warning repeating across a fleet. By 2 a.m., it’s hard to tell what’s real. That’s alert fatigue—and it burns out teams and hides true incidents.

Here’s the good news: you can put a low-friction filter in front of your alerts that de-duplicates, throttles, and uses a tiny bit of AI to prioritize the stuff that matters. You don’t need to re-architect your stack. You can do it with Bash, a few standard CLI tools, and a simple curl call.

This post shows you how, step by step.

Why this works (and why now)

  • Modern stacks generate a firehose of alerts. Many are duplicates, context-free, or plain noisy.

  • LLMs are good at quick classification and summarization of unstructured text. A small, cheap model can rate “actionability” better than a rigid regex.

  • You can glue all this together with Bash: normalize messages, hash them for dedupe, apply a time window, then ask AI to rate priority and route accordingly—without buying a new platform.

What we’ll build

  • A tiny Bash pipeline that:
    • Normalizes alerts, fingerprints them, and throttles duplicates in a sliding window.
    • Optionally uses an LLM over HTTPS to rate actionability (low/medium/high/critical).
    • Sends high/critical to Slack immediately; queues low/medium into hourly digests.
    • Lets operators maintain a simple ignore list that actually sticks.

You can feed it from journalctl, syslog, or your existing alert feed.

Install the prerequisites

We’ll use jq (JSON), sqlite3 (state), curl (webhooks/API), and coreutils (sha256sum, date). Install with your distro’s package manager:

  • Debian/Ubuntu (apt):

    sudo apt update
sudo apt install -y jq sqlite3 curl coreutils

  • Fedora/RHEL/CentOS Stream (dnf):

    sudo dnf install -y jq sqlite curl coreutils

  • openSUSE/SLE (zypper):

    sudo zypper install -y jq sqlite3 curl coreutils

Create working directories:

sudo mkdir -p /var/lib/ai-alerts/spool
sudo chown -R "$USER":"$USER" /var/lib/ai-alerts

Optional: Configure a Slack Incoming Webhook and keep the URL handy. For AI, set an OpenAI-compatible endpoint and API key (you can start in DRY_RUN mode with no external calls).

Step 1 — The Bash reducer: normalize, fingerprint, throttle, and route

Save this as /usr/local/bin/ai_alert_reducer.sh and make it executable.

#!/usr/bin/env bash
set -euo pipefail

# Config
DB=${DB:-/var/lib/ai-alerts/alerts.db}
SPOOL_DIR=${SPOOL_DIR:-/var/lib/ai-alerts/spool}
WINDOW_SECONDS=${WINDOW_SECONDS:-600}        # throttle duplicates for 10 minutes
IGNORE_FILE=${IGNORE_FILE:-/var/lib/ai-alerts/ignore_patterns.txt}

# Routing
SLACK_WEBHOOK=${SLACK_WEBHOOK:-}            # set to your Slack Incoming Webhook
DRY_RUN=${DRY_RUN:-1}                       # 1 = don’t send, just log actions

# AI (OpenAI-compatible chat completions)
AI_API_URL=${AI_API_URL:-https://api.openai.com/v1/chat/completions}
AI_MODEL=${AI_MODEL:-gpt-4o-mini}           # pick a small, cheap, fast model
OPENAI_API_KEY=${OPENAI_API_KEY:-}          # set if using AI scoring

mkdir -p "$(dirname "$DB")" "$SPOOL_DIR"

# Init database
sqlite3 "$DB" <<'SQL'
CREATE TABLE IF NOT EXISTS alerts (
  fingerprint TEXT PRIMARY KEY,
  first_seen  INTEGER,
  last_seen   INTEGER,
  count       INTEGER,
  summary     TEXT,
  source      TEXT,
  instance    TEXT
);
SQL

sql_escape() { sed "s/'/''/g"; }

level_from_priority() {
  # Map common fields to a textual level
  local p=$1
  case "$p" in
    0|1|2|3|emerg|alert|crit|err|error|panic|fatal) echo "high" ;;
    4|warn|warning) echo "medium" ;;
    5|6|notice|info) echo "low" ;;
    7|debug) echo "low" ;;
    *) echo "low" ;;
  esac
}

ai_score() {
  # Returns: low|medium|high|critical (fallback to static mapping on error)
  local norm="$1" summary="$2" severity_hint="$3"
  if [[ -z "${OPENAI_API_KEY:-}" ]]; then
    echo "$severity_hint"; return 0
  fi
  local prompt
  prompt=$(
    printf '%s' "Rate the actionability of this Linux ops alert.
Return only one word: low, medium, high, or critical.

Summary: $summary

Normalized context: $norm"
  )
  local resp
  resp=$(curl -sS "$AI_API_URL" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg m "$AI_MODEL" --arg p "$prompt" \
        '{model:$m, temperature:0,
          messages:[
            {role:"system", content:"You are a pragmatic SRE triaging alerts. Output one word only."},
            {role:"user", content:$p}
          ]}')" || true)

  local out
  out=$(jq -r '.choices[0].message.content' <<<"$resp" 2>/dev/null | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z')
  case "$out" in
    low|medium|high|critical) echo "$out" ;;
    *) echo "$severity_hint" ;;
  esac
}

send_slack() {
  local text="$1"
  if [[ -z "${SLACK_WEBHOOK:-}" ]]; then
    echo "[SLACK] $text"; return 0
  fi
  if [[ "${DRY_RUN:-0}" = "1" ]]; then
    echo "[DRY-RUN SLACK] $text"; return 0
  fi
  curl -sS -X POST -H 'Content-type: application/json' \
    --data "$(jq -n --arg t "$text" '{text:$t}')" \
    "$SLACK_WEBHOOK" >/dev/null
}

queue_digest() {
  local ts fp source instance summary
  ts="$1"; fp="$2"; source="$3"; instance="$4"; summary="$5"
  local outfile="$SPOOL_DIR/digest-$(date +%Y%m%d%H).jsonl"
  jq -n --arg ts "$ts" --arg fp "$fp" --arg s "$source" --arg i "$instance" --arg su "$summary" \
     '{ts:($ts|tonumber),fp:$fp,source:$s,instance:$i,summary:$su}' >> "$outfile"
}

sanitize_for_fingerprint() {
  # Replace variable parts like long hex strings and big numbers to group duplicates
  sed -E 's/[0-9a-fA-F]{8,}/<id>/g; s/[0-9]{3,}/<num>/g'
}

# Optional redaction: put patterns (regex) one per line to drop known noise
touch "$IGNORE_FILE"

echo "ai_alert_reducer: starting, window=${WINDOW_SECONDS}s DRY_RUN=${DRY_RUN}"

# Read JSON lines from stdin
# Example feed:
#   journalctl -f -o json | /usr/local/bin/ai_alert_reducer.sh
#   tail -F /var/log/alerts.jsonl | /usr/local/bin/ai_alert_reducer.sh
while IFS= read -r json; do
  [[ -z "$json" ]] && continue

  # Extract common fields from several alert styles (systemd-journald, custom JSON, Alertmanager-like)
  summary=$(jq -r '(.MESSAGE // .annotations.summary // .summary // .msg // .message // "no-summary")' <<<"$json")
  source=$(jq -r '(.SYSLOG_IDENTIFIER // .source // .service // .labels.job // "unknown")' <<<"$json")
  instance=$(jq -r '(.HOSTNAME // .host // .instance // .labels.instance // "localhost")' <<<"$json")
  severity=$(jq -r '(.level // .severity // .labels.severity // .PRIORITY // "info")' <<<"$json" | tr '[:upper:]' '[:lower:]')
  sev_text=$(level_from_priority "$severity")

  # Ignore list (operator-maintained)
  if [[ -s "$IGNORE_FILE" ]] && echo "$summary" | grep -Eqi -f "$IGNORE_FILE"; then
    echo "ignored (pattern): $summary"
    continue
  fi

  # Build normalized string and fingerprint
  canon="$(printf '%s|%s|%s' "$source" "$instance" "$summary" | sanitize_for_fingerprint)"
  fp=$(printf '%s' "$canon" | sha256sum | awk '{print $1}')
  now=$(date +%s)

  # Create/Update state
  existing=$(sqlite3 -separator '|' "$DB" "SELECT last_seen, count FROM alerts WHERE fingerprint='$fp';" || true)
  if [[ -z "$existing" ]]; then
    esc_summary=$(printf '%s' "$summary" | sql_escape)
    esc_source=$(printf '%s' "$source" | sql_escape)
    esc_instance=$(printf '%s' "$instance" | sql_escape)
    sqlite3 "$DB" "INSERT INTO alerts (fingerprint, first_seen, last_seen, count, summary, source, instance)
                   VALUES ('$fp', $now, $now, 1, '$esc_summary', '$esc_source', '$esc_instance');"
    is_duplicate_within_window=0
  else
    last_seen=$(cut -d'|' -f1 <<<"$existing")
    count=$(cut -d'|' -f2 <<<"$existing")
    diff=$(( now - last_seen ))
    if (( diff < WINDOW_SECONDS )); then
      sqlite3 "$DB" "UPDATE alerts SET last_seen=$now, count=count+1 WHERE fingerprint='$fp';"
      is_duplicate_within_window=1
    else
      sqlite3 "$DB" "UPDATE alerts SET last_seen=$now, count=count+1 WHERE fingerprint='$fp';"
      is_duplicate_within_window=0
    fi
  fi

  if (( is_duplicate_within_window == 1 )); then
    echo "throttled: $summary"
    continue
  fi

  # Ask AI for a final actionability score (fallback to severity hint)
  final_score=$(ai_score "$canon" "$summary" "$sev_text")

  case "$final_score" in
    critical|high)
      send_slack "[${final_score^^}] ${source}/${instance}: ${summary}"
      ;;
    medium|low|*)
      queue_digest "$now" "$fp" "$source" "$instance" "$summary"
      echo "queued to digest: $summary"
      ;;
  esac
done

Make it executable:

sudo chmod +x /usr/local/bin/ai_alert_reducer.sh

Test in “shadow mode” (DRY_RUN=1):

journalctl -f -o json | DB=/var/lib/ai-alerts/alerts.db DRY_RUN=1 /usr/local/bin/ai_alert_reducer.sh

When ready to enable Slack and AI:

export SLACK_WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
export OPENAI_API_KEY="sk-..."
export AI_API_URL="https://api.openai.com/v1/chat/completions"
export AI_MODEL="gpt-4o-mini"
journalctl -f -o json | DRY_RUN=0 /usr/local/bin/ai_alert_reducer.sh

Security tip: If using a third-party AI API, ensure you’re not sending secrets or PII. You can add redaction rules before the AI call (e.g., replace IPs/emails/tokens with placeholders).

Step 2 — Hourly digests to shrink the noise floor

Batch low/medium alerts into a single summary. Save as /usr/local/bin/ai_alert_digest.sh:

#!/usr/bin/env bash
set -euo pipefail

SPOOL_DIR=${SPOOL_DIR:-/var/lib/ai-alerts/spool}
SLACK_WEBHOOK=${SLACK_WEBHOOK:-}
DRY_RUN=${DRY_RUN:-1}

hour=$(date +%Y%m%d%H -d '1 hour ago' 2>/dev/null || date -v-1H +%Y%m%d%H)
file="$SPOOL_DIR/digest-$hour.jsonl"

[[ ! -s "$file" ]] && { echo "no digest data for $hour"; exit 0; }

summary_json=$(jq -s '
  group_by(.fp)
  | map({count: length, sample: .[0]})
  | sort_by(-.count)
  | .[0:20]
' "$file")

text=$(
  printf "Hourly Alert Digest (%s)\n" "$hour"
  echo "$summary_json" | jq -r '.[] | "- (\(.count)x) \(.sample.source)/\(.sample.instance): \(.sample.summary)"'
)

if [[ -z "${SLACK_WEBHOOK:-}" ]]; then
  echo "[DIGEST]\n$text"
  exit 0
fi

if [[ "${DRY_RUN:-0}" = "1" ]]; then
  echo "[DRY-RUN DIGEST]\n$text"
  exit 0
fi

curl -sS -X POST -H 'Content-type: application/json' \
  --data "$(jq -n --arg t "$text" '{text:$t}')" \
  "$SLACK_WEBHOOK" >/dev/null

# Rotate
rm -f -- "$file"

Make executable:

sudo chmod +x /usr/local/bin/ai_alert_digest.sh

Run it hourly with systemd (recommended):

/etc/systemd/system/ai-alert-digest.service

[Unit]
Description=AI Alert Digest Sender

[Service]
Type=oneshot
Environment=SPOOL_DIR=/var/lib/ai-alerts/spool
Environment=SLACK_WEBHOOK=YOUR_SLACK_WEBHOOK_URL
Environment=DRY_RUN=0
ExecStart=/usr/local/bin/ai_alert_digest.sh

/etc/systemd/system/ai-alert-digest.timer

[Unit]
Description=Run AI Alert Digest Hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-alert-digest.timer

Step 3 — Keep a living ignore list (the “noise killer”)

Start with an empty file and add patterns as you confirm non-actionable alerts:

sudo touch /var/lib/ai-alerts/ignore_patterns.txt
sudo chown "$USER":"$USER" /var/lib/ai-alerts/ignore_patterns.txt

Add examples (extended regex, case-insensitive):

# flaky dev-only checks
^Readiness probe failed:.*namespace=dev
# low-value cron noise
^cron\[[0-9]+\]: (INFO|Notice):
# spammy disk “almost full” at 70%
(Usage at 7[0-9]%|threshold=70%)

The reducer will skip those early, before fingerprinting.

Real-world examples that benefit immediately

  • Repeating “Disk almost full” messages from many nodes: de-duplicated per node and summarized hourly; escalated only if AI (or static mapping) deems it high and it reappears after the window.

  • Flaky health checks that flip quickly: throttled within a 10-minute window, preventing dozens of pages.

  • Verbose stack traces: AI reduces them to “actionable? likely yes/no” based on keywords like “panic/segfault/oom-kill,” pushing only the high-signal cases immediately.

  • TLS certificate warnings: queued into digest unless they’re near expiry and recurring (in which case they bubble to high).

Optional: run it as a service

You can wire the reducer to your alert source with systemd. For example, to read from journald:

/etc/systemd/system/ai-alert-reducer.service

[Unit]
Description=AI Alert Reducer (journald)
After=network-online.target

[Service]
Environment=SLACK_WEBHOOK=YOUR_SLACK_WEBHOOK_URL
Environment=OPENAI_API_KEY=YOUR_API_KEY
Environment=AI_API_URL=https://api.openai.com/v1/chat/completions
Environment=AI_MODEL=gpt-4o-mini
Environment=DRY_RUN=0
ExecStart=/bin/sh -c 'journalctl -f -o json | /usr/local/bin/ai_alert_reducer.sh'
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-alert-reducer.service

If your alerts come from a file or pipe, replace the ExecStart pipeline accordingly (e.g., tail -F /var/log/alerts.jsonl | …).

3–5 actionable takeaways

  • Throttle first, then think: A simple fingerprint + time window can remove most duplicate noise before any AI call.

  • Use AI as a tie-breaker: Let a small, cheap model rate actionability only for de-duplicated alerts; always keep a safe fallback.

  • Batch the long tail: Move low/medium alerts into hourly digests so on-call sees context, not chaos.

  • Make “ignore” a team sport: Keep a shared regex list; prune it during postmortems.

  • Start in shadow mode: Run with DRY_RUN=1, compare what would have paged vs. what actually paged, then flip the switch.

Conclusion and next steps

Alert fatigue isn’t just annoying—it hides real incidents. With a few lines of Bash, jq, sqlite, and curl, you can chop your noise floor, page only for what’s truly urgent, and give teams room to think.

Your next step:

  • Deploy the reducer in shadow mode for a week.

  • Review the hourly digests with your team.

  • Refine the ignore patterns and the AI prompt.

  • Turn on real routing for high/critical items.

If you want a deeper dive (e.g., Prometheus Alertmanager webhooks, service ownership routing, or on-call calendars), extend the same pattern: normalize → dedupe → AI score → route/digest. Keep it simple, observable, and versioned.

Happy de-noising. Your 2 a.m. self will thank you.

Related posts