Posted on
• Artificial Intelligence

Artificial Intelligence SRE Workflows

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered SRE Workflows from the Bash Prompt

If you’ve ever stared at a sea of logs at 2 a.m., hopping across dashboards while a pager screams, you know the real bottleneck in incident response isn’t data—it’s time-to-insight. This post shows how to fold Artificial Intelligence directly into your Linux Bash workflows, compressing noise into decisions without leaving the terminal you already trust.

We’ll cover why AI belongs in SRE, four practical workflows you can adopt today, and the minimal setup needed. Everything runs from Bash with simple pipes and curl, so you can pilot it in minutes.

Why AI in SRE (and why now)

  • Signal compression: LLMs excel at summarizing logs and proposing next steps—exactly what you need in the first 15 minutes of an incident.

  • Fits the CLI: You don’t need a new UI or dashboard. Stream logs, metrics, YAML, or timelines straight to AI via pipes.

  • Repeatable and auditable: Prompts and responses live in your shell history or repo; you can diff, review, and iterate like code.

  • Human-in-the-loop: AI augments judgment; it doesn’t replace it. Use it to draft summaries, runbooks, and remediations faster—and then decide.

Prerequisites: one-time install

These tools are commonly available on most distros. Pick the block for your package manager.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq ripgrep git
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl jq ripgrep git
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq ripgrep git

You’ll also need an LLM endpoint. The examples below use the OpenAI Chat Completions API via curl for simplicity, but you can adapt the function to any compatible provider.

Set your API key:

export OPENAI_API_KEY="sk-..."

Optional environment knobs:

export AI_MODEL="gpt-4o-mini"     # default model used in examples
export AI_TEMPERATURE="0"         # safer, deterministic answers
export AI_MAX_TOKENS="512"        # response length cap

Drop-in: a tiny Bash “ai” helper

Add this to your shell profile or ai.sh and source it. It reads context from stdin and a prompt from the first argument, then prints the model’s answer.

ai() {
  local model="${AI_MODEL:-gpt-4o-mini}"
  local temperature="${AI_TEMPERATURE:-0}"
  local max_tokens="${AI_MAX_TOKENS:-512}"

  if [ -z "$OPENAI_API_KEY" ]; then
    echo "OPENAI_API_KEY is not set." >&2
    return 1
  fi

  local prompt="$1"; shift || true
  local context
  context="$(cat)"

  jq -n \
    --arg model "$model" \
    --arg temp "$temperature" \
    --arg max_tokens "$max_tokens" \
    --arg prompt "$prompt" \
    --arg context "$context" \
    '{
      model: $model,
      temperature: ($temp|tonumber),
      max_tokens: ($max_tokens|tonumber),
      messages: [
        {role: "system", content: "You are a Linux SRE assistant. Provide concise, step-by-step, safe guidance. Prefer idempotent commands. Never propose destructive actions without explicit warnings."},
        {role: "user", content: ($prompt + "\n\nContext:\n" + $context)}
      ]
    }' | \
  curl -sS https://api.openai.com/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d @- | jq -r '.choices[0].message.content'
}

Optional: a tiny redactor for secrets before sending context:

mask_secrets() {
  sed -E 's/(pass(word)?|token|secret|apikey|auth)=\S+/\1=REDACTED/gi'
}

Usage example:

journalctl -u myapp --since -15m | mask_secrets | ai "Summarize likely root cause and list the next 3 safe diagnostic steps."

1) Lightning triage: turn raw logs into next steps

Problem: Incidents start with too much text and not enough direction.

Approach: Pipe the last chunk of logs into ai with a clear ask: probable cause, confidence, and the top three diagnostics.

Example:

journalctl -u payments.service --since -15m \
  | rg -v "healthcheck|liveness" \
  | tail -n 800 \
  | ai "From these logs, infer the most likely root cause. Provide: 

- 1-sentence summary 

- Top 3 next diagnostic commands (safe/idempotent) 

- Any obvious rollback or traffic-routing suggestions."

What you’ll get: a concise summary and a short, actionable checklist that your on-call can copy/paste. Use it to direct the first 10 minutes of response rather than guess.

Tips:

  • Filter noise first (e.g., health checks) with rg -v.

  • Keep context tight (last 500–1,000 lines) for faster, cheaper results.

  • Ask for commands to be safe and idempotent.

2) AI-aware runbooks that adapt to the host

Problem: Stale static runbooks don’t match today’s host state.

Approach: Gather live context (OS/kernels, disk/mem, top offenders, ports), then ask the AI to tailor a minimal, reversible plan.

Collector + runbook draft:

(
  echo "# Host context"
  echo "Hostname: $(hostname)"
  echo "Uname: $(uname -a)"
  echo "Uptime: $(uptime -p)"
  echo; echo "Disk usage:"; df -h
  echo; echo "Memory:"; free -m
  echo; echo "Top CPU:"; ps -eo pcpu,pid,comm --sort=-pcpu | head -n 10
  echo; echo "Top MEM:"; ps -eo pmem,pid,comm --sort=-pmem | head -n 10
  echo; echo "Open TCP:"; ss -lntp | head -n 50
) | mask_secrets | ai "Given this host context, propose a minimal, idempotent runbook to diagnose high 5xx errors in nginx. 
Output sections: Hypotheses, Verifications (commands), Low-risk Mitigations, Rollback."

What you’ll get: a focused checklist that fits your host’s reality, not a generic wiki page.

3) Automatic postmortem timeline from raw events

Problem: Writing timelines steals hours after an incident.

Approach: Harvest recent events from system logs and app logs, then ask AI to normalize into a Markdown postmortem with Impact, Timeline, and Follow-ups.

Example (adjust paths/services to taste):

{
  echo "=== journalctl (last 60m) ==="
  journalctl --since -60m -p 3..7 -o short-iso | tail -n 1000
  echo
  echo "=== nginx error.log (last 60m) ==="
  awk -v d="$(date -u -d '60 minutes ago' '+%Y/%m/%d %H:%M:%S')" \
      '$0 >= d' /var/log/nginx/error.log 2>/dev/null | tail -n 1000
} | mask_secrets | ai "Create a concise post-incident report in Markdown with sections:

- Summary (what, impact, duration)

- Root Cause (most plausible, with caveats)

- Timeline (UTC, deduplicated, 10–20 key points)

- Immediate Remediation

- Preventative Actions (ranked)
Cite log lines minimally where needed."

What you’ll get: a first draft that a human can quickly edit rather than author from scratch.

4) Explain and tune alert rules before they page

Problem: Noisy alerts hide real incidents.

Approach: Feed your alert rule YAML (e.g., Prometheus) to the model and ask it to explain the logic in plain language, then suggest safer thresholds or rate-limits.

Example:

cat rules/alerts/latency.yml \
  | ai "Explain these alert rules to a new on-call in plain English. 
Identify high-churn alerts and propose safer thresholds or for/how modifiers, with rationale. 
Output a diff-like suggestion block we can review."

What you’ll get: a readable explanation and concrete tuning ideas you can review with the team.

Real-world usage notes and guardrails

  • Redact first: Use mask_secrets or custom filters to scrub tokens, emails, and PII before sending logs.

  • Keep humans in control: Treat AI output as a draft. Don’t auto-execute returned commands.

  • Constrain output: Use AI_TEMPERATURE=0 and AI_MAX_TOKENS=512 for crisp, focused responses.

  • Be explicit: Tell the model what format you want (bullets, sections, diffs) and what to avoid (destructive commands).

  • Log the prompts: Pipe AI inputs/outputs to files in incidents to improve your prompts over time.

Troubleshooting

  • 401/403 errors: Check OPENAI_API_KEY and network egress.

  • 429 rate limits: Reduce frequency, shrink context, or backoff/retry.

  • Bad JSON or garbled responses: Validate inputs; keep logs under a few thousand lines per call.

  • Privacy/compliance: If cloud egress is not allowed, adapt the function to a self-hosted or on-prem model/API with similar JSON.

Conclusion and next step

AI won’t fix your infra, but it can dramatically shorten the path from “too much data” to “good first decision.” Start small:

1) Install curl, jq, ripgrep.
2) Add the ai function to your shell.
3) Pilot one workflow at your next incident (log triage or timeline).
4) Iterate on prompts and redaction as you learn.

If this helped, turn these snippets into a shared ai.sh in your team’s dotfiles or SRE toolkit, and make “AI from the Bash prompt” a standard part of your on-call playbook.

Related posts