Posted on
• Artificial Intelligence

Artificial Intelligence Runbooks for DevOps

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Runbooks for DevOps: Bash-first Playbooks that Think

If your incidents still begin with “paste logs into an LLM” and end with “hope the suggestion works,” you’re leaving safety, speed, and repeatability on the table. AI can help—but not as ad‑hoc chat. The real leverage comes from AI runbooks: versioned, auditable, Bash-driven playbooks that use a model to reason, propose commands, and execute only with your explicit approval.

This post shows you how to build AI runbooks that fit cleanly into a Linux/Bash workflow. You’ll get a minimal implementation you can copy, concrete examples, and clear guardrails so AI remains an accelerator, not a risk.

Why AI Runbooks, and Why Now

  • Scale and complexity: Modern estates (Kubernetes, microservices, hybrids) produce more signals than humans can triage quickly.

  • Toil reduction: Repetitive triage, log digging, and command lookups are perfect for AI-assisted workflows.

  • Safety over chat: “Chatting” a fix into production is not a control. Codifying prompts, constraints, and approvals in a runbook is.

  • Auditability: Runbooks-as-code give you diffable prompts, versioned guardrails, and logged execution transcripts for postmortems and compliance.

What Is an AI Runbook?

An AI runbook is:

  • A small YAML file that describes the incident type, inputs, constraints, and prompt.

  • A Bash harness that:

    • Gathers system context (logs, metrics, env),
    • Asks an LLM to propose a plan and explicit commands (JSON only),
    • Displays a human-readable plan,
    • Executes commands only on your approval,
    • Logs everything for audit.

You keep everything in Git. You can run locally, in CI, or during an incident bridge.

Prerequisites and Installation

You’ll need curl, jq, Python 3 with pip (to install yq), and optionally Podman if you want to run a local model in a container.

Install base tools:

  • Debian/Ubuntu (apt):

    sudo apt-get update
sudo apt-get install -y jq curl python3 python3-pip git podman

  • RHEL/Fedora/CentOS (dnf):

    sudo dnf install -y jq curl python3 python3-pip git podman

  • openSUSE/SLE (zypper):

    sudo zypper refresh
sudo zypper install -y jq curl python3 python3-pip git podman

Install yq (Python-based; uses jq under the hood):

pip3 install --user yq
# Ensure ~/.local/bin is in your PATH, e.g.:
export PATH="$HOME/.local/bin:$PATH"

Choose an LLM backend:

Option A: Local model via Ollama (simple)

curl -fsSL https://ollama.com/install.sh | sh
# Start the service (new shell may be required depending on installer)
ollama serve &
# Pull a capable model (example: llama3)
ollama pull llama3

Option B: Containerized Ollama (Podman)

podman run -d --name ollama -p 11434:11434 ollama/ollama
podman exec -it ollama ollama pull llama3

Option C: Cloud API

  • Export your provider key and model, for example:
export OPENAI_API_KEY="sk-..."
export AI_MODEL="gpt-4o-mini"

The Minimal AI Runbook System (Copy/Paste)

Create a directory layout:

mkdir -p ai-runbooks ai-logs

Example runbook: Investigate Nginx 502 spikes, stored at ai-runbooks/nginx-502.yaml

name: nginx-502
description: Investigate a spike of 502 Bad Gateway on an Nginx reverse proxy
inputs:
  - SERVICE_HOST
  - LOG_PATH
limits:
  allowed_commands:
    - "journalctl*"
    - "grep*"
    - "awk*"
    - "nginx*"
    - "curl*"
prompt: |
  You are an SRE assistant. The goal is to triage 502 Bad Gateway reports on an Nginx reverse proxy.
  Constraints:
  - Only propose commands that match the allowed_commands whitelist.
  - Prefer read-only operations first (mode=read_only).
  - If suggesting writes/restarts, explain why and set mode=write.
  Context:
  - Service host: ${SERVICE_HOST}
  - Log path: ${LOG_PATH}
  - We are on Linux, Bash. Output JSON only.
output_format: |
  {
    "rationale": "one paragraph of reasoning",
    "commands": ["bash command 1", "bash command 2"],
    "mode": "read_only",
    "danger": 0
  }

Bash harness: ai-run.sh

#!/usr/bin/env bash
set -euo pipefail

# ai-run.sh <runbook.yaml> [KEY=VALUE ...]
# Example: ./ai-run.sh ai-runbooks/nginx-502.yaml SERVICE_HOST=proxy01 LOG_PATH=/var/log/nginx/access.log

if [[ $# -lt 1 ]]; then
  echo "Usage: $0 <runbook.yaml> [KEY=VALUE ...]" >&2
  exit 1
fi

RUNBOOK="$1"; shift || true
if [[ ! -f "$RUNBOOK" ]]; then
  echo "Runbook not found: $RUNBOOK" >&2
  exit 1
fi

# Apply input variables to environment
for kv in "$@"; do
  export "$kv"
done

# Read runbook fields
NAME="$(yq -r '.name' < "$RUNBOOK")"
DESC="$(yq -r '.description' < "$RUNBOOK")"
PROMPT_TMPL="$(yq -r '.prompt' < "$RUNBOOK")"
OUTFMT="$(yq -r '.output_format' < "$RUNBOOK")"

# Expand vars in prompt template using envsubst
if ! command -v envsubst >/dev/null 2>&1; then
  echo "envsubst not found; install gettext (provides envsubst) or use a shell that supports parameter expansion."
  exit 1
fi
PROMPT="$(printf "%s" "$PROMPT_TMPL" | envsubst)"

TS="$(date +%Y%m%d-%H%M%S)"
LOGDIR="ai-logs/${TS}-${NAME}"
mkdir -p "$LOGDIR"
TRANSCRIPT="$LOGDIR/transcript.jsonl"
PLAN="$LOGDIR/plan.json"
echo "{}" > "$PLAN"

echo "Runbook: $NAME"
echo "Desc:    $DESC"
echo "Log dir: $LOGDIR"
echo

# Choose backend: Ollama if available, otherwise OpenAI if key present
OLLAMA_URL="${OLLAMA_URL:-http://localhost:11434}"
AI_MODEL="${AI_MODEL:-llama3}"
BACKEND=""

if curl -sSf "$OLLAMA_URL/api/tags" >/dev/null 2>&1; then
  BACKEND="ollama"
elif [[ -n "${OPENAI_API_KEY:-}" ]]; then
  BACKEND="openai"
else
  echo "No AI backend detected. Start Ollama (default) or set OPENAI_API_KEY." >&2
  exit 1
fi

# Build messages
SYSTEM_MSG="You are a careful SRE assistant. You only output strict JSON as specified by the output_format. You are cautious with write operations."
USER_MSG="OUTPUT FORMAT:\n$OUTFMT\n\nPROMPT:\n$PROMPT"

echo '{"role":"system","content":'"$(jq -Rs . <<<"$SYSTEM_MSG")"'}' >> "$TRANSCRIPT"
echo '{"role":"user","content":'"$(jq -Rs . <<<"$USER_MSG")"'}' >> "$TRANSCRIPT"

CALL_AI() {
  if [[ "$BACKEND" == "ollama" ]]; then
    curl -sS -X POST "$OLLAMA_URL/api/chat" \
      -H "Content-Type: application/json" \
      -d @<(jq -n --arg m "$AI_MODEL" --arg sys "$SYSTEM_MSG" --arg user "$USER_MSG" '
        {model:$m, messages:[{role:"system",content:$sys},{role:"user",content:$user}], stream:false}')
  else
    curl -sS -X POST "https://api.openai.com/v1/chat/completions" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
      -d @<(jq -n --arg m "${AI_MODEL}" --arg sys "$SYSTEM_MSG" --arg user "$USER_MSG" '
        {model:$m, messages:[{role:"system",content:$sys},{role:"user",content:$user}], temperature:0}')
  fi
}

RAW="$(CALL_AI || true)"
if [[ -z "$RAW" ]]; then
  echo "AI call returned empty response" >&2
  exit 1
fi

# Extract assistant content
if [[ "$BACKEND" == "ollama" ]]; then
  CONTENT="$(jq -r '.message.content' <<<"$RAW")"
else
  CONTENT="$(jq -r '.choices[0].message.content' <<<"$RAW")"
fi

# Write to transcript
echo '{"role":"assistant","content":'"$(jq -Rs . <<<"$CONTENT")"'}' >> "$TRANSCRIPT"

# Validate JSON
if ! echo "$CONTENT" | jq . >/dev/null 2>&1; then
  echo "Assistant did not return valid JSON. Aborting." >&2
  echo "Raw content:" >&2
  echo "$CONTENT" >&2
  exit 1
fi

echo "$CONTENT" | tee "$PLAN" >/dev/null

RATIONALE="$(jq -r '.rationale' "$PLAN")"
MODE="$(jq -r '.mode' "$PLAN")"
DANGER="$(jq -r '.danger' "$PLAN")"
mapfile -t CMDS < <(jq -r '.commands[]?' "$PLAN")

echo
echo "Proposed plan:"
echo "Mode:     $MODE"
echo "Risk:     $DANGER/10"
echo "Why:      $RATIONALE"
echo "Commands:"
for c in "${CMDS[@]}"; do
  echo "  - $c"
done
echo

read -rp "Execute these commands? [y/N] " ans
if [[ "${ans,,}" != "y" ]]; then
  echo "Aborted by user."
  exit 0
fi

# Execute commands with logging
set -x
for c in "${CMDS[@]}"; do
  bash -c "$c" 2>&1 | tee -a "$LOGDIR/exec.log"
done
set +x
echo "Execution complete. Logs at $LOGDIR"

Make it executable:

chmod +x ai-run.sh

Try it:

./ai-run.sh ai-runbooks/nginx-502.yaml SERVICE_HOST=proxy01 LOG_PATH=/var/log/nginx/access.log

Tip: For Kubernetes-centric runbooks, set kubectl context before running the harness, then allow only read-only kubectl verbs initially.

4 Actionable Patterns for AI Runbooks

1) Codify repeatable triage in YAML, not in chat history

  • Example: Kubernetes CrashLoopBackOff triage (ai-runbooks/k8s-crashloop.yaml)
name: k8s-crashloop
description: Triage pods in CrashLoopBackOff and propose next steps
inputs:
  - NAMESPACE
limits:
  allowed_commands:
    - "kubectl -n ${NAMESPACE} get pods*"
    - "kubectl -n ${NAMESPACE} describe pod*"
    - "kubectl -n ${NAMESPACE} logs*"
    - "grep*"
prompt: |
  Identify pods in CrashLoopBackOff and summarize root cause hypotheses.
  Prefer read_only. DO NOT delete or restart pods.
output_format: |
  {"rationale":"...", "commands":["kubectl -n ${NAMESPACE} get pods","kubectl -n ${NAMESPACE} logs DEPLOYMENT_POD --previous | tail -n 200"], "mode":"read_only","danger":1}

Run:

./ai-run.sh ai-runbooks/k8s-crashloop.yaml NAMESPACE=payments

2) Keep strong guardrails

  • Whitelist commands in runbook.limits.allowed_commands.

  • Default to read-only mode; require an explicit, justified switch to write mode.

  • Dry-run step: the harness always prints the plan and asks for confirmation.

  • Timebox and scope: focus on a specific service/namespace/log path to reduce blast radius.

  • Redact secrets from prompts (e.g., replace tokens with placeholders before sending).

3) Use the model for reasoning, not blind execution

  • Ask for a rationale and a risk score.

  • Force strict JSON output and validate it with jq before acting.

  • Keep the human-in-the-loop: the script never auto-runs without your y/N.

4) Bake in package-aware remediation safely When patching or upgrading a specific package, propose only the minimal, package-manager-aware command. Examples (replace PACKAGE with the actual name):

  • Debian/Ubuntu (apt):

    sudo apt-get update
sudo apt-get install --only-upgrade -y PACKAGE

  • RHEL/Fedora/CentOS (dnf):

    sudo dnf upgrade -y PACKAGE

  • openSUSE/SLE (zypper):

    sudo zypper refresh
sudo zypper update -y PACKAGE

Teach your AI runbook to recommend these commands only after confirming the vulnerable version is installed and the candidate update resolves the CVE in your repo metadata.

Real-World Examples You Can Try Today

  • Nginx 502 triage

    • Gathers last 500 lines from access and error logs.
    • Correlates upstream timeouts/conn resets.
    • Suggests curl health checks to each upstream backend.
    • Proposes safe config lints (read-only) and highlights likely saturation.

  • System-wide log anomaly scan

    • Reads journalctl for the last 10 minutes, clusters errors, proposes the smallest set of confirmatory commands.
    • Keeps disk I/O low: tail instead of cat, time-windowed logs.

  • Package vulnerability remediation

    • Confirms installed version via package manager query, compares with repo candidate.
    • Offers upgrade commands for apt, dnf, and zypper, with a rollback/hold note if supported by your policy.

Operationalizing AI Runbooks

  • Version control: keep runbooks and the harness in Git; code review prompt changes like you would any infra change.

  • CI smoke tests: run the harness with mock logs/fixtures; verify the model returns valid JSON and only whitelisted commands.

  • Observability: log transcripts and executions to a central place; tag with incident IDs.

  • Secrets hygiene: never print or send secrets; scrub env and logs; pass only minimum context.

  • Model lifecycle: pin model versions, track regressions, and roll back if output quality drifts.

Conclusion and Next Steps (CTA)

AI runbooks move you from risky chat to safe, auditable action. Copy the harness, add one or two YAML runbooks for your most common incidents, and run them in read-only mode this week. Iterate quickly: tighten guardrails, improve prompts, and add regression tests.

Your next step:

  • Install the prerequisites (apt/dnf/zypper above).

  • Pick a single high-toil incident (e.g., Nginx 502 or K8s CrashLoopBackOff).

  • Create a YAML runbook and try the harness in read-only mode.

  • Share the logs and results with your team, then expand coverage.

If you want a follow-up post with a full test suite, containerized harness, and GitHub Actions examples, let me know which stack you’re running and I’ll tailor it.

Related posts