Posted on
• Artificial Intelligence

Artificial Intelligence DevOps Dashboards

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence DevOps Dashboards — A Bash‑First Guide on Linux

Ever stared at a Grafana board at 3 AM, drowning in panels, wishing it would just tell you what’s wrong? AI‑assisted DevOps dashboards do exactly that: they condense noisy metrics into short, human‑readable insights so you can act faster. In this post, you’ll stand up a lightweight observability stack, add a Bash‑driven AI summary, and surface it directly in Grafana.

What you’ll get:

  • A minimal, distro‑friendly Prometheus + Node Exporter + Grafana setup

  • A textfile collector to ship custom SLO metrics from Bash

  • A script that queries Prometheus and produces an AI incident/capacity summary

  • A way to post that AI summary back into Grafana as an annotation

Why this matters:

  • Reduces MTTR: AI summarizes anomalies and capacity risks

  • Decreases cognitive load: from dozens of time series to one actionable paragraph

  • Fits your Linux/Bash workflow: no heavy orchestration required to get started

1) Install the core observability stack

You’ll install:

  • Prometheus (time‑series database + scraper)

  • Node Exporter (basic host metrics)

  • Grafana (dashboards)

  • jq and curl (Bash plumbing)

Ports (defaults): Prometheus 9090, Grafana 3000, Node Exporter 9100.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y prometheus prometheus-node-exporter jq curl

# Add Grafana APT repo (official, recommended for up-to-date builds)
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

# Enable and start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-node-exporter || sudo systemctl enable --now node_exporter
sudo systemctl enable --now grafana-server

Fedora/RHEL/CentOS Stream (dnf):

# For RHEL/CentOS, enable EPEL first (Fedora does not need this)
sudo dnf install -y epel-release

sudo dnf install -y prometheus node_exporter jq curl

# Add Grafana YUM repo (official)
sudo tee /etc/yum.repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
EOF

sudo dnf install -y grafana

# Enable and start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now node_exporter || sudo systemctl enable --now prometheus-node-exporter
sudo systemctl enable --now grafana-server

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y prometheus prometheus-node_exporter jq curl

# Add Grafana repo (RPM-based)
sudo rpm --import https://packages.grafana.com/gpg.key
sudo zypper addrepo https://packages.grafana.com/oss/rpm grafana
sudo zypper refresh
sudo zypper install -y grafana

# Enable and start services
sudo systemctl enable --now prometheus
sudo systemctl enable --now prometheus-node_exporter || sudo systemctl enable --now node_exporter
sudo systemctl enable --now grafana-server

Quick checks:

# Prometheus
curl -s http://localhost:9090/-/ready
# Node Exporter
curl -s http://localhost:9100/metrics | head
# Grafana (login at http://localhost:3000, default admin/admin on first run)

2) Ship a custom SLO metric from Bash (textfile collector)

The Node Exporter textfile collector lets you write custom metrics with simple files. We’ll enable it and publish an example “error budget” metric.

Enable the textfile collector and set its directory:

Debian/Ubuntu:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
echo 'EXTRA_FLAGS="--collector.textfile --collector.textfile.directory=/var/lib/node_exporter/textfile_collector"' | \
  sudo tee /etc/default/prometheus-node-exporter
sudo systemctl daemon-reload
sudo systemctl restart prometheus-node-exporter || sudo systemctl restart node_exporter

Fedora/RHEL:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
echo 'OPTIONS="--collector.textfile --collector.textfile.directory=/var/lib/node_exporter/textfile_collector"' | \
  sudo tee /etc/sysconfig/node_exporter
sudo systemctl daemon-reload
sudo systemctl restart node_exporter || sudo systemctl restart prometheus-node-exporter

openSUSE/SLE:

sudo mkdir -p /var/lib/node_exporter/textfile_collector
# Try the sysconfig used by your package; use one of:
echo 'OPTIONS="--collector.textfile --collector.textfile.directory=/var/lib/node_exporter/textfile_collector"' | \
  sudo tee /etc/sysconfig/prometheus-node_exporter
sudo systemctl daemon-reload
sudo systemctl restart prometheus-node_exporter || sudo systemctl restart node_exporter

Publish a sample SLO metric:

cat <<'EOF' | sudo tee /var/lib/node_exporter/textfile_collector/app_slo.prom
# HELP app_slo_error_budget_burn_rate 1m rolling burn rate for the current SLO (ratio)
# TYPE app_slo_error_budget_burn_rate gauge
app_slo_error_budget_burn_rate 0.12
EOF

# Verify Node Exporter exposes it
curl -s http://localhost:9100/metrics | grep app_slo_error_budget_burn_rate

Ensure Prometheus scrapes Node Exporter (most distro configs do). If not, add to /etc/prometheus/prometheus.yml:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Then:

sudo systemctl restart prometheus

3) Generate an AI daily triage from Prometheus (Bash + curl + jq)

This script:

  • Pulls 24h CPU, memory, and disk utilization from Prometheus

  • Includes your custom SLO metric if present

  • Asks an LLM to produce a brief, actionable summary

Requirements: jq and curl (installed above), and an LLM endpoint. Works with:

  • OpenAI‑compatible API (default: https://api.openai.com/v1)

  • Local OpenAI‑compatible servers (e.g., Ollama’s compatible endpoint)

Environment:

  • PROM_URL (default http://localhost:9090)

  • LLM_ENDPOINT (default https://api.openai.com/v1/chat/completions)

  • LLM_API_KEY (export your key)

  • LLM_MODEL (e.g., gpt-4o-mini, llama3.1, etc.)

Script (save as ai-daily-summary.sh):

#!/usr/bin/env bash
set -euo pipefail

PROM_URL="${PROM_URL:-http://localhost:9090}"
LLM_ENDPOINT="${LLM_ENDPOINT:-https://api.openai.com/v1/chat/completions}"
LLM_MODEL="${LLM_MODEL:-gpt-4o-mini}"
LLM_API_KEY="${LLM_API_KEY:-}"

if [[ -z "${LLM_API_KEY}" && "${LLM_ENDPOINT}" == "https://api.openai.com/v1/chat/completions" ]]; then
  echo "Set LLM_API_KEY to use OpenAI (or point LLM_ENDPOINT to a local compatible server that doesn't require a key)." >&2
fi

promq() {
  local q="$1"
  curl -fsS --get "${PROM_URL}/api/v1/query" --data-urlencode "query=${q}"
}

# 24h max CPU utilization by instance
Q_CPU='max_over_time((1 - avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])))[24h:5m])'
# 24h max memory utilization by instance
Q_MEM='max_over_time(((1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)))[24h:5m])'
# 24h max root filesystem utilization by instance
Q_DISK='max_over_time((1 - (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|devtmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|devtmpfs"}))[24h:5m])'
# Latest SLO burn rate if present
Q_SLO='app_slo_error_budget_burn_rate'

cpu_json="$(promq "$Q_CPU"   | jq -c '.data.result | map({instance: .metric.instance, value: (.value[1]|tonumber)})')"
mem_json="$(promq "$Q_MEM"   | jq -c '.data.result | map({instance: .metric.instance, value: (.value[1]|tonumber)})')"
disk_json="$(promq "$Q_DISK" | jq -c '.data.result | map({instance: .metric.instance, value: (.value[1]|tonumber)})')"

# SLO may not exist; handle gracefully
slo_json="$(promq "$Q_SLO" 2>/dev/null | jq -c '.data.result | map({metric: "app_slo_error_budget_burn_rate", value: (.value[1]|tonumber)})' || echo '[]')"

context="$(jq -n \
  --arg date "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
  --arg qcpu "$Q_CPU" --arg qmem "$Q_MEM" --arg qdisk "$Q_DISK" --arg qslo "$Q_SLO" \
  --argjson cpu "$cpu_json" --argjson mem "$mem_json" --argjson disk "$disk_json" --argjson slo "$slo_json" \
  '{generated_at:$date, queries:{cpu:$qcpu, mem:$qmem, disk:$qdisk, slo:$qslo}, metrics:{cpu:$cpu, mem:$mem, disk:$disk, slo:$slo}}'
)"

read -r -d '' SYSTEM <<'EOS'
You are an SRE assistant. Produce a concise markdown summary (<= 12 lines).

- Highlight top capacity risks (CPU, memory, disk) across instances.

- Flag sudden spikes or outliers and suggest immediate actions.

- If SLO burn rate is present, interpret it in plain language (is the error budget safe or burning too fast?).

- Be specific: include instance names and thresholds where possible.
EOS

PROMPT="Here is 24h metrics context in JSON. Summarize for an on-call handoff with bullet points and short actions.

${context}
"

# Build JSON payload for OpenAI-compatible endpoint
payload="$(jq -n --arg model "$LLM_MODEL" --arg system "$SYSTEM" --arg user "$PROMPT" \
  '{model:$model, temperature:0.2, messages:[{role:"system",content:$system},{role:"user",content:$user}] }')"

# Call the LLM
if [[ -n "${LLM_API_KEY}" ]]; then
  summary="$(curl -fsS -H "Authorization: Bearer ${LLM_API_KEY}" -H "Content-Type: application/json" \
    -d "$payload" "$LLM_ENDPOINT" | jq -r '.choices[0].message.content')"
else
  summary="$(curl -fsS -H "Content-Type: application/json" -d "$payload" "$LLM_ENDPOINT" | jq -r '.choices[0].message.content')"
fi

echo "=== AI Daily Summary ($(date -u)) ==="
echo "$summary"
echo

# Optional: post to Grafana annotations if credentials are set
if [[ -n "${GRAFANA_URL:-}" && -n "${GRAFANA_API_KEY:-}" ]]; then
  ts_ms="$(date +%s%3N)"
  # Build annotation JSON safely
  ann_payload="$(jq -n --arg text "$summary" --argjson time "$ts_ms" \
    '{text:$text, tags:["ai","daily-summary"], time:$time}')"
  curl -fsS -X POST "${GRAFANA_URL}/api/annotations" \
    -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "$ann_payload" >/dev/null && echo "Posted summary to Grafana annotations."
fi

Make it executable and run:

chmod +x ai-daily-summary.sh
./ai-daily-summary.sh

Tip: Use a local OpenAI‑compatible server if you prefer not to call external APIs. For example, if you run an on‑prem LLM exposing the same API shape, set:

export LLM_ENDPOINT="http://localhost:11434/v1/chat/completions"
export LLM_MODEL="llama3.1"
unset LLM_API_KEY

Automate with cron:

crontab -e
# Run at 08:00 UTC daily
0 8 * * * PROM_URL=http://localhost:9090 LLM_ENDPOINT=https://api.openai.com/v1/chat/completions LLM_MODEL=gpt-4o-mini LLM_API_KEY=YOUR_KEY GRAFANA_URL=http://localhost:3000 GRAFANA_API_KEY=YOUR_GRAFANA_TOKEN /path/to/ai-daily-summary.sh >> /var/log/ai-summary.log 2>&1

4) Surface AI insights in Grafana (no plugins needed)

The script above can post directly to Grafana’s Annotations API. In Grafana:

  • Create an API Key (Settings > API Keys > Add, Role: Editor or Admin)

  • Export it as GRAFANA_API_KEY and set GRAFANA_URL (e.g., http://localhost:3000)

  • The AI summary appears as a timeline annotation on any dashboard, tagged ai and daily-summary

You can then:

  • Filter by tags to view only AI handoffs

  • Click annotations to read the full summary

  • Correlate the text with metric spikes

5) Real‑world patterns that work

  • Capacity planning snapshots

    • The summary highlights instances with >80% 24h max CPU/memory/disk. Action: schedule vertical scaling or rebalance workloads.

  • Regression triage after deploys

    • Include a custom textfile metric like app_release_info{version="1.2.3"} 1 and a burn‑rate metric. The AI will connect “post‑deploy” to “increased burn” and suggest rollback or feature flagging.

  • Noisy alert reduction

    • Point the script at common “flapping” metrics and let it recommend threshold changes (e.g., use 95th percentile over 10m instead of 1m peaks).

  • Multi‑instance hotspots

    • Because the prompt includes instance labels, the AI calls out which hosts are outliers—useful in fleets and mixed hardware generations.

Troubleshooting and tips

  • If you don’t see node metrics in Prometheus, visit http://localhost:9090/targets and check the node job state.

  • For the textfile collector, confirm the directory flag matches where you write .prom files.

  • Use curl -v to debug API calls; jq -C . for colored JSON when testing.

  • Keep prompts short and specific; include only the metrics you care about to reduce noise and cost.

Conclusion and next steps

You just built an AI‑assisted DevOps dashboard flow using standard Linux tools and Bash:

  • Prometheus + Node Exporter + Grafana for collection and visualization

  • A textfile collector for quick, scriptable SLOs

  • A portable Bash script that turns raw PromQL into a daily, actionable AI summary

  • Optional auto‑posting into Grafana annotations

Next steps:

  • Add logs (Grafana Loki) and traces (Tempo) to teach the AI cross‑signal context

  • Expand queries to include SLIs from app/service exporters

  • Hook this into Alertmanager webhooks to generate AI triage on page

If this helped, try wiring the script into your team’s main dashboard today, then iterate on the prompt and queries for your environment. Your future 3 AM self will thank you.

Related posts