Posted on
• Artificial Intelligence

Artificial Intelligence Linux Troubleshooting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Linux Troubleshooting with Bash: From Noisy Logs to Clear Fixes

Ever stared at a wall of logs at 2 a.m., trying to connect “why did the service die” with “what changed”? Artificial Intelligence can be your second set of eyes: not a replacement for your judgment, but a tireless assistant that summarizes, classifies, and suggests next steps while you stay in control.

In this article, you’ll learn how to wire AI into your Bash workflow so you can:

  • Snapshot a sick system and get a concise, prioritized diagnosis.

  • Turn vague symptoms into safe, reviewable command suggestions.

  • Classify noisy logs into labeled root causes with actions.

  • Summarize complex traces (strace/lsof) into something you can act on.

You’ll also get distro-agnostic install instructions and copy‑pasteable scripts.

Why use AI in Linux troubleshooting?

  • Speed: AI can sift through hundreds of lines of journalctl, dmesg, iostat, or strace faster than you can scroll, surfacing likely culprits.

  • Clarity: Good prompts produce actionable, structured summaries: “Disk I/O bottleneck on sda”, “OOM killer targeting service X”, “DNS failure path”.

  • Safety net: Combine AI’s suggestions with guardrails (ShellCheck, dry‑runs, confirmation prompts) to avoid risky commands.

  • Repeatability: Wrap your checks and AI prompts in scripts. Now your on‑call runbook is a command, not a memory.

Note: AI is a helper. Keep secrets out of prompts. Prefer local models for strict privacy, and always review before executing any suggested commands.

Prerequisites (install once)

We’ll rely on a few command‑line tools.

  • curl – HTTP client to call APIs

  • jq – JSON processing

  • ShellCheck – lint Bash produced by AI

  • strace – syscall tracing

  • sysstat – iostat/mpstat for performance triage

  • glances – cross‑stack snapshot

  • lsof – what’s using files/ports

Install them with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq shellcheck strace sysstat glances lsof
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y curl jq ShellCheck strace sysstat glances lsof
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq ShellCheck strace sysstat glances lsof

If you want iostat history, enable sysstat collection:

sudo systemctl enable --now sysstat

If you plan to use a cloud AI API, set your key in your shell:

export OPENAI_API_KEY="your_api_key_here"

Prefer local/offline? You can use a local LLM via Ollama and replace the API call with ollama run model_name (optional).

Pattern 1: One‑command AI triage of a broken host

This script collects a compact snapshot (CPU, memory, disk, network, logs) and asks an AI to produce a prioritized diagnosis plus next steps.

Save as ai-triage.sh and make it executable:

chmod +x ai-triage.sh

#!/usr/bin/env bash
set -euo pipefail

# Configuration
SINCE="${1:-2 hours ago}"   # e.g., "30 minutes ago", "2 hours ago"
MAX_LOG_LINES=800           # keep prompts small enough
TMPDIR="$(mktemp -d)"
REPORT="$TMPDIR/report.txt"

# 1) System snapshot
{
  echo "# Host and uptime"
  hostnamectl || true
  echo; uptime; date; who -b

  echo; echo "# CPU/Mem quick view"
  top -b -n1 | head -n 20
  echo; free -h

  echo; echo "# Disk"
  df -hT | sort -k6
  echo; iostat -xz 1 3 2>/dev/null || true

  echo; echo "# Network listeners"
  ss -tulpn 2>/dev/null | head -n 50 || true

  echo; echo "# Recent kernel messages"
  dmesg --ctime | tail -n 300

  echo; echo "# Recent systemd journal (errors+)"
  journalctl -p 3 -S "$SINCE" --no-pager | tail -n "$MAX_LOG_LINES"
} > "$REPORT"

# 2) Redact obvious secrets
sed -i -E \
  -e 's/(Authorization: Bearer )[A-Za-z0-9\._-]+/\1REDACTED/g' \
  -e 's/(password=)[^ ]+/\1REDACTED/g' \
  "$REPORT"

# 3) Ask AI for a structured diagnosis
read -r -d '' PROMPT << 'EOF' || true
You are a Linux SRE. Read the triage snapshot and produce:
1) Top 3 likely root causes (bullet list, bold key signal with file/metric names).
2) Minimal actionable checks to confirm each cause (exact commands).
3) Safe, reversible remediation steps (exact commands; note risks).
4) If inconclusive, ask me for the smallest next artifact.

Only reference evidence that appears in the snapshot. Keep it under 250 words.
EOF

if [[ -n "${OPENAI_API_KEY:-}" ]]; then
  # Cloud API example (OpenAI Chat Completions)
  RESPONSE="$(
    jq -Rs --arg prompt "$PROMPT" \
      '{model:"gpt-4o-mini",messages:[{role:"system",content:$prompt},{role:"user",content:.}],temperature:0.2}' \
      < "$REPORT" |
    curl -sS https://api.openai.com/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
      -d @- | jq -r '.choices[0].message.content'
  )"
  echo -e "\n===== AI DIAGNOSIS =====\n$RESPONSE"
else
  echo "No OPENAI_API_KEY found. Falling back to plain snapshot output."
  echo "Set OPENAI_API_KEY to get an AI diagnosis, or pipe to a local LLM like:"
  echo "cat $REPORT | ollama run llama3:8b"
fi

echo -e "\nSnapshot saved at: $REPORT"

How to use:

./ai-triage.sh "30 minutes ago"

Real‑world wins:

  • Disk full: The AI highlights “No space left on device” in journalctl and df anomalies, suggests du -xh --max-depth=1 /var | sort -h | tail and a safe cleanup plan.

  • OOM killer: Notices Out of memory in dmesg, points to the service hit, proposes systemd-cgtop, oomctl, and a memory limit tweak.

  • DNS flap: Finds repeated Temporary failure in name resolution and recommends checking resolvers, dig +trace, or reverting a recent /etc/resolv.conf change.

Pattern 2: AI‑assisted command suggestions with guardrails

Turn a fuzzy task (“rotate logs without downtime”) into a specific, reviewed command. ShellCheck prevents obvious foot‑guns; you approve before execution.

Save as ai-cmd.sh:

chmod +x ai-cmd.sh

#!/usr/bin/env bash
set -euo pipefail

QUERY="${*:-Describe what you want done, e.g. 'find top 5 largest dirs in /var/log and show sizes' }"

read -r -d '' SYSTEM << 'EOF' || true
You output a single Bash one-liner that is:

- Safe by default (read-only where possible, uses --dry-run if available).

- Portable across common distros.

- Contains comments inline with # explaining key flags.

- No destructive actions unless explicitly requested; then ask for confirmation via read -p.
Return ONLY the command; no extra prose.
EOF

if [[ -z "${OPENAI_API_KEY:-}" ]]; then
  echo "Set OPENAI_API_KEY to use the cloud API. Example query:"
  echo "$0 find top 5 largest dirs in /var and show sizes"
  exit 1
fi

CMD="$(
  jq -nc --arg sys "$SYSTEM" --arg q "$QUERY" \
    '{model:"gpt-4o-mini",messages:[{role:"system",content:$sys},{role:"user",content:$q}],temperature:0.1}' |
  curl -sS https://api.openai.com/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
    -d @- | jq -r '.choices[0].message.content' | sed -e 's/^```.*//; s/```$//'
)"

echo "Suggested command:"
echo "$CMD"
echo

# Lint the command for common issues
echo "$CMD" | shellcheck -s bash - || true
echo

read -rp "Run this command? [y/N] " ans
if [[ "$ans" =~ ^[Yy]$ ]]; then
  eval "$CMD"
fi

Example:

./ai-cmd.sh "Show top 10 processes by open file descriptors"

You’ll see a one‑liner, ShellCheck warnings (if any), and a prompt to confirm before running.

Pattern 3: Label and de‑noise logs in real time

This script batches recent errors and asks AI to bucket them into causes with suggested actions.

Save as ai-log-labeler.sh:

chmod +x ai-log-labeler.sh

#!/usr/bin/env bash
set -euo pipefail

BATCH=200
INTERVAL=30   # seconds

read -r -d '' INSTR << 'EOF' || true
You are a log triage assistant. Given a batch of mixed logs, output:

- A short list of labeled issues with frequency counts (descending).

- For each label: 1 verifying command, 1 likely fix, 1 escalation hint.
Do not invent details not present in logs. Keep under 180 words total.
EOF

while true; do
  LOGS="$(journalctl -p 3 -n "$BATCH" --no-pager | sed -E 's/(password|token)=\S+/\1=REDACTED/g')"

  if [[ -n "${OPENAI_API_KEY:-}" && -n "$LOGS" ]]; then
    OUT="$(
      jq -nc --arg sys "$INSTR" --arg logs "$LOGS" \
        '{model:"gpt-4o-mini",messages:[{role:"system",content:$sys},{role:"user",content:$logs}],temperature:0.2}' |
      curl -sS https://api.openai.com/v1/chat/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
        -d @- | jq -r '.choices[0].message.content'
    )"
    date
    echo "$OUT"
    echo "—"
  else
    echo "Set OPENAI_API_KEY, or install/run a local model and replace the curl call."
  fi

  sleep "$INTERVAL"
done

Try it during an incident to quickly see the “shape” of errors (auth failures vs. I/O errors vs. timeouts) without reading every line.

Pattern 4: Summarize strace/lsof output into something you can act on

When a service is stuck, strace and lsof hold the truth—but they’re verbose. Let AI pinpoint the blocking syscalls or files.

Example (one‑off):

sudo strace -tt -f -p $(pidof your-service) -o /tmp/trace.txt -s 200 -qq -e trace=file,network,process &
sleep 10; sudo kill %1

SUMMARY="$(
  jq -nc --arg sys "Summarize the strace. Identify blocking syscalls, repeated errors, key files/sockets, and most likely root cause. Output 5 bullets max." \
         --arg data "$(tail -n 2000 /tmp/trace.txt)" \
    '{model:"gpt-4o-mini",messages:[{role:"system",content:$sys},{role:"user",content:$data}],temperature:0.2}' |
  curl -sS https://api.openai.com/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer '"$OPENAI_API_KEY"'" \
    -d @- | jq -r '.choices[0].message.content'
)"
echo "$SUMMARY"

For open files:

sudo lsof -p $(pidof your-service) -nP | head -n 200 | sed 's/[[:cntrl:]]//g' > /tmp/lsof.txt
# Prompt similarly to extract top directories, sockets, deleted-but-open files, etc.

Tips for safe and effective AI troubleshooting

  • Redact: Strip secrets/tokens before sending any data.

  • Limit scope: Send only the relevant slice (last N lines, last 30 minutes).

  • Verify: Use ShellCheck and your own review before running scripts from AI.

  • Prefer local for sensitive systems: Use a local LLM (e.g., via Ollama) to keep data on‑box.

  • Keep artifacts: Save snapshots and AI outputs alongside incident notes for postmortems.

Conclusion and next steps

AI won’t replace your Linux chops—but paired with disciplined prompts and Bash, it dramatically reduces time‑to‑insight. Start by dropping ai-triage.sh into your toolbox. Then add ai-cmd.sh for safe command suggestions and ai-log-labeler.sh to tame noisy journals.

Your next step:

  • Install the prerequisites (apt/dnf/zypper commands above).

  • Add your API key or wire up a local LLM.

  • Run ./ai-triage.sh "2 hours ago" on a test box and iterate.

Have a favorite prompt or a tricky log that AI cracked? Share your recipe and improvements with your team—and turn those 2 a.m. mysteries into 10‑minute fixes.

Related posts