Posted on
• Artificial Intelligence

AI-Assisted Network Troubleshooting on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Assisted Network Troubleshooting on Linux: From Bash to Better RCA

It’s 2 a.m., your service is timing out, and you’re staring at a blinking cursor. Was it DNS? A route change? MTU? What if you could capture the right data in minutes and have an AI summarize likely root causes while you keep the incident moving?

This post shows how to combine battle-tested Linux networking tools with AI to accelerate triage and root-cause analysis (RCA), all from Bash.

  • Problem: Network incidents are noisy and multi-layered. Humans waste cycles deciding “what to collect” and “what it means.”

  • Value: Small, repeatable Bash scripts can snapshot the essentials. Feed that to an AI assistant to get structured hypotheses, next steps, and likely fixes—fast.

Why AI-assisted troubleshooting is worth it

  • Networks are complex and dynamic: routes, DNS, TLS, MTU, firewalls, NAT, VPNs.

  • Good data > guesswork: Consistent snapshots reduce blind spots and finger-pointing.

  • AI thrives on structured context: Given logs, routes, packet loss, and timing, an LLM can rapidly classify failure modes (DNS vs. path vs. port vs. app) and propose focused tests.

  • You stay in control: Use AI as a copilot. You decide what to execute and what to change.

Prerequisites

We’ll use standard CLI tools. Install what’s missing.

Apt-based (Debian/Ubuntu):

sudo apt update
sudo apt install -y iproute2 mtr-tiny traceroute dnsutils tcpdump nmap jq curl network-manager

DNF-based (Fedora/RHEL/CentOS Stream):

sudo dnf install -y iproute mtr traceroute bind-utils tcpdump nmap jq curl NetworkManager

Zypper-based (openSUSE):

sudo zypper refresh
sudo zypper install -y iproute2 mtr traceroute bind-utils tcpdump nmap jq curl NetworkManager

Notes:

  • ip/ss are provided by iproute2 (or iproute on Fedora).

  • nmcli is provided by NetworkManager (often preinstalled on desktops).

  • dig is in dnsutils (apt) or bind-utils (dnf/zypper).

1) Create a fast “network snapshot” you can feed to AI

The goal: capture interface state, routes, DNS, path health, and basic service checks—then bundle it.

Save this as net-snapshot.sh:

#!/usr/bin/env bash
set -euo pipefail

OUT="${1:-net-snapshot_$(hostname)_$(date -u +%Y%m%dT%H%M%SZ)}"
mkdir -p "$OUT"

# System and time
uname -a >"$OUT/uname.txt"
date -u +"%Y-%m-%dT%H:%M:%SZ" >"$OUT/utc_time.txt"

# Interfaces, routes, sockets
ip -brief addr show >"$OUT/ip_addr.txt" || true
ip route show table main >"$OUT/ip_route.txt" || true
ip -s link >"$OUT/ip_link_stats.txt" || true
ss -lntu >"$OUT/ss_listen.txt" || true

# DNS and resolver state
resolvectl status >"$OUT/resolvectl_status.txt" 2>/dev/null || true
cat /etc/resolv.conf >"$OUT/resolv.conf.txt" || true

# Default connectivity checks (edit targets as needed)
TARGET_IP="8.8.8.8"
TARGET_HOST="example.com"
ping -c 4 -w 5 "$TARGET_IP" >"$OUT/ping_ip.txt" || true
ping -c 4 -w 5 "$TARGET_HOST" >"$OUT/ping_dns.txt" || true

# Path tests
mtr -rwzc 50 "$TARGET_IP" >"$OUT/mtr_${TARGET_IP}.txt" 2>/dev/null || true
mtr -rwzc 50 "$TARGET_HOST" >"$OUT/mtr_${TARGET_HOST}.txt" 2>/dev/null || true
traceroute -n "$TARGET_IP" >"$OUT/traceroute_${TARGET_IP}.txt" 2>/dev/null || true

# DNS queries
dig "$TARGET_HOST" +trace +time=2 +tries=1 >"$OUT/dig_trace_${TARGET_HOST}.txt" || true
dig A "$TARGET_HOST" >"$OUT/dig_a_${TARGET_HOST}.txt" || true

# HTTP checks
curl -sS -o /dev/null -w "http_code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" "http://$TARGET_HOST" >"$OUT/http_${TARGET_HOST}.txt" || true
curl -sS -o /dev/null -w "http_code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" "https://$TARGET_HOST" >"$OUT/https_${TARGET_HOST}.txt" || true
curl -v --max-time 5 "https://$TARGET_HOST" >"$OUT/https_verbose_${TARGET_HOST}.txt" 2>&1 || true

# NetworkManager (if present)
nmcli -t device status >"$OUT/nmcli_device_status.txt" 2>/dev/null || true
nmcli -t connection show --active >"$OUT/nmcli_conn_active.txt" 2>/dev/null || true

# Summarize key bits as JSON index for AI
jq -n \
  --arg host "$(hostname)" \
  --arg now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --arg target_ip "$TARGET_IP" \
  --arg target_host "$TARGET_HOST" \
  '{host:$host, captured_at_utc:$now, targets:{ip:$target_ip, host:$target_host}, files: ( [inputs] ) }' \
  "$OUT"/*.txt >"$OUT/index.json" 2>/dev/null || true

tar -czf "${OUT}.tar.gz" "$OUT"
echo "Snapshot created: ${OUT}.tar.gz"

Usage:

chmod +x net-snapshot.sh
./net-snapshot.sh

This produces a timestamped tarball you can upload to your AI tool of choice.

Tip: Before sharing snapshots externally, review/scrub sensitive data (IPs, hostnames, tokens) as needed.

2) A 10-minute triage recipe (and what each step tells you)

  • Link and IP status

    • Check carrier, addresses, routes:
    ip -brief addr
ip route
ip -s link
    • If no default route or no address: it’s a local config or DHCP issue.

  • DNS vs. network

    • IP works but name fails ⇒ DNS problem:
    ping -c4 8.8.8.8
ping -c4 example.com
dig example.com
resolvectl status
    • No IP ping either ⇒ path, firewall, or upstream outage.

  • Path and latency

    • Quickly see where loss or jitter appears:
    mtr -rwzc 100 example.com
traceroute -n 8.8.8.8
    • Loss only at an intermediate hop that doesn’t propagate further may be ICMP de-prioritization (not necessarily real loss).

  • Service reachability

    • Distinguish port issues from app issues:
    nmap -Pn -p 80,443 example.com
curl -v --max-time 5 https://example.com
ss -lntu
    • SYN sent but no SYN-ACK ⇒ firewall or ACL. TLS handshake stalls ⇒ MTU/PMTU or middlebox inspection sometimes to blame.

  • When to capture packets

    • Keep it surgical and short:
    sudo tcpdump -i any -c 200 'port 53' -vv
sudo tcpdump -i any -c 200 'tcp port 443 and (tcp[tcpflags] & (tcp-syn|tcp-ack) != 0)' -nn
sudo tcpdump -i any -c 100 'icmp or icmp6' -nn
    • Always get approval on production systems and store captures securely.

3) Let AI summarize and suggest next steps

Give the model a clear job: summarize, classify the failure mode, and propose focused tests or fixes.

Create a prompt template prompt.md:

You are a Linux network SRE. Analyze the attached snapshot.
1) Summarize symptoms and key anomalies.
2) Classify: DNS vs path vs port/firewall vs MTU/PMTU vs app.
3) Propose 3–5 next diagnostic steps with exact commands.
4) If likely cause is found, propose a safe, minimal change.

Context:

- Host: {{HOSTNAME}}

- Targets: {{TARGET_IP}} / {{TARGET_HOST}}

- Important files: index.json references all included text files.

- Treat intermittent ICMP “loss on hop but not downstream” as non-fatal unless corroborated.

- Call out any MTU risks (VPN/tunnel), asymmetric routing signs, or DNS misconfig.

Example of a generic CLI wrapper you can adapt to your AI platform (ask_ai.sh). Set AI_ENDPOINT and AI_TOKEN according to your provider.

#!/usr/bin/env bash
set -euo pipefail
SNAP="${1:?Usage: ask_ai.sh SNAPSHOT_DIR}"
PROMPT_FILE="${2:-prompt.md}"

# Build a single prompt with small snippets (truncate big logs)
SUMMARY=$(jq -r '.host, .captured_at_utc, .targets.ip, .targets.host' "$SNAP/index.json" 2>/dev/null || true)
EXCERPTS=$(for f in "$SNAP"/*.txt; do
  echo "===== $(basename "$f") ====="
  head -n 80 "$f"
done)

PAYLOAD=$(jq -n --arg content "$(sed 's/"/\\"/g' "$PROMPT_FILE")\n\nHost summary:\n$SUMMARY\n\nExcerpts:\n$EXCERPTS" \
  '{model:"your-model",messages:[{role:"system",content:"You are a senior Linux network engineer."},{role:"user",content:$content}],temperature:0.2}')

curl -sS -H "Authorization: Bearer $AI_TOKEN" -H "Content-Type: application/json" \
  -d "$PAYLOAD" "$AI_ENDPOINT"

Usage:

./net-snapshot.sh
tar -xzf net-snapshot_*.tar.gz
./ask_ai.sh net-snapshot_<host>_<timestamp> prompt.md

This keeps your workflow provider-agnostic. You can swap in any local or remote model with a REST interface.

4) Real-world patterns AI can spot quickly

  • DNS misconfiguration (classic)

    • Symptoms: ping 8.8.8.8 works; ping example.com fails; dig times out; resolvectl status shows unreachable nameserver.
    • Fix ideas:
    sudo resolvectl dns eth0 1.1.1.1 8.8.8.8
sudo resolvectl flush-caches
# or edit /etc/resolv.conf if not managed by systemd-resolved/NetworkManager

  • MTU/PMTU blackhole (common with VPN/tunnels)

    • Symptoms: ICMP and small HTTP OK, HTTPS/TLS stalls; curl -v shows connect ok but no handshake progress; MTR looks clean.
    • Validation:
    ping -M do -s 1472 8.8.8.8   # IPv4 DF test (adjust size)
ip link show | grep -i mtu
    • Workarounds:
    sudo ip link set dev eth0 mtu 1400
# or enable TCP MTU probing:
echo 1 | sudo tee /proc/sys/net/ipv4/tcp_mtu_probing

  • Port blocked by firewall or upstream ACL

    • Symptoms: nmap -p443 host shows filtered/closed; curl -v shows SYN retries; path ping fine.
    • Checks:
    nmap -Pn -p 443 <target>
sudo tcpdump -i any -c 50 'tcp port 443' -nn
    • Action: Review local firewall or upstream security group; confirm return path.

  • Asymmetric routing/NAT confusion

    • Symptoms: Outbound seems fine; returns don’t arrive; MTR from you looks ok but service never completes.
    • Clues: Source IP not expected by peer; unexpected SNAT; mismatched routes.
    ip route get <target_ip>
ss -tnp | grep ':443'

5) Make it repeatable and safe

  • Automate where possible: store net-snapshot.sh in your dotfiles or a team repo.

  • Keep snapshots small: cap line counts, prefer summaries, avoid full packet captures unless needed.

  • Scrub secrets: hostnames, IPs, URLs, tokens—redact before sharing.

  • Document the deltas: AI is most helpful when it can compare “last known good” to “now.”

Conclusion and next steps

With a compact Bash snapshot and a clear AI prompt, you can compress hours of wandering into minutes of directed action. You still own the keyboard and the changes; the model accelerates the heavy lifting of pattern matching and hypothesis generation.

Your next step: 1) Install the tools above. 2) Drop net-snapshot.sh and ask_ai.sh into your toolkit. 3) On your next incident, run the snapshot immediately and feed it to your AI assistant. 4) Iterate: add or trim checks based on your environment.

If you found this useful, standardize the snapshot across your fleet and wire it into your incident runbooks. Your future 2 a.m. self will thank you.

Related posts