Posted on
• Artificial Intelligence

Artificial Intelligence Virtual Network Troubleshooting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI-Powered Virtual Network Troubleshooting on Linux (Bash-first)

When your AI workload goes dark at 3 a.m., is it the VM’s vNIC, the OVS bridge, a rogue firewall rule, or a silent MTU mismatch breaking your overlay? Modern AI stacks ride on complex virtual networks—namespaces, bridges, overlays, VXLAN, SR-IOV, you name it. This post gives you a practical, Bash-first playbook to find root causes fast, plus optional ways to let AI assist your triage.

You’ll get:

  • Why this topic matters for AI platforms

  • A toolkit you can install with apt, dnf, and zypper

  • 5 actionable steps, with copy/paste commands

  • A real-world example and a small “SOS” collector script

  • An optional “let AI summarize the mess” workflow

Why virtual networking for AI is tricky (and worth your time)

  • AI pipelines are distributed: datasets in object stores, models on GPU nodes/VMs, vector DBs and feature stores in cluster networks. A single routing or DNS issue can stall the pipeline.

  • Virtualization layers multiply failure modes: Linux bridges, Open vSwitch, namespaces, NAT, CNI overlays, security groups, and hardware offloads can interact in surprising ways.

  • Downtime is expensive: GPU idle time, missed training windows, and delayed inference SLOs add up fast.

Translation: you need a fast, deterministic way to map, measure, and isolate problems. Let’s do that with Bash.

Install the troubleshooting toolkit

Run the commands for your distro. Some packages may already be installed. Optional packages are marked (optional).

Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y \
  iproute2 ethtool tcpdump mtr traceroute nmap netcat-openbsd \
  bridge-utils openvswitch-switch \
  libvirt-daemon-system libvirt-clients qemu-kvm virtinst \
  conntrack dnsutils nftables arping iperf3
# Optional services
sudo systemctl enable --now libvirtd || true
sudo systemctl enable --now openvswitch-switch || true
# Optional: access libvirt without sudo
sudo usermod -aG libvirt,kvm "$USER"

RHEL/Fedora/CentOS Stream (dnf):

sudo dnf install -y \
  iproute ethtool tcpdump mtr traceroute nmap-ncat \
  bridge-utils openvswitch \
  libvirt-client libvirt-daemon-kvm qemu-kvm virt-install \
  conntrack-tools bind-utils nftables arping iperf3
sudo systemctl enable --now libvirtd || true
sudo systemctl enable --now openvswitch || true
sudo usermod -aG libvirt,kvm "$USER"

SUSE/openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y \
  iproute2 ethtool tcpdump mtr traceroute netcat-openbsd \
  bridge-utils openvswitch \
  libvirt-client libvirt-daemon qemu-kvm virt-install \
  conntrack-tools bind-utils nftables arping iperf3
sudo systemctl enable --now libvirtd || true
sudo systemctl enable --now openvswitch || true
sudo usermod -aG libvirt,kvm "$USER"

Log out/in to apply group membership changes.

The 5-step playbook

1) Map the topology fast

Know exactly what connects where before you change anything.

  • Interfaces, bridges, routes:
ip -br link
ip -br addr
ip route show table all
ip rule show
bridge link
bridge vlan
  • Namespaces and veth:
ip netns list
ip -n <ns> -br link
  • Libvirt/KVM:
virsh net-list --all
virsh domiflist <vm>
virsh domifaddr <vm>
  • Open vSwitch:
ovs-vsctl show
ovs-ofctl dump-flows br0
  • Quick container context (if applicable):
docker network ls
podman network ls

Deliverable: a quick diagram or notes of NICs, bridges (linux/OVS), VLANs, tunnels, and routes.

2) Prove the data plane (connectivity, MTU, offloads)

Many AI failures are blunt network issues: blackholes, oversized packets, or checksum problems.

  • Basic link/route sanity:
ip -br link | grep -E "DOWN|LOWERLAYERDOWN|UNKNOWN"
ip route get 8.8.8.8
  • MTU tests (discover PMTU with “do not fragment”):
ping -c3 <target>
ping -M do -s 1472 <target>      # 1500 MTU path test
ping -M do -s 1430 <target>      # common overlay (~1450) test
  • Throughput/latency:
iperf3 -s                       # on server
iperf3 -c <server> -P 4 -t 10   # on client
mtr -rzbc 5 <target>
  • Offloads that can confuse captures (especially with OVS/virtIO):
ethtool -k <iface> | grep -E 'gro|gso|tso'
# Temporarily disable (test only):
sudo ethtool -K <iface> gro off gso off tso off
  • Packet trace (pick the right interface: bridge, tap, or VM vNIC):
sudo tcpdump -i br0 -n -vvv -c 50
sudo tcpdump -i any -n host <vm_ip> and not port 22 -c 100

3) Validate the control plane (ARP/ND, DHCP, DNS, firewall/NAT)

AI apps often fail from “can’t resolve name” or “can’t get IP”.

  • ARP/neighbor discovery:
ip neigh show
arp -an
arping -I <iface> <peer_ip>
  • DHCP:
sudo tcpdump -i <iface> -n port 67 or port 68
  • DNS:
resolvectl status || cat /etc/resolv.conf
dig example.com A
dig @<dns_ip> example.com AAAA +trace
  • Firewall/NAT:
sudo nft list ruleset | less
sudo iptables-save | less
sudo conntrack -L | grep -E 'dport=|sport='

Look for blocked INPUT/FORWARD chains, wrong zones (firewalld), or stale conntrack entries.

4) Reproduce issues in an isolated lab (namespaces)

Recreate symptoms without the rest of the cluster. Namespaces let you iterate quickly.

  • Minimal L2 setup:
sudo ip netns add ns1
sudo ip netns add ns2
sudo ip link add veth1 type veth peer name veth2
sudo ip link set veth1 netns ns1
sudo ip link set veth2 netns ns2
sudo ip -n ns1 addr add 10.10.10.1/24 dev veth1
sudo ip -n ns2 addr add 10.10.10.2/24 dev veth2
sudo ip -n ns1 link set veth1 up
sudo ip -n ns2 link set veth2 up
sudo ip -n ns1 ping -c3 10.10.10.2
  • Simulate MTU mismatch:
sudo ip -n ns1 link set veth1 mtu 1450
sudo ip -n ns2 link set veth2 mtu 1500
sudo ip -n ns1 ping -c1 -M do -s 1472 10.10.10.2   # should fail
sudo ip -n ns1 ping -c1 -M do -s 1430 10.10.10.2   # should succeed
  • Quick app test (TCP):
sudo ip netns exec ns2 ncat -l -k -p 8080 --keep-open --verbose &
sudo ip netns exec ns1 ncat -vz 10.10.10.2 8080

Once reproduced, adjust MTU, routes, or firewall rules and verify the fix here before changing production.

5) Let AI assist your triage (optional)

AI can summarize logs and traces so you focus on the fix. First, collect a reproducible bundle.

  • Save as net-sos.sh and run it:
cat > net-sos.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
OUT="${1:-net-sos.$(hostname).$(date +%F-%H%M%S)}"
mkdir -p "$OUT"
# Logs and state
sudo journalctl -S -2h > "$OUT/journal.log" || true
ip -br link > "$OUT/ip_link.txt"
ip -br addr > "$OUT/ip_addr.txt"
ip route show table all > "$OUT/routes.txt"
ip rule show > "$OUT/rules.txt"
bridge -j link > "$OUT/bridge_link.json" || true
bridge -j vlan > "$OUT/bridge_vlan.json" || true
nft list ruleset > "$OUT/nft.txt" || iptables-save > "$OUT/iptables.txt" || true
ss -tuna > "$OUT/sockets.txt"
arp -an > "$OUT/arp.txt" || true
ip neigh show > "$OUT/neigh.txt"
resolvectl status > "$OUT/resolved.txt" || cat /etc/resolv.conf > "$OUT/resolv.conf"
# Short sample capture (5s or 100 packets)
( timeout 5 tcpdump -i any -s 0 -c 100 -w "$OUT/sample.pcap" ) || true
tar -czf "$OUT.tar.gz" "$OUT"
echo "Wrote $OUT.tar.gz"
EOF
chmod +x net-sos.sh
./net-sos.sh
  • If you want a local LLM to summarize:
    • Install a local runner (example: Ollama). Review the script before running in sensitive environments.
curl -fsSL https://ollama.com/install.sh | sh
  • Summarize targeted files (avoid sensitive data):
awk 'NR<2000' net-sos.*/*journal.log > summary_in.txt || true
echo -e "\n### ip_addr\n" >> summary_in.txt; cat net-sos.*/*ip_addr.txt >> summary_in.txt
echo -e "\n### routes\n" >> summary_in.txt; cat net-sos.*/*routes.txt >> summary_in.txt
echo -e "\n### nft\n" >> summary_in.txt; head -n 200 net-sos.*/*nft.txt >> summary_in.txt 2>/dev/null || true
ollama run llama3:latest < summary_in.txt

Security note: never paste secrets or customer data into AI tools. Redact first.

Real-world example: VM loses DNS behind OVS

Symptom: An inference VM (libvirt on OVS) intermittently fails to resolve names and drops HTTPS to an artifact store.

Findings:

  • ping -M do -s 1472 8.8.8.8 fails; -s 1430 works → MTU problem

  • ethtool -k ens3 shows GRO/TSO enabled; captures show bogus checksums on bridge

  • DNS queries seen on VM, but no replies crossing OVS during large bursts

Fix:

  • Set consistent MTU across vNIC, bridge, and uplink:
sudo ip link set dev br-int mtu 1450
sudo ip link set dev <uplink_nic> mtu 1450
sudo ip link set dev <tap_of_vm> mtu 1450
  • Or enforce clamp via routes (less ideal than fixing interfaces):
sudo ip route change default via <gw> dev br-int mtu 1450
  • While debugging, disable GRO/TSO on relevant interfaces to get accurate captures:
sudo ethtool -K br-int gro off gso off tso off
sudo ethtool -K <uplink_nic> gro off gso off tso off
  • Validate:
ping -M do -s 1430 8.8.8.8
dig github.com A
iperf3 -c <remote> -P 4

Result: DNS and TLS stable; captures show correct checksums; no fragmentation-related drops.

Wrap-up and next steps (CTA)

  • Save this as your “AI virtual network SRE runbook.”

  • Automate: keep the net-sos.sh collector in your bastion and attach bundles to incidents.

  • Standardize MTUs and audit offloads in environments using OVS/overlays.

  • Optionally integrate a local LLM to summarize logs for postmortems.

If you want, I can tailor this playbook to your stack (libvirt-only, OVS+K8s CNI, or bare-metal namespaces) and produce a one-command diagnostics script for your hosts.

Related posts