Documenting Your AI Network: A Bash-First Playbook for Repeatable, Reliable Performance

You spent six figures on GPUs—don’t let a poorly documented network be the bottleneck. AI workloads are brutal on east–west traffic, sensitive to tiny misconfigurations (MTU, ECN, QoS), and notoriously hard to troubleshoot when documentation drifts. Good network documentation turns firefights into playbooks, accelerates onboarding, and makes performance predictable.

This guide explains what to document, why it matters for AI, and gives you Bash-first, cross-distro steps you can run today. You’ll leave with templates, test commands, and a lightweight tooling stack (with apt, dnf, and zypper install lines) to make your AI network documentation living, accurate, and useful.

---

Why AI Network Documentation Is Non-Negotiable

• AI traffic is different:

  • Massive east–west flows (GPU collective/comms).
  • High, sustained bandwidth to storage backends.
  • Latency-sensitive RPC/control planes.

• Small mismatches, big fallout:

  • MTU inconsistency silently kills throughput and inflates CPU.
  • ECN/QoS misalignment causes queueing and tail latencies.
  • Unclear VLAN/VXLAN design leads to noisy neighbors and security gaps.

• Documentation is an operational multiplier:

  • Faster incident resolution with known-good baselines and runbooks.
  • Easier audits and capacity planning.
  • Stronger collaboration between NetOps, SRE, and ML engineers.

---

Install the Lightweight Tooling

These packages let you inventory, baseline, and validate your AI network. Install with your distro’s package manager.

• iperf3: throughput/latency benchmarking

• tcpdump: packet capture and inspection

• ethtool: NIC and driver details, statistics

• nmap: quick reachability and port checks

• lldpd: LLDP neighbors to map wiring

• bmon: live bandwidth monitoring

• jq: parse JSON from ip -j, lldpcli, etc.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y iperf3 tcpdump ethtool nmap lldpd bmon jq
sudo systemctl enable --now lldpd

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y iperf3 tcpdump ethtool nmap lldpd bmon jq
sudo systemctl enable --now lldpd

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y iperf3 tcpdump ethtool nmap lldpd bmon jq
sudo systemctl enable --now lldpd

---

What “Good” AI Network Documentation Looks Like

Start a repo (e.g., ai-network-docs/) where Markdown is your source of truth. Include:

• Topology and fabrics:

  • Physical diagram: spine/leaf, uplinks, dual-homing.
  • Logical fabrics: training, storage, management (VLANs/VXLANs).

• Addressing/Routing:

  • IP plans, VRFs, gatewaying strategy, BGP/ECMP notes.

• Traffic engineering:

  • MTU policy, ECN/QoS markings, DCB/RoCEv2 (if applicable).

• Change and incident runbooks:

  • Standard MTU/QoS change procedures, validation steps, rollback.

• Benchmarks and baselines:

  • iperf3 matrices, storage throughput, NIC stats snapshots.

• Monitoring hooks:

  • What to watch, where it’s scraped, alert thresholds.

Use the template below to seed your repo.

# AI Network Documentation (Cluster: <name>)

## 1) Overview

- Fabrics: Training (VLAN 110), Storage (VLAN 120), Management (VLAN 10)

- MTU Policy: 9000 everywhere (hosts, switches, storage)

- QoS/ECN: ECN enabled on training fabric, DSCP AF31 for storage

## 2) Topology

- Spine/Leaf: 2x spine, 4x leaf (100G down, 400G up)

- Dual-homed servers (LAG/LACP): yes (LACP fast timers)

## 3) Addressing/Segmentation

- VLAN 110 (Training): 10.110.0.0/16, GW 10.110.0.1

- VLAN 120 (Storage): 10.120.0.0/16, GW 10.120.0.1

- VLAN 10 (Mgmt): 10.10.0.0/16, GW 10.10.0.1

## 4) Host Standards

- MTU: 9000 (persistent)

- NIC Offloads: GRO on, LRO off, tso/gso on (validate per NIC)

- RoCEv2: if used, DCQCN enabled

## 5) Benchmarks (Last run: 2026-07-01)

- iperf3 (Training fabric): 8 streams x 30s, P95 = 94 Gbps per link

- Storage (NFS): 25G link saturates at 23.5 Gbps read, 21.8 Gbps write

## 6) Runbooks

- MTU Change: steps, validation, rollback

- Packet Loss: triage script, counters to check

- Congestion: ECN validation, queue stats

## 7) Monitoring

- Exporters: node-exporter, switch telemetry (sFlow/Streaming)

- Dashboards: Net errors, pause frames, ECN marks, queue depth

## 8) Change Log

- 2026-06-25: Enabled jumbo frames on VLAN 110 end-to-end

---

5 Actionable Steps With Bash You Can Run Today

1) Map Your AI Traffic Domains and Make Them Explicit

• Separate at least three planes:

  • Training/collectives (east–west, high BW).
  • Storage (consistent throughput).
  • Management/control (low BW, latency tolerant).

• Assign VLANs/VXLANs and CIDRs; document who talks to what and why.

• Validate reachability and segmentation quickly:

# Quick sanity checks from a compute node
ip -br addr
ip route
nmap -Pn -p 2049 <storage_ip>     # e.g., NFS/DAOS endpoint
nmap -Pn -p 22 <peer_mgmt_ip>     # mgmt plane SSH reachability

2) Build a Living Inventory (Drivers, Firmware, Neighbors) With One Script

Keep wiring and NIC details fresh and verifiable. Run this on each node and commit outputs to your repo.

#!/usr/bin/env bash
# ai-net-inventory.sh: Emit CSV of NIC details + LLDP neighbors
set -euo pipefail

echo "host,iface,state,mtu,speed,driver,firmware,pcibus,mac,ipv4,lldp_neighbor"

HOST="$(hostname -s)"
for IF in $(ls /sys/class/net | grep -vE 'lo|veth|docker|virbr|br-|tap|tun'); do
  STATE=$(cat /sys/class/net/$IF/operstate || echo unknown)
  MTU=$(ip -j link show "$IF" | jq -r '.[0].mtu')
  SPEED=$(ethtool "$IF" 2>/dev/null | awk -F': ' '/Speed:/ {print $2}' | tr -d ' ')
  DRV=$(ethtool -i "$IF" 2>/dev/null | awk -F': ' '/driver:/ {print $2}')
  FW=$(ethtool -i "$IF" 2>/dev/null | awk -F': ' '/firmware-version:/ {print $2}')
  BUS=$(ethtool -i "$IF" 2>/dev/null | awk -F': ' '/bus-info:/ {print $2}')
  MAC=$(ip -j link show "$IF" | jq -r '.[0].address')
  IP4=$(ip -4 -br addr show "$IF" | awk '{print $3}')
  # Try to fetch LLDP neighbor sysname for this port
  LLDP=$(lldpcli -f keyvalue show neighbors ports "$IF" 2>/dev/null \
        | awk -F'= ' '/lldp.rem.sysname/ {print $2}' | head -1)
  echo "$HOST,$IF,$STATE,$MTU,$SPEED,$DRV,$FW,$BUS,$MAC,${IP4:-},${LLDP:-}"
done

Run it and save:

chmod +x ai-net-inventory.sh
./ai-net-inventory.sh | tee inventory-$(hostname -s).csv

Tip: aggregate across nodes with a simple fan-out (Ansible, pssh, or a for-loop over SSH), then commit to ai-network-docs/inventory/.

3) Baseline Performance and Record the Results

Use iperf3 to establish a fabric baseline you can compare over time or after changes.

On the server (destination):

iperf3 -s

On the client (source):

# Single stream baseline (TCP)
iperf3 -c <server_ip> -t 30

# Parallel streams to approach line rate (e.g., 8)
iperf3 -c <server_ip> -P 8 -t 30

# UDP test to observe loss/jitter at high rates (careful in prod)
iperf3 -u -b 90G -c <server_ip> -t 15

Automate a quick matrix:

#!/usr/bin/env bash
SERVER="$1"; : "${SERVER:?usage: $0 <server_ip>}"
for P in 1 4 8 16; do
  echo "--- P=$P ---"
  iperf3 -c "$SERVER" -P "$P" -t 20 | tee -a iperf3-baseline.txt
done

Record NIC stats before/after to catch errors or pause frames:

IF=eth0
ethtool -S $IF | tee ethtool-${IF}-before.txt
# ...run tests...
ethtool -S $IF | tee ethtool-${IF}-after.txt

4) Make MTU and QoS Policies Real (and Verifiable)

• MTU: AI east–west and storage often benefit from jumbo frames (e.g., 9000). Apply end-to-end or don’t apply at all.

Set runtime MTU (test first):

sudo ip link set dev eth0 mtu 9000
ip -d link show dev eth0 | grep mtu

Persist it using your distro’s network manager (document the method you use):

• Netplan (Ubuntu), NetworkManager (nmcli), systemd-networkd, wicked (SUSE), or ifcfg scripts (RHEL). Always capture the exact file and PR the change to your repo.

• QoS/DSCP: Mark known traffic classes (example: storage on TCP/2049 as AF21). Verify with tcpdump.

Mark (iptables mangle example):

# Mark NFS client egress as AF21 (DSCP 0x12)
sudo iptables -t mangle -A OUTPUT -p tcp --dport 2049 -j DSCP --set-dscp 0x12
sudo iptables -t mangle -A PREROUTING -p tcp --sport 2049 -j DSCP --set-dscp 0x12

Verify DSCP is present on the wire:

sudo tcpdump -i eth0 -vvv -n 'tcp port 2049' -c 5
# Look for "tos 0x48" (example) and map to DSCP; record a capture snippet in docs

If you run RoCEv2, document and validate:

• UDP 4791 presence, ECN markings, DCB/DCQCN settings, and lossless queueing policy on switches.

• Capture example (non-invasive sample):

sudo tcpdump -i eth0 -nn -c 10 'udp port 4791'

5) Create Runbooks for the Top 3 Failure Modes

Codify what “good” looks like and how to recover when it’s not.

Example: “Throughput unexpectedly low on training fabric”

Symptom:

- iperf3 P=8 peaks at < 40 Gbps on 100G link

Checks:

- Confirm MTU everywhere:
  ip link show eth0 | grep mtu
  ping -M do -s 8972 <peer_ip>   # Jumbo ping (ICMP payload + headers ~ 9000)

- Look for NIC errors:
  ethtool -S eth0 | egrep 'err|drop|pause|disc'

- Verify LACP is up (if bonded):
  cat /proc/net/bonding/bond0

Remediation:

- Align MTU on host/switch/storage to 9000

- Clear conflicting offloads if NIC/driver bug suspected:
  sudo ethtool -K eth0 gro on lro off

- Re-run baseline and record results

Rollback:

- Revert MTU to 1500 and remove QoS rules:
  sudo ip link set dev eth0 mtu 1500
  sudo iptables -t mangle -F

Save each runbook in docs/runbooks/, including commands, expected outputs, and rollback.

---

Real-World Example: 8-Node, Dual-Fabric Cluster

• Topology:

  • 8 x GPU servers
  • Dual 100G (VLAN 110) for training
  • 25G (VLAN 120) for storage
  • 1G (VLAN 10) for management
  • MTU 9000 on VLANs 110/120, 1500 on mgmt

• Baseline:

  • iperf3 P=8 sustained 94–96 Gbps per link on training fabric
  • NFS over 25G at 23.5 Gbps read, 21.8 Gbps write

• Incident (and fix) captured in docs:

  • Symptom: training throughput halved on two nodes
  • Root cause: switch trunk MTU at 1500 for those ports
  • Fix: corrected trunk MTU to 9216, revalidated jumbo ping and iperf3
  • Lesson: added LLDP neighbor and MTU audit script to CI job

---

Conclusion and Call To Action

Your AI network is as strong as your documentation. Start today: 1) Create an ai-network-docs/ repo and commit the template. 2) Install the tooling (iperf3, tcpdump, ethtool, nmap, lldpd, bmon, jq). 3) Run the inventory script on all nodes and check in the CSVs. 4) Baseline iperf3 across fabrics; save outputs. 5) Write one runbook for your most likely failure mode.

Next step: Schedule a monthly “network health” job to regenerate inventory and baselines, and open a pull request with diffs. When the next incident hits, you’ll have facts, not guesses.