Artificial Intelligence Network Capacity Planning: A Bash-first Playbook

AI workloads don’t just burn GPU cycles—they guzzle network bandwidth. If your cluster’s network planning is off, you’ll watch eight-figure accelerators sit idle while a single 25GbE link blinks green. The fix isn’t magic hardware; it’s disciplined capacity planning using the Linux tools you already know.

This article shows you how to baseline, model, size, and tune your network for AI training and inference using Bash-friendly tooling. You’ll get actionable steps, real commands, and distro-agnostic install instructions.

Why AI network capacity planning is different (and urgent)

• East–west traffic dominates: Training jobs synchronize gradients (all-reduce, parameter servers) across nodes. That’s high-throughput, bursty, and latency-sensitive.

• Microbursts and incast happen: Many workers send at once, creating short-lived queues that crush buffers and drop packets.

• Storage-to-GPU ingest grows fast: Data pipelines stream gigabytes per second from object stores or parallel filesystems.

• Underestimation is expensive: 10% under-provisioning can mean 50% longer epochs or repeated job failures.

The takeaway: You can’t “eyeball” this. You need measurements, emulation, and back-of-the-envelope math—before buying links or blaming the fabric.

Tools you’ll need (with install commands)

We’ll use standard Linux utilities. Install them with your package manager:

• iperf3 (throughput/streams), bmon or nload (live usage), iftop (per-host flows), vnstat (long-term trends), sysstat (sar), ethtool (NIC stats), iproute2/iproute (tc, ss, ip)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y iperf3 bmon iftop nload vnstat sysstat ethtool iproute2

Fedora/RHEL/CentOS Stream (dnf):

# RHEL may need EPEL for some packages:
# sudo dnf install -y epel-release
sudo dnf install -y iperf3 bmon iftop nload vnstat sysstat ethtool iproute

openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y iperf3 bmon iftop nload vnstat sysstat ethtool iproute2

Note: Some commands below require root. Test carefully—network changes can disrupt connectivity.

---

Step 1: Establish a baseline (before you change anything)

Answer two questions: What is my actual utilization? Where are packets dropping?

Quick live views:

# Interface-level utilization every second
sar -n DEV 1

# By-application/host flows (press t for sorting, q to quit)
sudo iftop -i eth0

# Terminal-friendly graph
bmon

Check kernel and NIC counters for loss, overruns, and softnet backlog:

# Drops, errors, overruns by interface
ip -s link show dev eth0

# NIC hardware stats (look for rx_dropped, rx_no_buffer_count, tx_timeout)
sudo ethtool -S eth0 | egrep 'drop|errors|buffer|timeout|miss|pause'

# Kernel network stack counters (TCP resets, retransmits, etc.)
nstat -az | egrep 'TcpRetransSegs|TcpInSegs|TcpOutSegs|IpInDiscards|IpInDelivers'

# Per-CPU softnet stats (column 2 = dropped)
cat /proc/net/softnet_stat

Start long-term logging:

# Enable sar data collection (Debian/Ubuntu)
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat || true
sudo systemctl enable --now sysstat

# vnstat DB and daemon
sudo systemctl enable --now vnstat
vnstat -l -i eth0   # live sampling
vnstat -d           # daily summary after some hours

Export a quick 5-minute snapshot to a CSV for later comparisons:

echo "ts,iface,rx_kbps,tx_kbps" > net_snap.csv
for i in {1..300}; do
  date +%s | tr -d '\n' >> net_snap.csv
  sar -n DEV 1 1 | awk '/eth0/ {print ","$2","$5","$6}' >> net_snap.csv
done

What to look for:

• Average utilization >50% during jobs, or spikes to 100%

• Any rx/tx drops, softnet drops, or retransmits increasing during peak periods

• Single flows capped (e.g., 3–4 Gbps on 25GbE due to small MTU or RTOs)

---

Step 2: Emulate AI traffic patterns

AI training uses many concurrent streams with bursty sync phases. Model that with iperf3.

Run a server on one node:

iperf3 -s

Drive multiple parallel streams from a client:

# 16 parallel TCP streams for 60s
iperf3 -c SERVER_IP -P 16 -t 60

Bidirectional or reverse direction (useful for duplex testing):

iperf3 -c SERVER_IP -P 16 -t 60 -R

Fan-in/fan-out (incast): run iperf3 clients simultaneously from many workers to one server.

Emulate WAN or congested fabric characteristics:

# Add 1ms latency and 0.1% loss to test sensitivity
sudo tc qdisc add dev eth0 root netem delay 1ms loss 0.1%

# Remove it later
sudo tc qdisc del dev eth0 root netem

Capture queue stats while running tests:

sudo tc -s qdisc show dev eth0

Expectation:

• For 25GbE, a well-tuned host with MTU 9000 should see per-node >20 Gbps aggregate on parallel streams.

• If you only get a few Gbps, you’re probably hitting MTU, offload, or CPU/IRQ limits.

---

Step 3: Do the math (right-size links with headroom)

Use simple assumptions to avoid GPU starvation:

• Target max steady-state utilization per link: 70% (headroom for bursts)

• Training all-reduce overhead: roughly (N-1)/N of model/gradient size per step on ring all-reduce (varies with algo)

• Ingestion and checkpoint I/O often overlap with training sync phases

Example: 8 GPUs per node, 8 nodes, 25GbE per node

• Measured per-node steady load during step: 10 Gbps Tx/Rx, with bursts to 22 Gbps

• Sizing with headroom: 22 Gbps / 0.7 ≈ 31 Gbps → a single 25GbE link will burst-cap and drop; consider 2x25GbE (LAG) or 100GbE

Quick Bash to estimate needed link rate:

# usage: need_rate_gbps <peak_gbps> [util_target]
need_rate_gbps() {
  peak=${1:-20}; util=${2:-0.7}
  awk -v p="$peak" -v u="$util" 'BEGIN { printf "%.1f\n", p/u }'
}
need_rate_gbps 22 0.7   # -> 31.4 Gbps

Back-of-envelope for inference:

• QPS * average response bytes / (seconds) → required egress

• Multiply by 1.1–1.2 for headers/overheads

• Keep p99 latency SLO by staying under 60–70% link utilization

---

Step 4: Tune hosts for AI-style traffic

Caution: Test these changes in a maintenance window.

Enable jumbo frames end-to-end:

# Check current MTU
ip link show dev eth0

# Set to 9000 (must be supported by switches and peer NICs)
sudo ip link set dev eth0 mtu 9000

Use modern queue disciplines to smooth bursts:

# Fair queueing to reduce head-of-line blocking
sudo tc qdisc replace dev eth0 root fq
sudo tc -s qdisc show dev eth0

Turn on ECN to combat congestion without loss:

# Enable Explicit Congestion Notification
sudo sysctl -w net.ipv4.tcp_ecn=1
# Persist it:
echo "net.ipv4.tcp_ecn=1" | sudo tee /etc/sysctl.d/99-ecn.conf

Ensure NIC offloads are sensible (GRO/LRO, TSO):

# Review current features
sudo ethtool -k eth0

# Example: enable GRO/TSO
sudo ethtool -K eth0 gro on tso on

Balance interrupts and scale RX:

# Make sure irqbalance is running
sudo systemctl enable --now irqbalance 2>/dev/null || true

# Receive Packet Steering (adjust CPU mask to your cores)
echo ffff | sudo tee /proc/sys/net/core/rps_sock_flow_entries >/dev/null
for Q in /sys/class/net/eth0/queues/rx-*; do echo 4096 | sudo tee $Q/rps_flow_cnt; done

Right-size queues and txqueuelen:

# Increase transmit queue length for bursty workloads
sudo ip link set dev eth0 txqueuelen 20000

# NIC ring buffers (verify supported values)
sudo ethtool -g eth0
sudo ethtool -G eth0 rx 4096 tx 4096

Re-test with iperf3 after each change to attribute gains.

---

Step 5: Make it continuous (monitor, alert, iterate)

Use vnstat and sar for trends; alert on early signals of pain:

# Alert if softnet drops increase
prev=$(awk '{print $2}' /proc/net/softnet_stat | paste -sd+ - | bc)
sleep 60
curr=$(awk '{print $2}' /proc/net/softnet_stat | paste -sd+ - | bc)
if [ "$curr" -gt "$prev" ]; then
  echo "WARN: softnet drops increased from $prev to $curr in the last minute" >&2
fi

Daily health snapshot (cron it):

#!/usr/bin/env bash
date
ip -s link show dev eth0
ethtool -S eth0 | egrep 'drop|error|timeout|pause' || true
nstat -az | egrep 'TcpRetransSegs|IpInDiscards'
vnstat -d

Track job timing alongside link utilization. If epochs or p99 latency spike at the same time as tx/rx saturation or retransmits, you’ve found a bottleneck.

---

Real-world patterns to recognize

• Training slowdown after moving to larger batch sizes: Bursts exceed small MTU and limited NIC ring sizes, causing drops and TCP backoff. Fix: MTU 9000, increase NIC rings, add fq qdisc, and move to 2x25GbE LAG or 100GbE.

• Inference tail latencies at p99.9: Microbursts + shallow switch buffers. Fix: Enable ECN end-to-end, use fq/fq_codel, ensure under 70% steady utilization, and split traffic classes if needed.

---

Summary and next steps

• Baseline: Measure before you buy. Use sar, vnstat, ethtool, and nstat to find real headroom and drops.

• Emulate: Use iperf3 with many streams and netem to mirror AI traffic.

• Size: Keep peak usage under ~70% of link rate; plan for bursts and overlapped I/O.

• Tune: Jumbo frames, fq, ECN, sensible offloads, and balanced IRQs go a long way.

• Iterate: Monitor continuously and tie network metrics to job performance.

Call to action: 1) Install the tools above on two nodes. 2) Run the baseline and iperf3 tests. 3) Apply one tuning change at a time and re-test. 4) Document your numbers and decide: stay, tune, or upgrade links.

Your GPUs are fast. Make your network worthy of them.