Posted on
• Artificial Intelligence

Artificial Intelligence Networking Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Networking Checklists: A Bash-First Guide for Linux Operators

Your GPUs might be ready to sprint, but your network could be tying their shoelaces together. In distributed AI training and inference, the network is the invisible multiplier: a mis-set MTU, a flaky driver, or an un-synced clock can easily halve throughput or stall a job. This guide gives you a compact, Bash-friendly checklist to validate and tune your Linux networking stack for AI workloads—along with real commands you can paste into your terminal.

What you’ll get:

  • Why networking is the silent bottleneck in AI

  • A practical, repeatable checklist you can run in minutes

  • Actionable Bash commands and a one-file script to automate checks

  • Installation instructions for apt, dnf, and zypper for every tool we cite

Why this matters

  • AI training is an all-reduce machine. When gradients shuttle between GPUs and nodes, bandwidth and latency dominate step time, not just FLOPs.

  • Clusters often degrade silently. A single node with a mismatched MTU or an outdated NIC firmware can drag down an entire job.

  • Repeatability beats hero debugging. A checklist you can run after imaging, before a training run, and after changes prevents regressions and speeds up root cause analysis.

Install the tooling (once per node)

The commands below install everything used in this checklist: low-level NIC tools, throughput testers, RDMA diagnostics, and time sync.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y iperf3 ethtool pciutils lshw pssh rdma-core perftest infiniband-diags chrony ufw iputils-ping

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y iperf3 ethtool pciutils lshw pssh rdma-core perftest infiniband-diags chrony firewalld iputils

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y iperf3 ethtool pciutils lshw pssh rdma-core perftest infiniband-diags chrony firewalld iputils

Optional service enablement:

# Time sync
sudo systemctl enable --now chrony

# If you use firewalls
# Ubuntu/Debian:
sudo ufw enable || true
# Fedora/openSUSE:
sudo systemctl enable --now firewalld

The AI Networking Checklist (with Bash you can copy-paste)

Pick 3–5 items based on your environment. If you use RDMA/InfiniBand/RoCE, don’t skip item 3.

1) Inventory and link health: speed, driver, firmware, MTU

Goal: Verify every NIC that will carry AI traffic is up, at expected speed (25/40/100/200/400G), with correct driver/firmware and consistent MTU.

Identify NICs and drivers:

lspci | grep -i -E 'ethernet|infiniband'
for dev in $(ls /sys/class/net | grep -v lo); do
  echo "=== $dev ==="
  ip -o link show "$dev" | awk -F': ' '{print $2}'
  ethtool "$dev" | grep -E 'Speed:|Duplex:|Link detected:'
  ethtool -i "$dev" | grep -E 'driver|version|firmware-version'
done

Check and set MTU (example: jumbo frames 9000):

# Show current
ip -o link | awk -F': ' '{print $2}' | xargs -I{} sh -c 'echo -n "{} "; ip link show "{}" | grep -o "mtu [0-9]*"'

# Set MTU (persist via your distro’s network manager)
sudo ip link set dev <iface> mtu 9000

Quick real-world gotcha:

  • Symptom: NCCL all-reduce 2–4x slower on a subset of nodes.

  • Cause: One TOR switch uplink stayed at MTU 1500 while hosts were 9000. Ping with DF instantly revealed it.

Validate end-to-end MTU with “don’t fragment”:

# For 9000 MTU on Ethernet, payload ~8972 works (accounting for headers)
ping -M do -s 8972 <peer-ip-or-dns>

If this fails, there’s an MTU mismatch somewhere along the path.

2) Baseline bandwidth and latency with iperf3 and ping

Goal: Establish a clean, repeatable measurement of throughput and packet loss between peers (or all-to-all for small clusters).

Run an iperf3 server on one node:

iperf3 -s
# If firewalls are on:
# Ubuntu/Debian: sudo ufw allow 5201/tcp
# Fedora/openSUSE: sudo firewall-cmd --add-port=5201/tcp --permanent && sudo firewall-cmd --reload

Run a client from a peer:

iperf3 -c <server> -P 8 -t 30        # parallel streams for throughput
iperf3 -c <server> -u -b 0 -t 15     # UDP to observe drops/jitter

Test latency and loss:

ping -c 50 <peer>

Fan-out test (parallel from many nodes):

# hosts.txt contains hostnames/IPs
parallel-ssh -h hosts.txt -i "iperf3 -c <server> -P 8 -t 20"

Rules of thumb:

  • 100G NICs should push ~90–95 Gbps TCP in clean lab conditions; production might be lower but consistent across nodes.

  • Any node >10% off the pack needs investigation.

3) RDMA/InfiniBand/RoCE sanity (if applicable)

Goal: Verify verbs stack is loaded, links are active, and lossless config is sane. This is critical for NCCL/RDMA collectives.

Check devices and state:

rdma link show
ibv_devinfo | egrep 'hca_id|fw_ver|vendor_part_id|phys_port_cnt' || true

Quick microbenchmarks (requires perftest):

# One node as server (choose the correct HCA port via -d/-i if needed)
ib_write_bw
# Peer node as client
ib_write_bw <server-hostname-or-ip>

# Latency (one-way and round-trip characteristics)
ib_send_lat <server>

GPU/NIC NUMA and topology insight:

command -v nvidia-smi && nvidia-smi topo -m || echo "nvidia-smi not found"

Look for GPU↔NIC “PIX/PHB” proximity; remote NUMA access penalties can be severe.

Note: For RoCE, ensure DCB/PFC and ECN are configured on switches and hosts. Packet loss will crater performance even if link speed looks fine.

4) Offloads and TCP buffers: tune, test, verify

Goal: Avoid pathological offload interactions and ensure buffers aren’t capping throughput.

Inspect offloads:

ethtool -k <iface> | egrep 'tx-checksumming|rx-checksumming|tso|gso|gro|lro'

If troubleshooting strange performance (e.g., asymmetric throughput), test with GRO/LRO off temporarily:

sudo ethtool -K <iface> gro off lro off gso off tso off
# Re-test iperf3, then restore:
sudo ethtool -K <iface> gro on lro on gso on tso on

Set buffer ceilings (persistent):

sudo tee /etc/sysctl.d/99-ainet.conf >/dev/null <<'EOF'
net.core.rmem_max = 1073741824
net.core.wmem_max = 1073741824
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_rmem = 4096 87380 1073741824
net.ipv4.tcp_wmem = 4096 65536 1073741824
net.ipv4.tcp_congestion_control = bbr
EOF
sudo sysctl --system

Validate active settings:

sysctl net.core.rmem_max net.core.wmem_max net.ipv4.tcp_congestion_control

5) Time sync, DNS, and firewall hygiene

Goal: Keep logs correlated, TLS valid, and cluster communication unblocked.

Time sync with chrony:

chronyc tracking
chronyc sources -v

Aim for sub-100 µs in PTP-driven environments; sub-ms with NTP is typical.

DNS and routing basics:

getent hosts <peer>
ip route show default

Firewall allowances (cluster-internal):

# iperf3 example ports
# Ubuntu/Debian:
sudo ufw allow 5201/tcp

# Fedora/openSUSE:
sudo firewall-cmd --add-port=5201/tcp --permanent
sudo firewall-cmd --reload

For AI frameworks (e.g., NCCL, PyTorch DDP), prefer allowing intra-cluster traffic on your fabric interface/subnet rather than “allow all.” Pin interfaces with env vars like:

export NCCL_SOCKET_IFNAME=eno1,ens5f0

One-file Bash: ainetcheck.sh

Drop this script on any node to print a fast, human-readable status report. Add or trim checks to fit your stack.

#!/usr/bin/env bash
set -euo pipefail

echo "== Host: $(hostname) =="
date

echo -e "\n== NICs, speed, driver, firmware, MTU =="
for dev in $(ls /sys/class/net | grep -v lo); do
  echo "--- $dev ---"
  ip -o link show "$dev" | awk -F': ' '{print $2}'
  ethtool "$dev" 2>/dev/null | grep -E 'Speed:|Duplex:|Link detected:' || true
  ethtool -i "$dev" 2>/dev/null | grep -E 'driver|version|firmware-version' || true
done

echo -e "\n== Offloads (subset) =="
for dev in $(ls /sys/class/net | grep -v lo); do
  echo "--- $dev ---"
  ethtool -k "$dev" 2>/dev/null | egrep 'tx-checksumming|rx-checksumming|tso|gso|gro|lro' || true
done

echo -e "\n== RDMA devices (if any) =="
rdma link show 2>/dev/null || echo "rdma tool not present or no devices"
command -v ibv_devinfo >/dev/null && ibv_devinfo | egrep 'hca_id|fw_ver|vendor_part_id|phys_port_cnt' | uniq || true

echo -e "\n== GPU/NIC topology (if NVIDIA GPUs) =="
command -v nvidia-smi >/dev/null && nvidia-smi topo -m || echo "nvidia-smi not found"

echo -e "\n== MTU reachability hint =="
default_gw=$(ip route show default 2>/dev/null | awk '{print $3}' | head -n1)
[ -n "${default_gw:-}" ] && echo "Default GW: $default_gw"
echo "To validate jumbo MTU to a peer:"
echo "  ping -M do -s 8972 <peer>"

echo -e "\n== TCP/Buffer tuning =="
sysctl net.core.rmem_max net.core.wmem_max net.ipv4.tcp_congestion_control 2>/dev/null || true

echo -e "\n== Time sync (chrony) =="
command -v chronyc >/dev/null && { chronyc tracking || true; } || echo "chronyc not installed"

echo -e "\n== DNS & default route =="
getent hosts localhost || true
ip route show default || true

echo -e "\nDone."

Usage:

chmod +x ainetcheck.sh
./ainetcheck.sh

Putting it all together: a quick runbook

  • After provisioning a node: run ainetcheck.sh and fix any red flags (link down/speed/firmware/MTU/time).

  • Before a big training job: iperf3 pairwise across workers; record baseline throughput.

  • If performance regresses: compare against your last saved report; check MTU path and RDMA microbenchmarks first.

  • After switch or driver changes: re-run the checklist and keep a dated log.

Conclusion and next steps (CTA)

A strong AI network doesn’t happen by accident—it’s verified. Turn this checklist into your team’s standard operating procedure:

  • Install the tools on all nodes using the commands above.

  • Save and customize ainetcheck.sh for your environment.

  • Run it after imaging, before every major job, and after any infra change.

  • Keep a simple baseline log per cluster to spot regressions fast.

Want a deeper dive? Extend the script to:

  • Capture iperf3 results across all node pairs

  • Validate NCCL env and fabric selection

  • Snapshot switch interface counters via your NMS or APIs

Your GPUs will thank you—and your training timelines will, too.

Related posts