Posted on
• Artificial Intelligence

Artificial Intelligence Infrastructure Reviews

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Infrastructure Reviews on Linux: A Bash-First, Hands-On Guide

If your models train slower than the whitepaper promised—or your inference cluster feels mysteriously “bursty”—the problem might not be your code. It’s your infrastructure. The fastest way to find out is to measure it directly, reproducibly, and in plain Bash.

In this article, you’ll get a practical, Linux-first workflow to review GPU nodes and AI clusters: quick checks, targeted benchmarks, and observability you can actually run today. You’ll also get install instructions for apt, dnf, and zypper wherever tools are used.

What you’ll learn:

  • How to baseline CPUs/GPUs, storage, and network in minutes

  • How to catch misconfigurations (PCIe bottlenecks, slow disks, noisy neighbors)

  • How to test “container parity” so workloads behave the same in and out of containers

Use this when validating new hardware, before/after driver changes, or during performance regressions.

Why AI infrastructure reviews matter

  • GPU time is expensive. A 10–20% bottleneck in I/O or PCIe can cost real money at scale.

  • AI stacks are layered. Drivers, runtimes, containers, compilers, and kernels all interact. Small mismatches lead to large slowdowns.

  • Benchmarks can lie. Vendor scores rarely match your data, your kernel version, or your network fabric.

  • Repeatability wins. A light, Bash-driven checklist makes results comparable across nodes, clusters, and clouds.

Quick setup: Install the essentials

We’ll use widely available tools: fio (storage), iperf3 (network), nvme-cli (NVMe info), nvtop/htop (monitoring), sysstat (iostat/sar), Podman (containers), and Python for a minimal GPU smoke test.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y fio iperf3 nvme-cli nvtop htop sysstat podman python3 python3-pip python3-venv git

DNF (Fedora/RHEL/CentOS Stream):

sudo dnf install -y fio iperf3 nvme-cli nvtop htop sysstat podman python3 python3-pip git

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y fio iperf3 nvme-cli nvtop htop sysstat podman python3 python3-pip git

Notes:

  • nvidia-smi is provided by the NVIDIA driver. If you’re on AMD, you can use rocm-smi (from AMD’s ROCm repos).

  • Enable sysstat collection if needed (e.g., on Debian/Ubuntu: sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat && sudo systemctl enable --now sysstat).

The core review: 5 actionable checks you can run now

1) Inventory and sanity-check the node

Goal: Confirm the hardware you paid for is the hardware you’re using, and catch obvious misconfigurations (e.g., GPUs on x4 PCIe lanes, missing hugepages, underclocked memory).

Run:

echo "=== CPU & Memory ==="
lscpu
free -h

echo "=== NUMA topology ==="
numactl --hardware 2>/dev/null || echo "Install numactl for NUMA detail"

echo "=== Block devices ==="
lsblk -o NAME,MODEL,SIZE,ROTA,MOUNTPOINT
sudo nvme list 2>/dev/null || echo "nvme-cli not available or no NVMe devices"

echo "=== PCIe GPUs / accelerators ==="
lspci | egrep -i 'vga|3d|nvidia|amd'

echo "=== NVIDIA (if present) ==="
nvidia-smi || echo "No NVIDIA driver or GPU detected"

echo "=== Kernel & IOMMU ==="
uname -a
dmesg | egrep -i 'iommu|pci|nvme' | tail -n 50

What to look for:

  • lscpu: expected core counts, SMT, and flags (avx2/avx512).

  • lsblk/nvme: expected NVMe model, non-rotational (ROTA=0), correct mountpoints.

  • lspci: each GPU present; avoid GPUs sharing narrow PCIe links behind oversubscribed switches.

  • nvidia-smi: proper driver load, no persistent ECC or Xid errors in dmesg.

Optional: Quick capture to a report file

outfile="ai_infra_baseline_$(hostname)_$(date +%F).txt"
{ lscpu; free -h; lsblk -o NAME,MODEL,SIZE,ROTA,MOUNTPOINT; sudo nvme list; lspci; nvidia-smi; uname -a; } > "$outfile" 2>&1 || true
echo "Wrote $outfile"

2) Validate storage throughput and IOPS with fio

Why: Data pipelines often throttle GPUs. Confirm your local NVMe or shared volume meets expectations.

Install (if not already done):

  • apt: sudo apt install -y fio

  • dnf: sudo dnf install -y fio

  • zypper: sudo zypper install -y fio

Test sequential read/write on a target mount (replace /mnt/fast):

TARGET=/mnt/fast
cd "$TARGET" || { echo "Mount $TARGET first"; exit 1; }

# 4 GiB sequential write then read, 1 MiB block, 4 jobs
fio --name=seqwrite --filename=fiotest.bin --size=4G --bs=1M --rw=write --direct=1 --numjobs=4 --iodepth=32
fio --name=seqread  --filename=fiotest.bin --size=4G --bs=1M --rw=read  --direct=1 --numjobs=4 --iodepth=32

# Random read (4k), IOPS focus
fio --name=randread --filename=fiotest.bin --size=4G --bs=4k --rw=randread --direct=1 --numjobs=8 --iodepth=64

rm -f fiotest.bin

Interpretation:

  • For modern PCIe 4.0 NVMe, expect multi-GB/s sequential read/write.

  • Random 4k reads measure IOPS; large deviations across identical nodes hint at throttling, queue depth, mount options, or filesystem issues.

Tip: For shared storage (NFS/SMB/object via FUSE), run the same tests to quantify per-mount performance.

3) Check east–west network with iperf3

Why: Distributed training and parameter sharding need bandwidth and stability; NCCL or RCCL performance depends on the underlying fabric.

Install (if not already done):

  • apt: sudo apt install -y iperf3

  • dnf: sudo dnf install -y iperf3

  • zypper: sudo zypper install -y iperf3

On Server (Node A):

iperf3 -s

On Client (Node B, replace SERVER_IP):

iperf3 -c SERVER_IP -P 8 -t 30

What to look for:

  • 100 GbE should approach ~12 GB/s TCP throughput under ideal conditions; RoCE/InfiniBand may differ.

  • Use -R for reverse tests, and -B <iface_ip> to pin interfaces on multi-NIC hosts.

  • Large variance suggests MTU mismatch (check 9000 vs 1500), queueing, or CPU scaling.

Optional latency snapshot (not a full RTT test, but a hint):

ping -c 50 SERVER_IP

4) GPU smoke test + live observability

Why: Drivers may load, but kernels might not run at full clocks or mixed precision may be disabled. Run a tiny workload and watch utilization.

Monitoring tools:

  • apt: sudo apt install -y nvtop htop sysstat

  • dnf: sudo dnf install -y nvtop htop sysstat

  • zypper: sudo zypper install -y nvtop htop sysstat

In one terminal, run:

nvtop

or

watch -n 1 nvidia-smi

Minimal PyTorch check (use a venv on Debian/Ubuntu; elsewhere system Python often suffices):

Ubuntu/Debian (venv recommended):

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
# Choose the right wheel for your GPU/driver per https://pytorch.org/get-started/locally/
# Example (adjust the CUDA version as appropriate for your system):
pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision

Fedora/RHEL/openSUSE (use system Python or create a venv if desired):

python3 -m pip install --user --upgrade pip
# As above, select the proper CUDA/ROCm wheel from PyTorch docs
pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision

GPU availability + quick matmul timing:

python - <<'PY'
import time, torch
print("CUDA available:", torch.cuda.is_available())
if not torch.cuda.is_available():
    raise SystemExit("Install a GPU-enabled torch wheel matching your driver.")

device = torch.device("cuda:0")
for dtype in (torch.float32, torch.float16, torch.bfloat16):
    if not torch.cuda.is_available(): break
    try:
        a = torch.randn((8192, 8192), device=device, dtype=dtype)
        b = torch.randn((8192, 8192), device=device, dtype=dtype)
        torch.cuda.synchronize()
        t0 = time.time()
        for _ in range(10):
            c = a @ b
        torch.cuda.synchronize()
        dt = time.time() - t0
        print(f"dtype={dtype} avg per matmul: {dt/10:.4f}s")
    except Exception as e:
        print(f"dtype={dtype} failed: {e}")
PY

What to look for:

  • nvidia-smi shows near-100% utilization during the loop.

  • FP16/bfloat16 should be significantly faster than FP32 on tensor-core GPUs.

  • Large variance across identical nodes suggests power limits, thermal throttling, or different driver/runtime stacks.

Tip: For AMD GPUs, use ROCm wheels from PyTorch and monitor with rocm-smi.

5) Container parity with Podman

Why: Many teams ship training/inference in containers. Your containerized performance should match bare metal, including datasets and GPU access.

Install Podman:

  • apt: sudo apt install -y podman

  • dnf: sudo dnf install -y podman

  • zypper: sudo zypper install -y podman

Test container and host volume mapping:

mkdir -p ~/ai-datasets
podman run --rm -v ~/ai-datasets:/data:Z docker.io/library/alpine:latest sh -c "echo ok > /data/smoke && ls -l /data"
ls -l ~/ai-datasets/smoke

If you use NVIDIA/AMD GPUs in containers:

  • Ensure the vendor’s container integration is installed and configured on the host (e.g., GPU device hooks/runtimes).

  • Run a known-good image (e.g., PyTorch or TensorRT) with GPU access and confirm frameworks detect the GPU.

  • The exact steps depend on vendor/runtime version—follow the current vendor docs for your distribution.

Quick PyTorch in container (CPU example, replace with GPU-enabled base if configured):

podman run --rm -it python:3.11 bash -lc "python - <<'PY'
import sys
print('Hello from container Python:', sys.version)
PY"

Container parity checks:

  • Same dataset path and performance (throughput) inside and outside the container.

  • Same driver/runtime versions visible to frameworks.

  • No cgroup/container memory or CPU limits unintentionally throttling workloads.

Real-world review flow (putting it together)

Use this minimal script to collect a snapshot before/after changes:

#!/usr/bin/env bash
set -euo pipefail
tag="${1:-baseline}"
outdir="ai-review-$(hostname)-$(date +%F-%H%M)-$tag"
mkdir -p "$outdir"

# Hardware and OS
{ date; uname -a; lscpu; free -h; lsblk -o NAME,MODEL,SIZE,ROTA,MOUNTPOINT; sudo nvme list 2>/dev/null || true; lspci; } > "$outdir/system.txt" 2>&1

# NVIDIA (if present)
nvidia-smi -q -x > "$outdir/nvidia-smi.xml" 2>/dev/null || true

# Storage quick test (safe, read-only)
mount | tee "$outdir/mounts.txt"

# Network neighbor test requires manual server/client coordination
echo "Run: iperf3 -s on a peer, then iperf3 -c PEER -P 8 -t 20 | tee $outdir/iperf3.txt"

echo "Snapshot written to $outdir"

Keep these snapshots in version control or object storage to compare regressions over time.

Conclusion and next steps

A solid AI infrastructure review doesn’t require a week of tuning or proprietary tools—just disciplined, Bash-first checks you can repeat. With the steps above, you can:

  • Catch miswired PCIe lanes and underperforming disks

  • Verify network headroom for distributed training

  • Prove GPU kernels actually run fast under your stack

  • Ensure containers behave like bare metal

Your next step:

  • Run the five checks on one node today and save the results.

  • Standardize this as a preflight for every new node or driver/runtime change.

  • Share your findings or bottlenecks with the team; fix the biggest gap first (it’s often storage or network).

If you want a one-command toolkit that wraps these tests and exports a report, reply and I’ll help you tailor a portable Bash harness for your environment and GPUs.

Related posts