Posted on
• Artificial Intelligence

Artificial Intelligence Storage Troubleshooting

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Storage Troubleshooting: A Bash-First Playbook for Linux

If your GPUs are yawning while your training loop “waits for data,” you likely have a storage problem, not a compute problem. AI workloads can hammer disks with small, random reads, high metadata churn, and parallel access patterns that ordinary defaults don’t handle well. The result: iowait spikes, low GPU utilization, and blown training schedules.

This guide gives you a practical, Bash-centered checklist to find and fix AI storage bottlenecks on Linux. You’ll get a baseline-measurement script, actionable tuning steps, and real-world examples—all with commands you can paste into your terminal.

Why AI storage is different (and why it matters)

  • Many small files: Image datasets often contain millions of tiny files, thrashing metadata and caches.

  • Highly parallel readers: DataLoader workers and multiple GPUs multiply I/O pressure.

  • Mixed I/O patterns: Random reads, sequential prefetch, checkpoint writes—simultaneously.

  • Distributed layers: Local NVMe, network storage (NFS/S3), containers, and scratch areas add complexity.

Get storage right, and you turn “GPU idle” into “GPU busy.” Get it wrong, and you pay for compute you can’t use.

Install the troubleshooting toolbox

We’ll use iostat/pidstat (sysstat), iotop, nvme-cli, smartmontools, fio, and hdparm (for SATA).

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y sysstat iotop nvme-cli smartmontools fio hdparm
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y sysstat iotop nvme-cli smartmontools fio hdparm
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y sysstat iotop nvme-cli smartmontools fio hdparm

Optional (for NFS clients):

  • Debian/Ubuntu:
sudo apt install -y nfs-common
  • Fedora/RHEL/CentOS:
sudo dnf install -y nfs-utils
  • openSUSE:
sudo zypper install -y nfs-client

1) Measure before you tweak: capture a baseline

First, verify it’s I/O and not CPU/network. Run these in a dedicated shell during a training step.

  • Device throughput, latency, and iowait:
iostat -xz 1
  • Top processes by I/O (needs root for full detail):
sudo iotop -oPa
  • Per-process I/O stats (part of sysstat):
pidstat -d 1
  • Quick one-minute snapshot script:
cat > ai-io-snapshot.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
OUT=${1:-ai-io-snapshot-$(date +%F-%H%M%S)}
mkdir -p "$OUT"
echo "Collecting baseline into $OUT"

uname -a > "$OUT/uname.txt"
lsblk -o NAME,MODEL,ROTA,SCHED,RA,MOUNTPOINT > "$OUT/lsblk.txt"
df -hT > "$OUT/df.txt"
df -i > "$OUT/df-inodes.txt"
free -h > "$OUT/free.txt"
cat /proc/cpuinfo > "$OUT/cpuinfo.txt"
dmesg -T | tail -n 500 > "$OUT/dmesg-tail.txt"

timeout 60 iostat -xz 1 > "$OUT/iostat.txt" || true
timeout 60 pidstat -d 1 > "$OUT/pidstat-d.txt" || true
sudo timeout 60 iotop -b -o -d 1 > "$OUT/iotop.txt" || true

for d in /sys/block/nvme* /sys/block/sd*; do
  [ -e "$d" ] || continue
  name=$(basename "$d")
  {
    echo "=== $name ==="
    [ -f "$d/queue/scheduler" ] && cat "$d/queue/scheduler"
    [ -f "$d/queue/read_ahead_kb" ] && echo "read_ahead_kb: $(cat "$d/queue/read_ahead_kb")"
    [ -f "$d/queue/nr_requests" ] && echo "nr_requests: $(cat "$d/queue/nr_requests")"
  } >> "$OUT/queues.txt"
done
echo "Done. Review $OUT/*.txt"
EOF
chmod +x ai-io-snapshot.sh
./ai-io-snapshot.sh

What to look for:

  • iowait high (>10–20%) while GPUs underutilized.

  • iostat shows high await/svctm and queue depth on a device.

  • iotop/pidstat point to the actual training/data-loader PIDs.

Real-world example: On a node with 8 GPUs, iowait ~35% and GPU utilization ~40% during ImageNet training. Root cause was millions of 256 KB files with default readahead and metadata churn.

2) Verify media and path health

Before tuning, confirm the hardware is okay.

  • SMART (SATA/SAS):
sudo smartctl -a /dev/sdX
sudo smartctl -t short /dev/sdX
  • NVMe health:
sudo nvme list
sudo nvme smart-log /dev/nvme0
  • Kernel messages (timeouts, resets):
dmesg -T | egrep -i 'nvme|blk|i/o error|reset|timeout' | tail -n +1

If you see media errors, rising CRCs, or controller resets, fix hardware/firmware/cabling first. No amount of tuning beats bad bits.

3) Filesystem and kernel knobs that actually move the needle

These are generally safe, reversible changes. Test on a staging box before applying widely.

  • Prefer XFS or tuned ext4 for parallelism:

    • XFS handles parallel metadata workloads well.
    • ext4 is fine; avoid small inode/dir hash defaults for huge trees.

  • Reduce metadata writes (noatime/lazytime):

    • Add or adjust fstab (example for XFS on /data):
UUID=xxxx-xxxx  /data  xfs  noatime,lazytime,discard  0  0

Then:

sudo mount -o remount /data

noatime stops access-time updates; lazytime batches atime/mtime in memory.

  • Increase read-ahead for sequential-ish reads:
# Check current
cat /sys/block/nvme0n1/queue/read_ahead_kb

# Set to 8 MiB (example). Try 4096–16384 (4–16 MiB). Tune per device.
echo 8192 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

For SATA disks:

sudo blockdev --getra /dev/sdX     # sectors (512B each)
sudo blockdev --setra 16384 /dev/sdX  # ~8 MiB
  • Use an I/O scheduler that matches the media:
    • NVMe: “none” or “mq-deadline” usually best.
    • SATA SSD/HDD: “mq-deadline” or “bfq” (for fairness).
cat /sys/block/nvme0n1/queue/scheduler
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
  • Keep writeback sane to avoid long stalls (conservative example):
# Temporary (until reboot)
sudo sysctl -w vm.dirty_background_ratio=5
sudo sysctl -w vm.dirty_ratio=20

Note: Extreme values can risk data loss on power failure and cause jitter. Keep changes modest and test.

  • Disable atime globally only if you’re sure you don’t need it (some software relies on it). Prefer mount options over global kernel switches.

Real-world example: Bumping read_ahead_kb from 128 KiB to 8 MiB and remounting with noatime,lazytime raised average GPU utilization from ~45% to ~80% on an image-classification workload without changing code.

4) Shape your dataset and I/O pattern

The biggest wins often come from changing how data hits the disk.

  • Aggregate tiny files into larger shards:
# Create ~1 GiB shards from a directory of small files
cd /dataset
find . -type f -print0 | xargs -0 stat -c '%s %n' | sort -n | awk '{print $2}' > filelist.txt

split -l 50000 filelist.txt files_part_
for p in files_part_*; do
  tar -cf "shard_${p##*_}.tar" -T "$p"
done

Use 64–256 shards for multi-GPU training. Frameworks like WebDataset/TFRecord/LMDB do this natively.

  • Cache hot datasets on local NVMe:
rsync -a --info=progress2 /net/share/dataset/ /local_nvme/dataset/

Point your training job to the local path. Validate with iostat that the network/NFS device goes idle during steady-state.

  • Pre-warm a small working set (when feasible and ephemeral):
# Touch metadata and read into page cache
find /local_nvme/dataset -type f -name '*.tar' -print0 | \
  xargs -0 -n1 -P4 sh -c 'head -c 4194304 "$0" >/dev/null'  # 4 MiB per file
  • Reproduce load with fio to test improvements without running the whole model:
cat > ai-read-rand.fio <<'EOF'
[global]
ioengine=libaio
direct=1
time_based=1
runtime=60
group_reporting=1

[randread]
filename=/local_nvme/dataset/shard_00.tar
rw=randread
bs=256k
iodepth=64
numjobs=4
EOF

fio ai-read-rand.fio

Compare IOPS/latency before and after tuning. Also try bs=4k and bs=1m to bracket extremes.

Real-world example: Moving 120M JPEGs into 128 tar shards + local NVMe caching cut metadata ops by ~95%, dropped iowait from 30% to <5%, and pushed GPU utilization from ~50% to ~92%.

5) Capacity and housekeeping traps that stall training

  • Inode exhaustion (millions of tiny files):
df -i
# Count files to spot pathologically deep trees
find /data -xdev -type f | wc -l

If inodes are low, pack files into shards or reformat with more inodes (ext4 -T largefile4 is not ideal for many small files).

  • /tmp or scratch filling up:
df -h /tmp
# Redirect temp to a bigger, faster mount for your job:
export TMPDIR=/data/tmp
mkdir -p "$TMPDIR"

If using systemd services, set TemporaryFileSystem or RuntimeDirectory appropriately.

  • Open-file limits too low (EMFILE errors, loader stalls):
ulimit -n
# Increase for the session:
ulimit -n 1048576

For systemd-managed services:

sudo systemctl edit my-training.service
# Then add:
# [Service]
# LimitNOFILE=1048576
sudo systemctl daemon-reload
sudo systemctl restart my-training.service
  • Journald or logs exploding on small roots:
journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Optional: If you’re on NFS

For read-heavy workloads, modern NFS with multiple TCP connections and large I/O sizes helps.

Mount example (Linux 5.3+ for nconnect):

sudo mkdir -p /mnt/dataset
sudo mount -t nfs -o vers=3,nconnect=8,rsize=1048576,wsize=1048576,timeo=600,retrans=2 server:/export/dataset /mnt/dataset

Measure with iostat and adjust nconnect/rsize/wsize to match your network and server.

Install NFS client utilities if missing:

  • apt:
sudo apt install -y nfs-common
  • dnf:
sudo dnf install -y nfs-utils
  • zypper:
sudo zypper install -y nfs-client

Quick checklist for a first pass

  • Run the baseline script and confirm I/O-bound behavior.

  • Check SMART/NVMe health; fix hardware issues first.

  • Mount datasets with noatime,lazytime; raise read_ahead_kb; pick proper scheduler.

  • Aggregate tiny files into shards; cache locally if possible.

  • Raise ulimit -n, watch /tmp, and check inodes.

Do these, and you’ll usually see GPU utilization jump without touching your model code.

Conclusion and Call to Action

Storage bottlenecks are silent budget killers in AI pipelines. With a few Bash commands and targeted changes, you can turn a disk-bound training loop into a compute-bound one.

Your next steps: 1) Install the toolbox and run ai-io-snapshot.sh during a representative training step. 2) Apply the safe tunings (noatime/lazytime, read_ahead_kb, scheduler) and re-test. 3) Restructure the dataset into shards and, if possible, cache locally. 4) Share the before/after iostat and fio results with your team to standardize fixes across nodes.

If you’d like a ready-made gist of the scripts and a sample fio profile matrix, let me know your environment (NVMe/SATA, XFS/ext4, single/multi-GPU), and I’ll tailor a starter kit.

Related posts