Posted on
• Artificial Intelligence

Artificial Intelligence Storage Performance

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Feed Your GPUs, Not Your Bottlenecks: A Practical Guide to AI Storage Performance on Linux

If your GPUs sit idle while “waiting for data,” you don’t have a compute problem—you have a storage problem. Modern AI training and inference can saturate storage subsystems with massive parallel reads, high throughput demands, and brutal small-file metadata storms. The result: expensive accelerators starved by slow disks and poorly tuned I/O paths.

This post explains how to measure, tune, and scale Linux storage for AI workloads. You’ll get concrete steps, real commands you can run, and safe defaults that help you ship faster while avoiding the most common pitfalls.

What you’ll learn:

  • Why AI storage performance is different (and how to think about it)

  • A minimal toolkit to baseline and monitor your system

  • 3–5 actionable steps: filesystem tuning, striping across NVMe, readahead and scheduler tweaks, data staging/sharding, and live monitoring

  • Commands for apt, dnf, and zypper to install everything you need

Note: Always test changes off-production first. Some steps (mkfs, RAID creation) destroy data.

AI Storage Performance: What Actually Matters

AI data pipelines stress storage differently than typical OLTP applications:

  • High throughput: Large batch reads for training/checkpoints (tens of GB/s aggregate on multi-GPU nodes).

  • Concurrency: Many workers/threads performing parallel reads, often against the same set of large files (or a mountain of tiny ones).

  • Latency consistency: GPU utilization tanks when tail latencies spike—even if averages seem fine.

  • Metadata storms: Millions of tiny files (images, tokens, parquet row groups) thrash directory and inode caches.

  • Multi-tier reality: Shared network storage + local NVMe scratch + OS page cache + optional object storage.

Your goal is to deliver sustained throughput with predictable latency under concurrency—long enough for the GPUs to stay busy.

Quick Wins Checklist (If You Only Do Five Things)

  • Measure with fio and iostat before changing anything.

  • Use XFS on local NVMe scratch with noatime and a sensible readahead (blockdev --setra).

  • Stripe across multiple NVMe devices (LVM or mdadm RAID0) for read-heavy scratch.

  • Pre-stage hot datasets to local NVMe and shard tiny files into large sequential archives (e.g., tar shards).

  • Monitor continuously during training to catch regressions early.

Install the Tools (apt, dnf, zypper)

You’ll need fio (benchmarking), sysstat (iostat), nvme-cli, mdadm (software RAID), LVM, XFS tools, and iotop. Use your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y fio sysstat nvme-cli mdadm lvm2 xfsprogs iotop
  • Fedora/RHEL/CentOS/Alma/Rocky (dnf):
sudo dnf install -y fio sysstat nvme-cli mdadm lvm2 xfsprogs iotop
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y fio sysstat nvme-cli mdadm lvm2 xfsprogs iotop

Optional (power-tuning profiles):

  • apt:
sudo apt install -y tuned
  • dnf:
sudo dnf install -y tuned
  • zypper:
sudo zypper install -y tuned

Enable tuned (optional; especially helpful on RHEL/Fedora/openSUSE):

sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance

1) Baseline: Measure Before You Tune

Find your devices and current settings:

lsblk -o NAME,TYPE,SIZE,MOUNTPOINT
nvme list
cat /sys/block/nvme0n1/queue/scheduler
cat /sys/block/nvme0n1/queue/nr_requests
lsblk -d -o NAME,RA,ROTA,SCHED,RQ-SZ,WQ-SZ

Check PCIe link speed/width (bad lanes = bad day):

lspci -vv | grep -A10 -i nvme | grep -E 'LnkCap|LnkSta'

Run live I/O stats while your workload runs (or during testing):

iostat -mx 1
iotop -oPa

Quick synthetic fio tests to emulate AI-ish patterns:

  • Large sequential reads (throughput bound):
sudo fio --name=seqread --filename=/mnt/ai_scratch/testfile \
  --rw=read --bs=1M --iodepth=64 --direct=1 --numjobs=4 --size=20G \
  --group_reporting
  • Random small reads (metadata/small-sample pain proxy):
sudo fio --name=randread --filename=/mnt/ai_scratch/testfile \
  --rw=randread --bs=4k --iodepth=128 --direct=1 --numjobs=8 --size=20G \
  --group_reporting

Interpretation tips:

  • If seqread MB/s is low, you need better striping or mount/queue tuning.

  • If randread IOPS or tail latency is bad, check scheduler, readahead, and dataset layout (sharding!).

2) Filesystem and I/O Path Tuning (Safe Defaults)

XFS is a great default for local NVMe scratch used by AI workloads.

Create and mount an XFS filesystem (WARNING: mkfs destroys data):

sudo mkfs.xfs -f /dev/nvme0n1
sudo mkdir -p /mnt/ai_scratch
echo '/dev/nvme0n1 /mnt/ai_scratch xfs noatime,nodiratime,lazytime,discard=async 0 0' | sudo tee -a /etc/fstab
sudo mount -a

Why these options?

  • noatime/nodiratime: cut metadata writes.

  • lazytime: batch timestamp updates.

  • discard=async: background TRIM for SSDs without blocking writes (alternatively, run sudo fstrim -av via cron/systemd timer).

Set readahead to help sequential reads:

sudo blockdev --report /dev/nvme0n1
sudo blockdev --setra 4096 /dev/nvme0n1   # 4096 sectors ~= 2 MiB

I/O scheduler for NVMe:

cat /sys/block/nvme0n1/queue/scheduler
# Try 'none' for max throughput or 'mq-deadline' if you need steadier tail latency.
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

Queue depth (nr_requests) can matter under high concurrency:

cat /sys/block/nvme0n1/queue/nr_requests
echo 2048 | sudo tee /sys/block/nvme0n1/queue/nr_requests

Persist udev tunables by adding a file like /etc/udev/rules.d/60-nvme-io.rules:

ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none", ATTR{queue/nr_requests}="2048"

Then:

sudo udevadm control --reload && sudo udevadm trigger

3) Scale I/O: Stripe Across Multiple NVMe Drives

For read-heavy, ephemeral scratch space, striping can dramatically boost throughput. Two options:

  • LVM striping (flexible; integrates well with Linux tooling)
# WARNING: destroys data on these devices
sudo pvcreate /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
sudo vgcreate vg_ai /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
sudo lvcreate -i4 -I256k -l 100%FREE -n lv_scratch vg_ai
sudo mkfs.xfs -f /dev/vg_ai/lv_scratch
sudo mkdir -p /mnt/ai_scratch
echo '/dev/vg_ai/lv_scratch /mnt/ai_scratch xfs noatime,nodiratime,lazytime,discard=async 0 0' | sudo tee -a /etc/fstab
sudo mount -a

-i4 = 4 stripes, -I256k = 256 KiB stripe size (good starting point for large reads).

  • mdadm RAID0 (simple and fast for scratch)
# WARNING: destroys data on these devices
sudo mdadm --create /dev/md0 --level=0 --raid-devices=4 /dev/nvme[0-3]n1
sudo mkfs.xfs -f /dev/md0
sudo mkdir -p /mnt/ai_scratch
echo '/dev/md0 /mnt/ai_scratch xfs noatime,nodiratime,lazytime,discard=async 0 0' | sudo tee -a /etc/fstab
sudo mount -a

Re-test with fio. You should see near-linear gains in sequential throughput when scaling from 1 to N devices.

4) Make Data Easy to Read Fast: Staging and Sharding

Even perfect disks crumble under millions of tiny files. You can win big by changing how data lands on disk.

  • Pre-stage hot datasets on local NVMe scratch (avoid network round-trips during epochs):
# Example: stage from shared storage to local scratch
rsync -a --info=progress2 /shared/datasets/imagenet/ /mnt/ai_scratch/imagenet/
  • Shard tiny files into large sequential archives (compatible with many frameworks, e.g., WebDataset):
# Create ~1 GiB tar shards from a directory of images
mkdir -p /mnt/ai_scratch/shards
cd /mnt/ai_scratch/imagenet

# List files into chunks of ~25000 (tune to hit ~1GiB per shard depending on file sizes)
find . -type f | split -l 25000 - filelist_
for f in filelist_*; do
  shard="shards/$(basename $f).tar"
  tar -cf "$shard" -T "$f"
done

Large shards turn metadata thrash into smooth sequential reads—exactly what SSDs love.

  • Warm the page cache before training (optional, when RAM allows):
# Read files once to populate cache (adjust path)
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'   # clear (careful on shared systems)
time find /mnt/ai_scratch/shards -type f -name "*.tar" -print0 | xargs -0 -n1 -P8 cat > /dev/null
  • Keep TRIM healthy for SSDs:
sudo fstrim -av

Schedule it weekly via systemd timers or cron if you don’t use discard=async.

5) Monitor During Training: Catch Regressions Early

Run lightweight monitoring alongside your training jobs:

  • Disk and CPU I/O wait:
iostat -mx 1

Look for high await, svctm, or sustained util near 100%.

  • Top talkers by I/O:
sudo iotop -oPa
  • NVMe health:
nvme list
sudo nvme smart-log /dev/nvme0
  • Quick “am I feeding GPUs?” proxy:
    • GPU utilization should be stable and high during steps/epochs.
    • If GPU util dips coincide with spikes in iostat await or iotop contention, storage is the culprit.

Real-World Patterns That Work

  • Use network storage (parallel FS or object store) for cold data; pre-stage to local NVMe RAID0 scratch for hot, epoch-level training.

  • Prefer fewer, larger files (shards) to millions of tiny files.

  • Tune readahead and scheduler per device; validate with fio under concurrency that matches your data loader settings.

  • Scale throughput by striping NVMe; don’t forget PCIe topology (avoid oversubscribed lanes).

Conclusion and Next Steps

Storage is the cheapest way to unlock trapped GPU performance. Start by measuring, apply safe filesystem and I/O tweaks, scale out with striping, and fix your dataset layout. Then monitor continuously to keep your accelerators fed.

Your next steps: 1) Install the tools with your package manager and run the baseline fio tests. 2) Create an XFS scratch volume with noatime and tuned readahead. 3) Stripe across multiple NVMe devices and re-test. 4) Shard and pre-stage your dataset; run a small training job and watch GPU utilization.

Have a tricky setup or need help reading fio outputs? Drop your device list (lsblk, nvme list) and fio results, and I’ll help you tune the next bottleneck.

Related posts