Artificial Intelligence Storage Best Practices on Linux (With Bash You Can Copy/Paste)

If your GPUs are fast but your training still crawls, your bottleneck is probably storage. In modern AI pipelines, GPUs routinely wait on I/O more than they wait on math. That’s lost time, lost money, and—worst—silent correctness issues from flaky disks, small-file overload, or poorly tuned filesystems.

This guide explains why storage matters for AI and gives you 5 actionable best practices with copy‑pasteable Bash. You’ll learn how to stage data to local NVMe, pick and tune filesystems, benchmark correctly, fix the small‑file problem, and protect data integrity—all with installation commands for apt, dnf, and zypper where needed.

---

Why storage tuning is a big deal for AI

• AI data access is bursty and parallel: many workers hammer the same shard or directory at once.

• Small files kill throughput and metadata servers; large sequential reads fly.

• Shared/network filesystems add latency and head‑of‑line blocking if misconfigured.

• NVMe drives throttle under heat and degrade if not trimmed or monitored.

• Without checksums/snapshots, silent corruption and “mystery” regressions creep in.

Do storage right, and you’ll:

• Keep GPUs fed (higher utilization, lower time-to-accuracy).

• Cut cloud bills by shrinking idle GPU time.

• Avoid “works on machine A but not B” anomalies due to inconsistent data.

---

Install the tooling once

You’ll use these tools throughout: fio, sysstat (iostat, pidstat), smartmontools, nvme-cli, ioping, mdadm, lvm2, XFS/EXT utilities, and NFS client tools.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y fio sysstat smartmontools nvme-cli ioping xfsprogs e2fsprogs mdadm lvm2 nfs-common

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y fio sysstat smartmontools nvme-cli ioping xfsprogs e2fsprogs mdadm lvm2 nfs-utils

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y fio sysstat smartmontools nvme-cli ioping xfsprogs e2fsprogs mdadm lvm2 nfs-client

---

1) Tier your storage and stage hot data locally

Use the right medium for the right job:

• Hot path (training shards, feature stores): local NVMe (scratch).

• Warm path (shared datasets across nodes): high‑throughput NFS/parallel FS.

• Cold/archive (checkpoints, rarely used datasets): object storage or HDD-backed arrays.

Stage hot datasets from shared storage to local NVMe before training:

# Example: stage from NFS to local NVMe scratch
sudo mkdir -p /mnt/ai_nfs /scratch
sudo mount -t nfs -o vers=4.2,nconnect=4,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noatime 10.0.0.10:/export/datasets /mnt/ai_nfs
rsync -aH --info=progress2 /mnt/ai_nfs/imagenet_shards/ /scratch/imagenet_shards/

Notes:

• nconnect (Linux 5.3+) opens multiple TCP connections to an NFS server to boost parallelism.

• Keep scratch ephemeral. If you need more throughput and can tolerate risk, stripe NVMe with RAID0:

# DANGER: RAID0 has no redundancy (scratch only!)
sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/nvme0n1 /dev/nvme1n1
sudo mkfs.xfs -f /dev/md0
sudo mkdir -p /scratch
sudo mount -o noatime /dev/md0 /scratch

Packages used above:

• NFS client: apt: nfs-common, dnf: nfs-utils, zypper: nfs-client

• mdadm (software RAID): apt/dnf/zypper: mdadm

• XFS tools: apt/dnf/zypper: xfsprogs

---

2) Format and mount for throughput (XFS/ext4, scheduler, read‑ahead)

For high‑parallelism AI workloads, XFS is a safe default. Ext4 is fine too. Choose one, format cleanly, and mount with low‑overhead options.

Create and mount XFS:

sudo mkfs.xfs -f /dev/nvme0n1
sudo mkdir -p /data
sudo mount -o noatime /dev/nvme0n1 /data

Create and mount ext4:

sudo mkfs.ext4 -F -E lazy_itable_init=1,lazy_journal_init=1 /dev/nvme0n1
sudo mkdir -p /data
sudo mount -o noatime /dev/nvme0n1 /data

Trim SSDs/NVMe automatically:

# Recommended to avoid mount-time discard; use periodic fstrim
sudo systemctl enable fstrim.timer
sudo systemctl start fstrim.timer

Set a low-overhead I/O scheduler for NVMe and tune read‑ahead:

# Check available schedulers (look for [none] or [mq-deadline])
cat /sys/block/nvme0n1/queue/scheduler

# Try 'none' for NVMe; alternatively try 'mq-deadline' and benchmark
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# Increase read-ahead to 64 MiB for large sequential reads
sudo blockdev --setra 65536 /dev/nvme0n1
blockdev --getra /dev/nvme0n1

Packages used above:

• XFS: xfsprogs

• ext4 tools: e2fsprogs

---

3) Benchmark the right way (throughput, latency, and saturation)

Use realistic tools and workloads. Benchmark both preparation (write) and consumption (read).

Prepare a test file and measure sequential read throughput:

# Create a 100G file (sequential write)
fio --name=prep --filename=/data/fio.test --rw=write --bs=1M --iodepth=64 --numjobs=1 --size=100G --direct=1

# Read test with parallel jobs (sequential read)
fio --name=seqread --filename=/data/fio.test --rw=read --bs=1M --iodepth=64 --numjobs=8 \
    --time_based --runtime=60 --direct=1 --group_reporting

Grab a quick latency snapshot:

ioping -c 10 /data

Watch device saturation and queue depths:

# Extended stats per device, every second
iostat -xz 1

# Per-process disk activity
pidstat -d 1

Monitor device health and temperature (throttling kills throughput):

# NVMe
sudo nvme smart-log /dev/nvme0

# SATA/SAS
sudo smartctl -a /dev/sda

Packages used above:

• fio: fio

• ioping: ioping

• iostat/pidstat: sysstat

• nvme-cli: nvme-cli

• smartctl: smartmontools

---

4) Kill the small‑file tax (shard for speed)

Millions of tiny files wreck metadata performance. Shard datasets into large archives (e.g., ~1–4 GiB) so workers read sequentially. Many frameworks support WebDataset/TFRecord/LMDB; here’s a portable tar‑based approach in Bash.

Create 10k‑file shards:

SRC=/datasets/images
DST=/datasets/shards
mkdir -p "$DST"
cd "$SRC"

# Make lists of 10,000 files each
find . -type f | sort | split -d -l 10000 - filelist_

# Create TAR shards
n=0
for list in filelist_*; do
  printf -v shard "%s/shard-%05d.tar" "$DST" "$n"
  tar -cf "$shard" -T "$list"
  n=$((n+1))
done

Raise file descriptor limits for heavy dataloaders:

# Temporarily
ulimit -n 1048576

# For permanence, add to /etc/security/limits.conf and reboot or re-login:
# * soft nofile 1048576
# * hard nofile 1048576

Optionally pre‑warm page cache before a run:

# Touch shards to seed cache (quick-and-dirty)
cat /datasets/shards/shard-*.tar > /dev/null

---

5) Protect integrity and stay ahead of failures (checksums, snapshots, RAID)

Checksums catch silent corruption:

cd /data
find . -type f -print0 | xargs -0 sha256sum > SHA256SUMS
# Later, verify:
sha256sum -c SHA256SUMS

Use LVM to take cheap snapshots before risky preprocessing steps:

# One-time setup (adjust sizes)
sudo pvcreate /dev/nvme0n1
sudo vgcreate vg_ai /dev/nvme0n1
sudo lvcreate -n data -L 2T vg_ai
sudo mkfs.xfs -f /dev/vg_ai/data
sudo mkdir -p /data
sudo mount /dev/vg_ai/data /data

# Snapshot before transforming data
sudo lvcreate -s -n data_snap -L 100G /dev/vg_ai/data
sudo mkdir -p /snap
sudo mount /dev/vg_ai/data_snap /snap

Mirror critical datasets (not just checkpoints) with RAID1 or RAID10:

# RAID1 example (two devices)
sudo mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda /dev/sdb
sudo mkfs.xfs -f /dev/md1
sudo mkdir -p /datasets
sudo mount /dev/md1 /datasets

Schedule health checks; watch temps:

# Quick on-demand checks
sudo smartctl -H /dev/sda
sudo nvme smart-log /dev/nvme0 | egrep 'temperature|percentage|media_errors'

# Nightly SMART log (root crontab)
# 0 3 * * * /usr/sbin/smartctl -a /dev/nvme0n1 >> /var/log/smartctl-nvme0n1.log

Packages used above:

• lvm2: lvm2

• mdadm: mdadm

• smartmontools: smartmontools

• nvme-cli: nvme-cli

---

Real‑world patterns to emulate

• Training on NFS but caching locally: stage shards to NVMe before each epoch; rsync delta only.

• Convert image folders to tar shards or TFRecords; measure epoch time before/after—expect 2–10× faster I/O.

• Use XFS on scratch NVMe with noatime and readahead 64 MiB; toggle NVMe scheduler (none vs mq-deadline) and pick the winner.

• Watch iostat -xz during a training step; if util is ~100% and svctm climbs, you’re I/O bound.

• Heat matters: if nvme smart-log shows high temps, add airflow or move drives; throttling can halve throughput.

---

Conclusion and next steps (CTA)

Storage is the cheapest speedup most AI stacks leave on the table. This week: 1) Install the tools and baseline your storage with fio, iostat, and ioping. 2) Stage your hottest dataset to local NVMe and shard it. 3) Tune filesystem mounts, scheduler, and readahead. Re‑benchmark. 4) Add checksums and snapshot before risky transforms.

If you want a one‑file starter, copy your favorite commands from this post into a setup.sh and run it on every training node. Then iterate: measure → change one thing → measure again. Share your before/after numbers with your team—or send them my way and I’ll help you squeeze more out of your storage stack.