Posted on
• Artificial Intelligence

Artificial Intelligence Object Storage

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI at Petabyte Scale: Object Storage for Your Linux/Bash Workflows

If your models are hungry and your datasets are ballooning, your trusty POSIX filesystem and NFS shares will eventually choke. AI workloads want cheap, durable, and massively parallel storage. That’s exactly what object storage delivers—and you can drive it entirely from Bash.

This post explains why object storage is a perfect fit for AI pipelines, how it differs from traditional storage, and gives you actionable Bash-centric steps to move data in/out efficiently, mount buckets, harden security, and run at scale.

TL;DR (The Value)

  • Object storage (S3-compatible) is built for scale, cost efficiency, and parallel I/O—ideal for datasets, model checkpoints, and artifacts.

  • You’ll use familiar CLI tools (awscli, s3cmd, rclone, s3fs) to script everything in Bash.

  • With a few config tweaks, you can saturate your network links, version your data, and keep costs under control.

Why Object Storage for AI?

  • Scalability: No directory inodes to sweat about. Buckets scale to billions of objects.

  • Cost & durability: High durability and tiered pricing. Cold tiers for archives; hot tiers for active training.

  • Parallelism: Upload thousands of shards or checkpoints concurrently; multipart uploads fly.

  • Ubiquity: S3 API has become the de facto standard—works with AWS S3, MinIO, Ceph RGW, Cloudflare R2, Wasabi, GCS/S3 gateways, and more.

Caveats (know your tools):

  • It isn’t POSIX. No atomic renames or hardlinks; eventual consistency can bite metadata-heavy workflows.

  • It’s HTTP-based. You’ll rely on CLIs/libraries and design for throughput via concurrency and multipart uploads.

1) Install the Essential CLI Tools

We’ll rely on these:

  • awscli: The swiss-army knife for S3-compatible backends (works with AWS and custom endpoints).

  • s3cmd: Mature S3 client with simple config.

  • rclone: High-performance sync/copy with lots of storage backends and tuning knobs.

  • s3fs-fuse: Mount an S3 bucket as a filesystem (handy, with caveats).

Install them with your distro’s package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y awscli s3cmd rclone s3fs
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y awscli s3cmd rclone s3fs-fuse
  • openSUSE/SLES (zypper):
sudo zypper refresh
sudo zypper install -y aws-cli s3cmd rclone s3fs

Optional but fast: s5cmd (ultra-fast S3 CLI). Install from release binaries:

curl -L https://github.com/peak/s5cmd/releases/latest/download/s5cmd_$(uname -s | tr '[:upper:]' '[:lower:]')_amd64.tar.gz \
  | tar -xz s5cmd
sudo install -m 0755 s5cmd /usr/local/bin/

2) Configure Credentials and Endpoints

Use environment variables for quick experiments:

export AWS_ACCESS_KEY_ID="YourKeyID"
export AWS_SECRET_ACCESS_KEY="YourSecret"
export AWS_DEFAULT_REGION="us-east-1"

# For S3-compatible servers (MinIO, Ceph RGW, etc.)
export AWS_ENDPOINT_URL_S3="https://minio.example.com"   # awscli v2
# or use: export AWS_ENDPOINT_URL="https://minio.example.com"

awscli config (works with AWS and S3-compatible endpoints):

mkdir -p ~/.aws
cat > ~/.aws/config <<'EOF'
[default]
region = us-east-1
s3 =
    addressing_style = path
    max_concurrent_requests = 64
    multipart_threshold = 64MB
    multipart_chunksize = 64MB
EOF

cat > ~/.aws/credentials <<'EOF'
[default]
aws_access_key_id = YourKeyID
aws_secret_access_key = YourSecret
EOF
  • addressing_style=path avoids TLS wildcard issues on some custom endpoints.

s3cmd quick setup:

s3cmd --configure
# When prompted:
# Access Key: YourKeyID
# Secret Key: YourSecret
# Default Region: us-east-1
# S3 Endpoint: minio.example.com (if using MinIO/Ceph), set HTTPS accordingly
# Use “Signature v2” only if your backend requires it (most use v4)

rclone remote (named "mlstore"):

rclone config create mlstore s3 \
  provider=Other \
  env_auth=false \
  access_key_id=YourKeyID \
  secret_access_key=YourSecret \
  endpoint=https://minio.example.com \
  acl=private

Tip: If you’re not on EC2, speed up awscli init by disabling metadata:

export AWS_EC2_METADATA_DISABLED=true

3) Move Data Fast: Parallel and Multipart Transfers

High throughput = many concurrent transfers + large enough multipart chunks.

  • awscli (sync multiple files/dirs):
# Upload
aws s3 sync ./data s3://my-bucket/data --no-progress --endpoint-url "$AWS_ENDPOINT_URL_S3"

# Download
aws s3 sync s3://my-bucket/checkpoints ./checkpoints --no-progress --endpoint-url "$AWS_ENDPOINT_URL_S3"
  • rclone (excellent knobs):
# Upload with parallelism
rclone copy ./shards mlstore:datasets/imagenet \
  --transfers=64 --checkers=8 \
  --s3-chunk-size=64M --s3-upload-concurrency=8 \
  --progress
  • s3cmd (tune multipart):
s3cmd put --recursive ./logs s3://my-bucket/logs \
  --multipart-chunk-size-mb=64 --no-progress
  • s5cmd (very fast for massive file counts):
# Copy tree with 64 concurrent workers
s5cmd --endpoint-url "$AWS_ENDPOINT_URL_S3" --numworkers 64 cp -u ./data/** s3://my-bucket/data/

Checksum sanity:

  • ETag is not a reliable MD5 for multipart/encrypted objects. Instead, maintain your own checksums.
# Generate and verify a manifest
find ./data -type f -print0 | xargs -0 sha256sum > data.sha256
aws s3 cp data.sha256 s3://my-bucket/data/ --endpoint-url "$AWS_ENDPOINT_URL_S3"

# Later:
aws s3 cp s3://my-bucket/data/data.sha256 . --endpoint-url "$AWS_ENDPOINT_URL_S3"
sha256sum -c data.sha256

4) POSIX-Like Access with s3fs (Use Judiciously)

Mounting a bucket as a filesystem is convenient for quick tooling, but not ideal for metadata-heavy or random I/O workloads. Great for simple reads/writes and compatibility.

Create credentials file:

echo "YourKeyID:YourSecret" > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Mount:

mkdir -p ~/mnt/ml-bucket

# Debian/Ubuntu
s3fs my-bucket ~/mnt/ml-bucket -o url="$AWS_ENDPOINT_URL_S3" -o use_path_request_style -o passwd_file=~/.passwd-s3fs

# Fedora/RHEL/CentOS (binary is s3fs-fuse, usage the same)
s3fs my-bucket ~/mnt/ml-bucket -o url="$AWS_ENDPOINT_URL_S3" -o use_path_request_style -o passwd_file=~/.passwd-s3fs

# openSUSE/SLES
s3fs my-bucket ~/mnt/ml-bucket -o url="$AWS_ENDPOINT_URL_S3" -o use_path_request_style -o passwd_file=~/.passwd-s3fs

Unmount:

fusermount -u ~/mnt/ml-bucket  # or: umount ~/mnt/ml-bucket

Caveats:

  • Expect higher latencies than local filesystems.

  • Avoid heavy small-file workloads. Prefer staging to local NVMe, then bulk upload.

5) Production Hardening: Lifecycle, Security, and Versioning

  • Lifecycle policies (auto-expire old artifacts, transition to cold tiers):
cat > lifecycle.json <<'EOF'
{
  "Rules": [
    {
      "ID": "expire-old-checkpoints",
      "Filter": { "Prefix": "checkpoints/" },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration file://lifecycle.json \
  --endpoint-url "$AWS_ENDPOINT_URL_S3"
  • Server-side encryption (SSE-S3/SSE-KMS) and bucket policies:
aws s3api put-bucket-encryption \
  --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]
  }' \
  --endpoint-url "$AWS_ENDPOINT_URL_S3"
  • Pre-signed URLs for secure, temporary sharing:
aws s3 presign s3://my-bucket/models/ckpt.pt --expires-in 3600 --endpoint-url "$AWS_ENDPOINT_URL_S3"
  • Versioning (recover from accidental deletes/overwrites):
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled \
  --endpoint-url "$AWS_ENDPOINT_URL_S3"
  • Data versioning patterns:
    • Keep immutable, content-addressed paths (e.g., sha256 prefixes).
    • Store manifests listing exact object keys and checksums.
    • Tag releases with a manifest pointer (simple text file).

Real-World Example: Bash-Friendly Training Workflow

Goal: Pre-stage dataset shards on fast local NVMe, train, then sync checkpoints and logs back to object storage—robust to retries and node preemption.

#!/usr/bin/env bash
set -euo pipefail

BUCKET="s3://my-bucket"
DATA_PREFIX="$BUCKET/datasets/cifar10"
RUN_PREFIX="$BUCKET/runs/$(date +%Y-%m-%d)/exp-$(uuidgen | cut -c1-8)"
ENDPOINT="${AWS_ENDPOINT_URL_S3:-https://minio.example.com}"
SCRATCH="/local_nvme"

mkdir -p "$SCRATCH/data" "$SCRATCH/out"

echo "[1/4] Prefetch dataset -> $SCRATCH/data"
rclone copy "$DATA_PREFIX" "$SCRATCH/data" \
  --transfers=32 --checkers=8 --s3-chunk-size=64M --s3-upload-concurrency=8 --progress

echo "[2/4] Train"
python train.py \
  --data "$SCRATCH/data" \
  --out "$SCRATCH/out" \
  --epochs 50 || true  # keep going to sync partial outputs on failure

echo "[3/4] Sync outputs -> $RUN_PREFIX"
aws s3 sync "$SCRATCH/out" "$RUN_PREFIX" --no-progress --endpoint-url "$ENDPOINT"

echo "[4/4] Write manifest"
find "$SCRATCH/out" -type f -print0 | xargs -0 sha256sum > "$SCRATCH/out/manifest.sha256"
aws s3 cp "$SCRATCH/out/manifest.sha256" "$RUN_PREFIX/" --endpoint-url "$ENDPOINT"

echo "Done. Artifacts: $RUN_PREFIX"

Notes:

  • Uses rclone for high-speed prefetch.

  • Uses awscli for robust final sync.

  • Keeps a checksum manifest for integrity/audit.

Self-Hosting Option: MinIO in a Minute

Great for on-prem or air-gapped clusters; S3-compatible out of the box.

Docker:

docker run -d --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=YourKeyID \
  -e MINIO_ROOT_PASSWORD=YourSecret \
  -v /data/minio:/data \
  quay.io/minio/minio server /data --console-address ":9001"

Then set:

export AWS_ACCESS_KEY_ID=YourKeyID
export AWS_SECRET_ACCESS_KEY=YourSecret
export AWS_ENDPOINT_URL_S3="http://localhost:9000"

You’re now ready to use all the same CLIs (awscli, s3cmd, rclone, s3fs) against your local MinIO.

Common Pitfalls and Pro Tips

  • Many small files kill throughput. Tar/zip/shard when possible; aim for 64–512 MB per part.

  • For S3-compatible endpoints, prefer path-style addressing and set the endpoint explicitly.

  • Keep retries/idempotency in mind; design scripts so they can be safely re-run.

  • Monitor with simple metrics: transfers/sec, MB/s, and tail server logs for 4xx/5xx errors.

  • Test both upload and download performance; distinct bottlenecks often appear.

Conclusion and Call to Action

Object storage gives AI teams the scalability and price/performance they need—without surrendering the simplicity of Bash. Your next steps:

1) Install the core tools with your package manager.
2) Wire up credentials and endpoints.
3) Run a 100–500 GB transfer test; tune concurrency and multipart sizes.
4) Decide where you need POSIX-like mounts vs. staged local NVMe.
5) Add lifecycle policies, encryption, and manifests for production.

When your storage stops being the bottleneck, your experiments get faster—and you ship models sooner. Grab your bucket and go build.

Related posts