Posted on
• Artificial Intelligence

Artificial Intelligence Cloud Cost Optimisation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Cloud Cost Optimisation: A Bash-First Playbook

If your AI cloud bill can double overnight, you’re not alone. GPU instances are expensive, storage grows without bound, and “just one more experiment” can burn thousands. The good news: you can take back control with simple, automation-friendly Bash tactics that deliver fast, measurable savings—without slowing your model velocity.

This article explains why AI cloud cost optimisation matters, then gives you 3–5 actionable, real-world steps (with Bash you can copy/paste) to cut waste, right-size compute, and make preemptible capacity practical. Installation instructions are provided for apt, dnf, and zypper wherever tools are cited.

Why this is worth your time

  • AI workloads are bursty and hardware-sensitive. Choosing the wrong GPU size or not switching instance types at the right time can double costs for the same result.

  • Idle time is the silent budget killer. Even a few hours/day of idle GPUs can add up to thousands per month.

  • Preemptible/spot instances are cheap but intimidating. With checkpointing and simple signal handling, they become reliable workhorses.

  • Storage and egress can quietly exceed compute costs. Object storage, lifecycle policies, and smart syncing tame this.

Bash is everywhere, auditable, and easy to automate via cron or CI. Let’s put it to work.

Prerequisites: install the core tools

We’ll use standard Linux tools and optional cloud CLIs. Install what you need for your distro.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y jq bc sysstat rclone awscli
sudo systemctl enable --now sysstat
  • Fedora/RHEL/CentOS Stream (dnf):
sudo dnf install -y jq bc sysstat rclone awscli
sudo systemctl enable --now sysstat
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y jq bc sysstat rclone aws-cli
sudo systemctl enable --now sysstat

Notes:

  • nvidia-smi comes with the NVIDIA driver; install your vendor’s driver if you need GPU metrics.

  • rclone requires a one-time rclone config to connect to S3/Google Cloud Storage/Azure Blob.

  • awscli requires credentials and IAM permissions; for cost APIs, Cost Explorer must be enabled in the console.

1) Profile first, then right-size your AI compute

Before switching instance types or GPUs, measure real usage. If your GPU sits <40% utilisation during training or inference, you’re overpaying.

  • Watch GPU usage live:
watch -n 5 'nvidia-smi --query-gpu=name,utilization.gpu,memory.used --format=csv'
  • Quick CPU utilisation check:
sar -u 1 5
  • Script: average GPU utilisation over a window (across all GPUs)
sudo tee /usr/local/bin/gpu-avg.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
dur="${1:-60}"
if ! command -v nvidia-smi >/dev/null 2>&1; then
  echo "nvidia-smi not found; GPU metrics unavailable" >&2
  exit 1
fi
end=$((SECONDS+dur))
sum=0; n=0
while [ $SECONDS -lt $end ]; do
  inst_avg=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null \
             | awk '{s+=$1; c+=1} END{if(c>0) print int(s/c); else print 0}')
  sum=$((sum+inst_avg)); n=$((n+1))
  sleep 1
done
echo $((sum/n))
EOF
sudo chmod +x /usr/local/bin/gpu-avg.sh
  • Sanity-check effective cost at your current utilisation:
COST_PER_HOUR="2.50"   # edit: instance $/hour
UTIL=$(gpu-avg.sh 60)  # average over 60s
if [ "$UTIL" -eq 0 ]; then
  echo "GPU util 0%; effective $/GPU-hr: infinite (all waste)"
else
  echo "Avg util: ${UTIL}%"
  echo -n "Effective $/100%-util GPU-hr: "
  echo "scale=2; $COST_PER_HOUR*100/$UTIL" | bc
fi

If that “effective cost” is eye-watering, try:

  • Smaller/cheaper GPUs (e.g., L4/L40s/T4 for lighter inference).

  • Mixed precision, flash attention, or kernel-optimised inference servers to raise utilisation.

  • CPU offload and dataloader tuning to relieve GPU stalls.

2) Make spot/preemptible compute safe with checkpointing

Preemptible capacity can be 50–90% cheaper, but you must handle termination. Two simple practices make it robust:

  • Save checkpoints frequently (and atomically) to object storage.

  • Trap SIGTERM and respond to preemption notices.

Example wrapper that traps termination and syncs checkpoints with rclone:

cat >/tmp/train_with_checkpoint.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

CHECKPOINT_DIR="${CHECKPOINT_DIR:-/workspace/checkpoints}"
REMOTE="${REMOTE:-s3:ai-checkpoints/my-run}"  # requires 'rclone config'

checkpoint() {
  echo "[checkpoint] $(date -Is) syncing $CHECKPOINT_DIR -> $REMOTE"
  rclone sync --fast-list --checksum "$CHECKPOINT_DIR" "$REMOTE"
}

trap 'echo "[signal] TERM received"; checkpoint; exit 143' TERM INT

# Optional: AWS Spot two-minute interruption notice
( while sleep 5; do
    if curl -fsS --max-time 1 http://169.254.169.254/latest/meta-data/spot/instance-action >/dev/null; then
      echo "[preemption] notice received"
      checkpoint
      sleep 60
      exit 0
    fi
  done ) &

# Your actual training command
python train.py --save-dir "$CHECKPOINT_DIR"
EOF
bash /tmp/train_with_checkpoint.sh

Tips:

  • Use smaller, frequent checkpoints (e.g., every N steps/minutes).

  • Store immutable artifacts (datasets, pretrained weights) in shared object storage to make rescheduling trivial.

3) Auto-stop idle GPUs with a lightweight cron job

Kill waste automatically. This script stops an EC2 instance when both GPU and CPU are idle for a sustained period. If no cloud CLI is configured, it falls back to powering off the machine.

sudo tee /usr/local/bin/ai-idle-stop.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

GPU_THRESH="${GPU_THRESH:-10}"        # % GPU util below which we consider idle
CPU_IDLE_THRESH="${CPU_IDLE_THRESH:-80}" # % CPU idle above which we consider idle
IDLE_WINDOW="${IDLE_WINDOW:-900}"     # seconds required idle (default 15m)
STATE_FILE="/var/run/ai-idle.state"

gpu_idle=true
if command -v nvidia-smi >/dev/null 2>&1; then
  util=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null \
         | awk '{s+=$1; c+=1} END{if(c>0) print int(s/c); else print 0}')
  [ "${util:-0}" -lt "$GPU_THRESH" ] || gpu_idle=false
fi

# CPU idle using sysstat (mpstat)
cpu_idle=100
if command -v mpstat >/dev/null 2>&1; then
  cpu_idle=$(LC_ALL=C mpstat 1 3 | awk '/Average.*all/ {print $NF+0}')
fi

cpu_ok=false
awk -v idle="$cpu_idle" -v thr="$CPU_IDLE_THRESH" 'BEGIN{exit !(idle>=thr)}' && cpu_ok=true

now=$(date +%s)
if $gpu_idle && $cpu_ok; then
  if [ -e "$STATE_FILE" ]; then
    first=$(cat "$STATE_FILE")
  else
    first=$now
  fi
  echo "$first" > "$STATE_FILE"
  if [ $((now-first)) -ge "$IDLE_WINDOW" ]; then
    echo "$(date -Is) idle window met; attempting to stop instance"
    if command -v aws >/dev/null 2>&1 && curl -fsS --max-time 1 http://169.254.169.254/latest/meta-data/instance-id >/dev/null; then
      IID=$(curl -fsS http://169.254.169.254/latest/meta-data/instance-id)
      REGION=$(curl -fsS http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
      aws ec2 stop-instances --instance-ids "$IID" --region "$REGION" >/dev/null 2>&1 || true
      echo "$(date -Is) sent stop to EC2 instance $IID in $REGION"
    else
      echo "$(date -Is) powering off locally"
      systemctl poweroff || shutdown -h now
    fi
  else
    echo "$(date -Is) idle but window not met yet: $((now-first))/$IDLE_WINDOW s"
  fi
else
  echo "$now" > "$STATE_FILE"  # reset window
  echo "$(date -Is) busy: gpu_idle=$gpu_idle cpu_idle=${cpu_idle}%"
fi
EOF
sudo chmod +x /usr/local/bin/ai-idle-stop.sh

Schedule it every 5 minutes as root:

sudo crontab -e
# add:
*/5 * * * * /usr/local/bin/ai-idle-stop.sh >> /var/log/ai-idle-stop.log 2>&1

Environment knobs (set in crontab if desired):

GPU_THRESH=15 CPU_IDLE_THRESH=85 IDLE_WINDOW=1200 /usr/local/bin/ai-idle-stop.sh

4) Optimise storage and data movement

  • Keep large, cold data in object storage (S3/GCS/Azure Blob); stage hot shards locally.

  • Sync checkpoints/logs out quickly to avoid losing work; prune aggressively.

Examples with rclone:

# One-time setup
rclone config

# Push checkpoints and logs (fast, checksum-aware)
rclone sync --fast-list --checksum /workspace/checkpoints s3:ai-checkpoints/run-123
rclone copy --fast-list /workspace/logs s3:ai-logs/run-123

# Periodic prune of old local artifacts (keep last 5)
ls -1dt /workspace/checkpoints/* | tail -n +6 | xargs -r rm -rf

Consider object storage lifecycle policies (transition to infrequent access or cold tiers after N days, expire after M days). This alone can cut storage bills dramatically for long-running experiments.

5) Tag and measure spend from the shell

Tag everything (instances, buckets, jobs) so you can slice spend by project or team. Then query costs programmatically.

  • Tag current EC2 instance with your project:
IID=$(curl -fsS http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -fsS http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
aws ec2 create-tags --resources "$IID" --region "$REGION" \
  --tags Key=Project,Value=nlp-finetune Key=Owner,Value=$USER
  • Quick Cost Explorer query by tag (last 7 days):
START=$(date -u -d '7 days ago' +%Y-%m-%d)
END=$(date -u +%Y-%m-%d)
aws ce get-cost-and-usage --region us-east-1 \
  --time-period Start=$START,End=$END \
  --granularity DAILY \
  --metrics UnblendedCost \
  --group-by Type=TAG,Key=Project \
| jq '.ResultsByTime[] | {date: .TimePeriod.Start, groups: .Groups}'

Use these reports to spot runaway projects, underutilised fleets, and to validate that your automation is paying off.

A quick, real-world-style workflow

  • Profiling shows your fine-tune job averages 45% GPU util on a large GPU.

  • Switch to a cheaper GPU type that still fits your model, enable mixed precision, and increase dataloader workers; utilisation climbs to ~75%.

  • Wrap training with the checkpoint script and move to spot/preemptible pools; interruptions no longer hurt.

  • Enable the idle-stop cron: after jobs finish, instances self-stop within 15 minutes.

  • Push logs/checkpoints to object storage with rclone; set a 30-day lifecycle to cold storage and 90-day expiration.

  • Tag everything; validate spend trends via the CLI each week.

Result: lower effective $/GPU-hr, fewer idle hours, and less storage waste—without sacrificing iteration speed.

Conclusion and next steps (CTA)

You don’t need a massive platform overhaul to slash AI cloud costs. Start with the 80/20:

  • Measure utilisation, then right-size.

  • Make spot/preemptible safe with checkpoints.

  • Auto-stop idle compute.

  • Sync and prune storage.

  • Tag and measure spend continuously.

Copy the scripts above, set up the cron job, and run a one-week experiment. Track your savings with the cost query. When you’re ready, expand to job schedulers, heterogeneous fleets, and deeper model-level optimisations.

If you want a follow-up post with provider-specific recipes (AWS/GCP/Azure) or container-native versions of these scripts, say the word.

Related posts