AI Capacity Forecasting on Linux with Pure Bash: From Raw Metrics to Next-Week Plans

You’ve booked eight expensive GPUs for a training run, but half your cluster sits idle in the afternoon—while queues explode overnight. Sound familiar? This is the hidden tax of modern AI: not just compute cost, but poorly matched supply and demand.

This post shows how to forecast AI capacity needs directly on Linux with Bash. You’ll collect the right signals (GPU, CPU, memory), normalize them into “hours” you can plan with, and produce a simple next-24-hours forecast—no heavy frameworks required. We’ll include optional installs using apt, dnf, and zypper, and close with ways to turn forecasts into savings and SLO wins.

Why AI capacity forecasting matters

• AI spend is dominated by compute. Mismatches (idle or overloaded) burn money and delay jobs.

• Demand is seasonal. Human work hours, nightly retrains, or batch backfills drive patterns you can predict.

• Forecasting informs scheduling (batch windows), autoscaling (K8s/Slurm), and procurement (GPUs/nodes).

• Even simple methods (hour-of-day averages) cut waste drastically before you consider more advanced models.

What you’ll build

• A tiny Bash collector that samples GPU/CPU/RAM every minute into CSV

• An aggregator that converts raw signals into normalized units (GPU-hours, CPU-core-hours)

• A seasonal hour-of-day forecast for the next 24 hours, also in Bash/AWK

• Optional: RRDtool Holt–Winters forecasting

All code runs on any Linux distro with standard tools. Optional packages are listed below.

Prerequisites and installation

The core pipeline uses Bash, awk, and /proc—no installs required. The following tools are optional but handy:

• sysstat (mpstat/iostat) for cross-checking utilization

• RRDtool for Holt–Winters (advanced)

• gnuplot for quick visualization

• bc for safe arithmetic in scripts

Install with your package manager of choice:

• Debian/Ubuntu (apt):

  sudo apt update
  sudo apt install -y sysstat rrdtool gnuplot bc
  

• Fedora/RHEL/CentOS Stream/Rocky (dnf):

  sudo dnf install -y sysstat rrdtool gnuplot bc
  

• openSUSE/SLES (zypper):

  sudo zypper refresh
  sudo zypper install -y sysstat rrdtool gnuplot bc
  

Note: If you have NVIDIA GPUs, the proprietary driver provides nvidia-smi. Installation methods vary by distro and driver version—use your vendor/distro guide. The scripts below auto-detect nvidia-smi and degrade gracefully if it’s not present.

---

Step 1: Decide what to measure (and how to normalize)

Raw percentages can’t be budgeted; “hours” can. Normalize your metrics:

• GPU-hours: Sum over time of (avg GPU utilization fraction × number of GPUs × hours).

• CPU-core-hours: (CPU utilization fraction × number of cores × hours).

• Memory GB-hours: (used GB × hours), useful for inference services.

• Queue-hours (optional): Useful if you run schedulers (Slurm/K8s) to capture unmet demand.

We’ll compute GPU-hours and CPU-core-hours directly from one-minute samples.

---

Step 2: Collect metrics every minute (Bash-only)

Create a lightweight collector that appends to a CSV. It reads /proc for CPU/RAM, optionally uses nvidia-smi for GPU.

Save as collector.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG="${LOG:-$HOME/ai-metrics.csv}"

# Write CSV header once
if [ ! -s "$LOG" ]; then
  echo "epoch,gpu_util_avg_pct,gpu_mem_used_mb,cpu_util_pct,mem_used_mb" > "$LOG"
fi

cpu_util_pct() {
  # Compute CPU utilization over ~1s from /proc/stat
  read cpu user nice system idle iowait irq softirq steal guest guest_n < /proc/stat
  total1=$((user+nice+system+idle+iowait+irq+softirq+steal))
  idle1=$((idle+iowait))
  sleep 1
  read cpu user nice system idle iowait irq softirq steal guest guest_n < /proc/stat
  total2=$((user+nice+system+idle+iowait+irq+softirq+steal))
  idle2=$((idle+iowait))
  dt=$((total2-total1))
  didle=$((idle2-idle1))
  if [ "$dt" -le 0 ]; then echo 0; return; fi
  # Utilization percentage
  awk -v dt="$dt" -v didle="$didle" 'BEGIN{printf "%.2f", (1 - didle/dt) * 100}'
}

mem_used_mb() {
  # Use MemAvailable when present (modern kernels)
  awk '
    /MemTotal:/ {mt=$2}
    /MemAvailable:/ {ma=$2}
    END {
      if (mt>0 && ma>0) used=(mt-ma); else used=0;
      printf "%.0f", used/1024
    }' /proc/meminfo
}

gpu_stats() {
  if command -v nvidia-smi >/dev/null 2>&1; then
    # Average utilization across GPUs; sum memory used
    nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader,nounits \
    | awk -F, '
        {gsub(/ /,"",$1); gsub(/ /,"",$2); u+=$1; m+=$2; n++}
        END {
          if(n>0) printf "%.2f,%.0f\n", u/n, m; else print "0.00,0"
        }'
  else
    echo "0.00,0"
  fi
}

main() {
  ts=$(date +%s)
  cpu=$(cpu_util_pct)
  mem=$(mem_used_mb)
  IFS=, read -r gpu_util gpu_mem < <(gpu_stats)

  echo "$ts,$gpu_util,$gpu_mem,$cpu,$mem" >> "$LOG"
}

main

Run it periodically:

• Cron (every minute):

  * * * * * /path/to/collector.sh
  

• Or a systemd timer (preferred for jitter control).

Let this run for at least one week to capture weekday/weekend patterns.

---

Step 3: Aggregate into hourly GPU-hours and CPU-core-hours

We’ll convert one-minute samples into hourly “hours” using your installed capacity. Export your capacities (or hardcode them):

• GPU_COUNT: total GPUs on the node

• CPU_CORES: nproc result or your scheduling quota

Create aggregate.sh:

#!/usr/bin/env bash
set -euo pipefail

INPUT="${1:-$HOME/ai-metrics.csv}"
GPU_COUNT="${GPU_COUNT:-0}"   # e.g., 4
CPU_CORES="${CPU_CORES:-$(nproc)}"
STEP_SEC="${STEP_SEC:-60}"    # sampling period; 60 if using cron/minutely

if [ ! -f "$INPUT" ]; then
  echo "Missing input: $INPUT" >&2
  exit 1
fi

# Output: epoch_hour,iso_hour,gpu_hours,cpu_core_hours,avg_vram_gb,avg_mem_gb
awk -F, -v step="$STEP_SEC" -v gpus="$GPU_COUNT" -v cores="$CPU_CORES" '
BEGIN {
  OFS=","
}
NR==1 { next } # skip header
{
  # Bucket by hour
  epoch=$1
  hour_epoch = epoch - (epoch % 3600)
  key = hour_epoch
  gpu_util=$2+0
  gpu_mem_mb=$3+0
  cpu_util=$4+0
  mem_mb=$5+0

  # Convert to hours: util_frac * capacity * (step/3600)
  gpu_hours[key] += (gpu_util/100.0) * gpus * (step/3600.0)
  cpu_core_hours[key] += (cpu_util/100.0) * cores * (step/3600.0)

  vram_sum_mb[key] += gpu_mem_mb
  mem_sum_mb[key]  += mem_mb
  count[key]++
  if (key > maxkey) maxkey = key
}
END {
  print "epoch_hour,iso_hour,gpu_hours,cpu_core_hours,avg_vram_gb,avg_mem_gb"
  PROCINFO["sorted_in"] = "@ind_num_asc"
  for (k in count) {
    iso = strftime("%Y-%m-%d %H:00:00", k)
    avg_vram_gb = (vram_sum_mb[k]/count[k])/1024.0
    avg_mem_gb  = (mem_sum_mb[k]/count[k])/1024.0
    printf "%d,%s,%.4f,%.4f,%.3f,%.3f\n", k, iso, gpu_hours[k], cpu_core_hours[k], avg_vram_gb, avg_mem_gb
  }
}' "$INPUT" | sort -n

Usage:

GPU_COUNT=4 CPU_CORES=32 ./aggregate.sh > hourly.csv

Now hourly.csv is planning-ready.

---

Step 4: Forecast the next 24 hours (seasonal hour-of-day)

We’ll build a simple, robust forecast: for each hour of day (0–23), use the historical average of that hour. This captures daily seasonality without overfitting. If you have <7 days, it still works—just less stable.

Create forecast.sh:

#!/usr/bin/env bash
set -euo pipefail

INPUT="${1:-hourly.csv}"
HORIZON_HOURS="${HORIZON_HOURS:-24}"

if [ ! -f "$INPUT" ]; then
  echo "Missing input: $INPUT" >&2
  exit 1
fi

# Input columns: epoch_hour,iso_hour,gpu_hours,cpu_core_hours,avg_vram_gb,avg_mem_gb
awk -F, -v H="$HORIZON_HOURS" '
NR==1 { next } # skip header
{
  epoch=$1+0
  gpu=$3+0
  cpu=$4+0
  hod=strftime("%H", epoch)+0  # 0..23
  hod_count[hod]++
  hod_gpu_sum[hod]+=gpu
  hod_cpu_sum[hod]+=cpu
  if (epoch > max_epoch) max_epoch=epoch
}
END {
  # Build hour-of-day averages
  for (h=0; h<24; h++) {
    if (hod_count[h]>0) {
      hod_gpu_avg[h] = hod_gpu_sum[h]/hod_count[h]
      hod_cpu_avg[h] = hod_cpu_sum[h]/hod_count[h]
    } else {
      # fallback to overall mean
      hod_gpu_avg[h] = (hod_gpu_sum[h]+0)/(hod_count[h]+1)
      hod_cpu_avg[h] = (hod_cpu_sum[h]+0)/(hod_count[h]+1)
    }
  }

  print "epoch_hour,iso_hour,fcast_gpu_hours,fcast_cpu_core_hours"

  for (i=1; i<=H; i++) {
    ts = max_epoch + i*3600
    h  = strftime("%H", ts)+0
    iso = strftime("%Y-%m-%d %H:00:00", ts)
    printf "%d,%s,%.4f,%.4f\n", ts, iso, hod_gpu_avg[h], hod_cpu_avg[h]
  }
}' "$INPUT"

Usage:

./forecast.sh hourly.csv > forecast_24h.csv

Now you have a next-24-hours forecast of GPU-hours and CPU-core-hours on this node. For a cluster, either:

• Run per node and sum the CSVs, or

• Collect/aggregate at a central log and then run these scripts once cluster-wide.

Tip: To compute a quick “capacity gap,” compare forecasts to what you can actually deliver in an hour:

# Example: 4 GPUs, 32 cores -> per-hour capacity is 4 GPU-hours and 32 core-hours.
GPU_CAP_PER_HOUR=4
CPU_CAP_PER_HOUR=32

awk -F, -v gcap="$GPU_CAP_PER_HOUR" -v ccap="$CPU_CAP_PER_HOUR" '
NR==1{print $0",gpu_gap,cpu_gap"; next}
{
  gpu_gap = gcap - $3   # positive = spare, negative = shortfall
  cpu_gap = ccap - $4
  printf "%s,%.2f,%.2f\n", $0, gpu_gap, cpu_gap
}' forecast_24h.csv

---

Step 5: Turn forecasts into actions

• Schedule around valleys: If your forecast shows a daily dip 02:00–05:00, push batch training or large shuffles there.

• Autoscale nodes/pools: Use gaps/shortfalls to pre-scale GPU pools in K8s or Slurm partitions (e.g., launch nodes 15–20 min before the peak).

• Budget and procure: Roll up daily GPU-hours trends into weekly/monthly growth. If your 95th percentile forecast exceeds installed capacity for >N hours/week, open a procurement ticket.

• SLO guardrails: If the shortfall crosses a threshold, reject new ad-hoc jobs or downgrade non-critical inference replicas.

Example trigger (pseudo):

# If any of the next 24 hours shows >10% GPU shortfall, page/oncall or scale out
awk -F, -v cap_per_hr=4 -v thresh=0.10 '
NR==1{next}
{
  gap = cap_per_hr - $3
  if (gap < -cap_per_hr*thresh) {
    print "Shortfall at", $2, "by", -gap, "GPU-hours"; exit 42
  }
}' forecast_24h.csv

if [ $? -eq 42 ]; then
  echo "Scale out signal here (e.g., kubectl/Slurm command)"
fi

---

Optional: Holt–Winters forecasts with RRDtool

If you want built-in seasonal forecasting with confidence bands, RRDtool’s Aberrant Behavior Detection supports Holt–Winters. Install it as shown above, then:

Create an RRD with seasonality (e.g., 24h season at 5-min step = 288 points):

rrdtool create ai_gpu.rrd --step 300 \
  DS:gpu_hours:GAUGE:600:0:U \
  RRA:AVERAGE:0.5:1:4032 \
  RRA:HWPREDICT:288:0.1:0.0035:288 \
  RRA:SEASONAL:288:0.1:2 \
  RRA:DEVSEASONAL:288:0.1:2 \
  RRA:DEVPREDICT:288:5

Every 5 minutes, update with the last 5 minutes of GPU-hours (sum of per-minute samples over 5 min):

# Example: timestamp is the end of the 5-min interval
rrdtool update ai_gpu.rrd ${TS}:${GPU_HOURS_LAST_5MIN}

Graph with prediction bands:

rrdtool graph gpu-forecast.png --start end-2d --end now \
  DEF:gh=ai_gpu.rrd:gpu_hours:AVERAGE \
  DEF:pred=ai_gpu.rrd:gpu_hours:HWPREDICT \
  DEF:dev=ai_gpu.rrd:gpu_hours:DEVPREDICT \
  CDEF:upper=pred,dev,2,*,+ \
  CDEF:lower=pred,dev,2,*,- \
  LINE1:gh#00ccff:"GPU-hours (obs)" \
  LINE1:pred#ff9900:"Holt-Winters pred" \
  LINE1:upper#aaaaaa:"Upper band" \
  LINE1:lower#aaaaaa:"Lower band"

This gives you seasonal predictions and bands that highlight anomalies.

---

Real-world patterns to watch for

• Weekday peaks / weekend troughs: Shift nightly retrains into troughs without hurting SLAs.

• Payday/reporting cycles: Anticipate end-of-month spikes in inference or ETL.

• Deployment days: New models often increase memory/GPU-hour profiles—plan headroom for rollouts.

---

Conclusion and next steps

Capacity forecasting doesn’t have to start with a massive observability stack. With a few lines of Bash you can:

1) Collect the right low-level signals 2) Normalize them into hours you can plan and budget 3) Forecast the next day with simple, seasonally-aware logic 4) Take concrete actions: schedule, autoscale, and procure

Your next step:

• Deploy the collector and aggregator today, let it run for 7–14 days.

• Generate your first forecast and compare it to reality—tune sampling and capacity variables.

• Add actions: a cron that scales pools or opens tickets when shortfalls are predicted.

If you want a deeper dive, bolt on RRDtool Holt–Winters or export your CSVs to your preferred time-series DB—but don’t wait for perfect. The Bash you have today can save you real money tomorrow.