Posted on
• Artificial Intelligence

Artificial Intelligence VM Capacity Planning

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence VM Capacity Planning: A Bash‑First Playbook

Your model is ready, but your cluster is on fire. GPUs are idle, CPUs are pegged, and p95 latency is creeping past your SLA. AI workloads are unforgiving: a small miscalculation in VRAM or IOPS can waste thousands per month or melt your SLOs. This post gives Linux and Bash practitioners a practical, command‑line oriented way to plan VM capacity for AI training and inference—so you can ship faster without overspending.

What problem are we solving?

AI workloads don’t look like “normal” web services. They are:

  • GPU‑bound (VRAM and tensor throughput dominate)

  • Memory‑intensive (activation/optimizer state for training; KV cache for LLM inference)

  • I/O‑sensitive (dataset streaming, checkpoints, feature stores)

  • Latency‑sensitive (p95/p99 inference latencies drive user experience)

Capacity planning translates business goals (throughput, latency, cost) into VM shapes (vCPU, RAM, GPU VRAM/cores, storage, NIC). Doing this with a Bash‑centric workflow keeps it reproducible, automatable, and distro‑friendly.

Why this matters now

  • Training and serving are different beasts: the same model can need 10× the memory when training vs. serving.

  • GPUs are scarce and expensive: right‑sizing VRAM and batch sizes matters.

  • Virtualization adds complexity: NUMA, hugepages, vCPU pinning, GPU passthrough/MIG all affect the outcome.

  • You can’t optimize what you don’t measure: lightweight, scriptable measurements beat guesses.

Core blueprint: 5 actionable steps (with Bash)

1) Classify the workload and do a back‑of‑the‑envelope sizing

Start by declaring the workload type, SLA, and “shape” (params, precision, batch, sequence length). Then estimate VRAM and RAM. These formulas are intentionally conservative to avoid under‑provisioning.

Rules of thumb:

  • Inference VRAM ≈ model_params × bytes_per_param × 1.2–2.0 + KV_cache

  • Training VRAM (Adam, mixed precision) often ≈ model_params × bytes_per_param × 6–12

  • System RAM ≈ 2–4 × GPU VRAM for data pipelines and CPU staging

  • Storage throughput: aim for sustained read IOPS/MBps that cover peak batch staging

  • Network: for distributed training or model sharding, ensure NIC ≥ dataset stream + inter‑GPU comms

Quick Bash estimator:

#!/usr/bin/env bash
# ai-vram-estimator.sh
# Usage: ./ai-vram-estimator.sh <params_billion> <precision> <workload> <batch> <seq_len>
# Example: ./ai-vram-estimator.sh 7 fp16 inference 1 2048

set -euo pipefail

P_BILLION="${1:-7}"
PRECISION="${2:-fp16}"     # fp32|fp16|bf16|int8|int4
WORKLOAD="${3:-inference}" # inference|training
BATCH="${4:-1}"
SEQ="${5:-2048}"

# Bytes per parameter by precision (approx)
case "$PRECISION" in
  fp32) BPP=4 ;;
  fp16|bf16) BPP=2 ;;
  int8) BPP=1 ;;
  int4) BPP=0.5 ;;
  *) echo "Unknown precision: $PRECISION" >&2; exit 1 ;;
esac

PARAMS=$(awk -v b="$P_BILLION" 'BEGIN{printf "%.0f", b*1e9}')
# Base model weights
WEIGHTS_BYTES=$(awk -v p="$PARAMS" -v b="$BPP" 'BEGIN{printf "%.0f", p*b}')

# Training multiplier (very rough; accounts for optimizer + grads + activations)
if [[ "$WORKLOAD" == "training" ]]; then
  # Mixed precision typical range 6–12 bytes/param equivalent
  MULT=8
else
  MULT=1.6 # inference overhead factor
fi

CORE_BYTES=$(awk -v w="$WEIGHTS_BYTES" -v m="$MULT" 'BEGIN{printf "%.0f", w*m}')

# KV cache rough estimate for decoder‑only LLMs: ~ hidden_state_bytes * tokens
# Use a very rough proxy: 2 * params_per_token relationship isn't exact, but provides a scale.
# For small batch/seq it's small; grows with BATCH*SEQ.
KV_BYTES=$(awk -v p="$PARAMS" -v b="$BATCH" -v s="$SEQ" -v BPP="$BPP" 'BEGIN{
  # Heuristic: scale KV cache with params^0.5 to reflect hidden size scaling
  hs = sqrt(p) * 4096; # proxy hidden size
  bytes_per_token = hs * BPP * 2; # K and V
  total = b * s * bytes_per_token;
  printf "%.0f", total
}')

TOTAL_BYTES="$CORE_BYTES"
if [[ "$WORKLOAD" == "inference" ]]; then
  TOTAL_BYTES=$(awk -v c="$CORE_BYTES" -v k="$KV_BYTES" 'BEGIN{printf "%.0f", c+k}')
fi

toGiB() { awk -v x="$1" 'BEGIN{printf "%.2f", x/1024/1024/1024}'; }

echo "Params (B):            $P_BILLION"
echo "Precision:             $PRECISION"
echo "Workload:              $WORKLOAD"
echo "Batch x Seq:           $BATCH x $SEQ"
echo "Weights (GiB):         $(toGiB "$WEIGHTS_BYTES")"
[[ "$WORKLOAD" == "inference" ]] && echo "KV cache (GiB):         $(toGiB "$KV_BYTES")"
echo "Est. VRAM need (GiB):  $(toGiB "$TOTAL_BYTES")"
# System RAM recommendation
RAM_GIB=$(awk -v v="$TOTAL_BYTES" 'BEGIN{printf "%.0f", (v/1024/1024/1024)*3}') # 3x VRAM
echo "Est. System RAM (GiB): $RAM_GIB"

Example runs:

  • 7B, fp16, inference, batch=1, seq=2048 → fits well on a 24 GB GPU with headroom

  • 13B, fp16, inference, batch=1, seq=2048 → often wants 30–40+ GB VRAM unless quantized

  • Training the same models may require 6–12× the parameter memory

2) Survey the host: collect CPU, NUMA, memory, disk, GPU, and NIC baselines

Install lightweight tools and capture a minimum useful baseline.

Install common tools:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop
  • Fedora/RHEL/CentOS (dnf):
# On RHEL-like, nvtop may require EPEL:
# sudo dnf install -y epel-release
sudo dnf install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop
  • openSUSE/SLE (zypper):
sudo zypper refresh
sudo zypper install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop

Run a 60‑second baseline sampler:

#!/usr/bin/env bash
# host-survey.sh
set -euo pipefail
OUT="${1:-host_survey_$(date +%F_%H%M%S).txt}"

{
  echo "=== CPU / NUMA ==="
  lscpu
  numactl -H || true
  echo
  echo "=== MEMORY ==="
  free -h
  vmstat 1 5
  echo
  echo "=== DISK ==="
  lsblk -o NAME,TYPE,FSTYPE,SIZE,MOUNTPOINT
  iostat -xz 1 5
  echo
  echo "=== NETWORK ==="
  ip -br a
  ss -s || true
  echo
  echo "=== GPU ==="
  nvidia-smi || true
  nvtop --json | jq . 2>/dev/null || true
} | tee "$OUT"

echo "Baseline saved to $OUT"

For storage throughput, a quick read test on your dataset volume:

# Adjust --filename to your dataset/path
fio --name=readtest --filename=/data/dataset.bin --rw=randread --bs=256k --iodepth=16 --numjobs=2 --runtime=60 --time_based --group_reporting

Network sanity check:

# Run iperf3 -s on a peer; here we run a client test:
iperf3 -c <server-ip> -t 30

3) Choose VM shapes and overcommit policy (CPU, RAM, GPU, storage, net)

  • CPU

    • Training: keep vCPU:pCPU ≤ 1:1 for steady throughput; inference: 2:1 to 4:1 can work if GPU‑bound and latency budget allows.
    • Pin latency‑sensitive workloads: use libvirt CPU pinning and align with NUMA.

  • Memory

    • Prefer no swap for GPU‑bound trainers; for inference VMs, a small swap can help burst but watch tail latency.
    • Transparent Huge Pages “madvise” or explicit hugepages can reduce TLB misses.

  • GPU

    • Single‑tenant passthrough for training; MIG/vGPU partitions for multi‑tenant inference on A100/H100 class GPUs.
    • Right‑size VRAM to model + overhead; avoid “VRAM fragmentation” from too many dissimilar workloads on one device.

  • Storage

    • Use local NVMe for scratch and checkpoints; object storage for cold datasets; network FS for shared evaluation sets with read‑heavy caching.

  • Network

    • Inference autoscaling: size NIC for egress and model shard sync; distributed training: ensure low‑latency fabric or at least 25–100 Gbps links.

Real‑world mapping examples:

  • LLM serving (7B, fp16, batch 1–4, seq ≤ 2k): 1× 24 GB GPU (e.g., L4/A10/T4‑ish), 8–16 vCPU, 32–64 GB RAM, NVMe scratch.

  • LLM serving (13B, fp16): 1× 40 GB GPU or 2× 24 GB with tensor/pp splitting; or int8/int4 quantization on 24 GB.

  • Finetuning 7B (LoRA, bf16): 1× 40 GB+ GPU or 2× 24 GB; 16–32 vCPU; 64–128 GB RAM; fast NVMe for checkpoints.

Optional (A100/H100) MIG partitioning example:

# WARNING: Disruptive; stop workloads first.
sudo nvidia-smi -i 0 -mig 1
# Example: create 2 MIG instances (check nvidia-smi mig -lgip for profiles on your GPU)
sudo nvidia-smi mig -i 0 -cgi 19,19 -C
nvidia-smi -L

4) Build repeatable capacity tests

Synthetic, fast checks keep you honest before spending hours on a full pipeline.

CPU/memory pressure:

stress-ng --cpu 8 --vm 2 --vm-bytes 2G --timeout 60s --metrics-brief

Disk throughput and latency:

fio --name=ai-randread --filename=/data/scratch.img --size=8G --rw=randread --bs=128k --iodepth=32 --runtime=60 --time_based --group_reporting
fio --name=ai-checkpoint --filename=/data/ckpt.img --size=16G --rw=write --bs=1M --iodepth=8 --runtime=60 --time_based --group_reporting

Network:

iperf3 -c <peer> -t 20 -P 4

GPU quick sanity (assuming Python/torch environment exists):

python - <<'PY'
import torch, time
d = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = torch.randn(4096,4096, device=d)
t0=time.time()
for _ in range(200): y = x@x; torch.cuda.synchronize()
print("OK, elapsed:", time.time()-t0, "sec")
PY

Track basic inference latency under load (HTTP):

# apt/dnf/zypper install -y hey (on some distros hey is in golang/contrib repos; if not, use wrk or ab)
# Fallback using curl in a loop:
URL="http://localhost:8000/infer"
for i in {1..50}; do /usr/bin/time -f '%E' curl -s -X POST "$URL" -d '{"input":"hello"}' -H 'Content-Type: application/json' >/dev/null; done

5) Operationalize: KVM/libvirt setup, monitoring, and guardrails

Install virtualization stack:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system virtinst virt-manager
sudo systemctl enable --now libvirtd
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y @virtualization virt-install virt-manager
sudo systemctl enable --now libvirtd
  • openSUSE/SLE (zypper):
sudo zypper install -y qemu-kvm libvirt virt-install virt-manager
sudo systemctl enable --now libvirtd

NUMA‑aware VM definition (example snippet):

# Create a VM with CPU pinning and hugepages (edit as needed)
virt-install \
  --name ai-infer-1 \
  --memory 65536 --memorybacking hugepages=on \
  --vcpus 16 \
  --cpu host-passthrough,cache.mode=passthrough \
  --disk path=/var/lib/libvirt/images/ai-infer-1.qcow2,size=200,cache=none,discard=unmap \
  --network network=default,model=virtio \
  --cdrom /iso/ubuntu-24.04.iso

Basic, scriptable monitoring loop (CSV):

#!/usr/bin/env bash
# ai-mon.sh
set -euo pipefail
OUT="${1:-ai_metrics.csv}"
echo "ts,cpu_user,cpu_sys,mem_used_gb,io_util,gpu_util,gpu_mem_gb" > "$OUT"
while :; do
  TS=$(date +%s)
  # CPU from mpstat
  CPU=($(mpstat 1 1 | awk '/all/ {print 100-$12, $5}')) # user%, sys%
  MEM_USED=$(free -g | awk '/Mem:/ {print $3}')
  IO_UTIL=$(iostat -dx 1 1 | awk '/^[sv]d/ {sum+=$NF} END{printf "%.1f", sum}')
  if command -v nvidia-smi >/dev/null; then
    GPU=($(nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader,nounits | head -n1 | tr ',' ' '))
    echo "$TS,${CPU[0]},${CPU[1]},$MEM_USED,$IO_UTIL,${GPU[0]},${GPU[1]}" >> "$OUT"
  else
    echo "$TS,${CPU[0]},${CPU[1]},$MEM_USED,$IO_UTIL,NA,NA" >> "$OUT"
  fi
  sleep 5
done

Guardrails to implement:

  • Admission control: refuse batch sizes that exceed VRAM by checking ahead with estimator + dry‑run allocation.

  • Autoscaling policy: trigger scale‑out if p95 latency or GPU memory usage > 90% for sustained windows.

  • Quotas: cap max parallel jobs per VM to avoid noisy neighbors.

Installation reference (tools used above)

  • sysstat (mpstat/iostat), numactl, glances, iotop, fio, iperf3, stress-ng, jq, nvtop:

    • apt:
    sudo apt update
sudo apt install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop
    • dnf:
    # On RHEL-like, you may need: sudo dnf install -y epel-release
sudo dnf install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop
    • zypper:
    sudo zypper refresh
sudo zypper install -y sysstat numactl glances iotop fio iperf3 stress-ng jq nvtop

  • KVM/libvirt/virt-install/virt-manager:

    • apt:
    sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system virtinst virt-manager
sudo systemctl enable --now libvirtd
    • dnf:
    sudo dnf install -y @virtualization virt-install virt-manager
sudo systemctl enable --now libvirtd
    • zypper:
    sudo zypper install -y qemu-kvm libvirt virt-install virt-manager
sudo systemctl enable --now libvirtd

Note: NVIDIA drivers (nvidia-smi) vary by distro/vendor repo. Ensure the proprietary driver is installed and loaded before using GPU tools.

Putting it together: a quick planning checklist

  • Define: model size, precision, batch, seq length, SLA targets

  • Estimate: run ai-vram-estimator.sh to bound VRAM and RAM

  • Baseline: run host-survey.sh and spot I/O/network bottlenecks

  • Shape: pick VM vCPU/RAM/GPU/storage/NIC from rules of thumb; decide on passthrough vs MIG

  • Validate: run stress-ng/fio/iperf3 + a short GPU matmul to confirm headroom

  • Operate: deploy via KVM/libvirt with NUMA/hugepages; start ai-mon.sh and set thresholds for autoscaling

Conclusion and next steps (CTA)

Capacity planning for AI VMs doesn’t have to be guesswork. With a few Bash scripts and the right Linux tools, you can translate model specs into reproducible VM shapes, verify them quickly, and operate them safely.

  • Clone your baseline: turn the snippets above into a repo in your org and wire them into CI.

  • Run a pilot: pick one model (e.g., a 7B LLM), generate a plan, and validate it on two VM sizes.

  • Iterate: tune batch sizes, quantization, and MIG partitions for the best $/throughput within SLA.

If you want a ready‑to‑use toolkit, reply and I’ll package these scripts with a Makefile and sample dashboards so you can drop them into your environment.

Related posts