Artificial Intelligence with Proxmox Automation: A Bash‑First Guide

Tired of babysitting AI workloads? What if your Proxmox lab could spin up GPU‑ready inference VMs on demand, then tear them down the moment they’re idle—no clicks, no guesswork, just automation. In this post, you’ll learn how to wire Proxmox VE to your AI pipeline using Bash, curl, and a pinch of Python so you can scale fast, stay reproducible, and cut costs.

Why this matters:

• AI experiments and inference often come in unpredictable bursts.

• Proxmox VE gives you enterprise‑grade virtualization, GPU passthrough, and a clean API—without SaaS lock‑in.

• With a small amount of automation, you can create a self‑service “AI lab” that boots, configures, and retires resources automatically.

What you’ll get:

• A practical, Bash‑centric playbook to automate AI VMs via the Proxmox API

• A golden cloud‑init template pipeline for fast, reproducible clones

• Token‑based auth, real examples with curl and Python (proxmoxer)

• Optional GPU passthrough guidance to run modern model servers

Note: Proxmox VE runs on Debian. The automation client (your laptop or CI runner) can be any Linux distro. We’ll install tools on that client.

---

1) Install the tools and set up API access

On your management workstation (not necessarily the Proxmox host), install curl/jq/Python/Ansible and the proxmoxer Python library. Use your distro’s package manager:

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y curl jq python3 python3-venv python3-pip ansible git
python3 -m pip install --user proxmoxer requests
ansible-galaxy collection install community.general

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y curl jq python3 python3-pip python3-virtualenv ansible git
python3 -m pip install --user proxmoxer requests
ansible-galaxy collection install community.general

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y curl jq python3 python3-pip python3-virtualenv ansible git
python3 -m pip install --user proxmoxer requests
ansible-galaxy collection install community.general

Create a Proxmox API Token:

• In Proxmox UI: Datacenter -> Permissions -> API Tokens

• Create a user with minimal privileges (e.g., PVEAuditor or a custom role that can clone/start/stop under a specific pool).

• Create a token for that user, copy the Token ID and Secret.

• Prefer pool‑scoped privileges and avoid root@pam tokens in production.

Environment variables for curl:

export PVE_HOST="pve.yourdomain.local"
export PVE_NODE="pve-node1"        # Proxmox node name
export PVE_API="https://${PVE_HOST}:8006/api2/json"
export PVE_TOKEN_ID="aiuser@pve!aitoken"   # user@realm!tokenid
export PVE_TOKEN_SECRET="REDACTEDSECRET"
# Add -k if your Proxmox uses a self-signed cert (development/lab only)
alias pcurl='curl -sS --fail -k -H "Authorization: PVEAPIToken=${PVE_TOKEN_ID}=${PVE_TOKEN_SECRET}"'

---

2) Build a golden cloud‑init template for AI

A template lets you clone VMs in seconds with cloud‑init setting users, SSH keys, and network. Run the following on the Proxmox host (via SSH). Example for Ubuntu 22.04 cloud image:

# On the Proxmox node:
cd /tmp
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

# Create a base VM (pick a free VMID, e.g., 9000)
qm create 9000 --name ai-ubuntu-2204 --memory 8192 --cores 8 --cpu x86-64-v2-AES \
  --net0 virtio,bridge=vmbr0
qm importdisk 9000 jammy-server-cloudimg-amd64.img local-lvm
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --serial0 socket --vga serial0
qm set 9000 --agent 1

Optional: inject a baseline cloud‑init user‑data (packages, Docker, qemu-guest-agent). First, prepare a snippets storage in Proxmox and drop a file like /var/lib/vz/snippets/ai-userdata.yaml:

# /var/lib/vz/snippets/ai-userdata.yaml
#cloud-config
packages:
  - qemu-guest-agent
  - docker.io
  - python3-pip
runcmd:
  - systemctl enable --now qemu-guest-agent
  - systemctl enable --now docker

Attach it:

qm set 9000 --cicustom "user=local:snippets/ai-userdata.yaml"

Finalize as a template:

qm template 9000

You now have a reusable AI‑ready VM template.

---

3) Automate VM lifecycle with pure Bash and curl

Let’s clone from the template, set cloud‑init options (SSH key, network, tags), and start the VM. This is 100% API‑driven.

#!/usr/bin/env bash
set -euo pipefail

: "${PVE_HOST:?missing}"; : "${PVE_NODE:?missing}"
: "${PVE_API:?missing}"; : "${PVE_TOKEN_ID:?missing}"; : "${PVE_TOKEN_SECRET:?missing}"

TEMPLATE_VMID=9000
NEW_NAME="ai-worker-$(date +%y%m%d-%H%M%S)"
STORAGE="local-lvm"
SSH_PUBKEY="$(cat ~/.ssh/id_ed25519.pub)"

# 1) Ask Proxmox for the next free VMID
NEWID="$(pcurl "${PVE_API}/cluster/nextid")" 
NEWID="$(echo "$NEWID" | jq -r '.data')"

# 2) Clone the template
pcurl -X POST "${PVE_API}/nodes/${PVE_NODE}/qemu/${TEMPLATE_VMID}/clone" \
  -d newid="${NEWID}" \
  -d name="${NEW_NAME}" \
  -d storage="${STORAGE}" \
  -d full=1 > /dev/null

# 3) Configure cloud-init: user, ssh key, DHCP, tags
pcurl -X POST "${PVE_API}/nodes/${PVE_NODE}/qemu/${NEWID}/config" \
  -d ciuser="ubuntu" \
  -d sshkeys="$SSH_PUBKEY" \
  -d ipconfig0="ip=dhcp" \
  -d tags="ephemeral-ai" > /dev/null

# 4) Start it
pcurl -X POST "${PVE_API}/nodes/${PVE_NODE}/qemu/${NEWID}/status/start" > /dev/null

echo "Launched ${NEW_NAME} (VMID ${NEWID})"

Cleanup stale workers by tag (destroy VMs labeled ephemeral-ai and powered off for >60 minutes):

#!/usr/bin/env bash
set -euo pipefail

VM_LIST="$(pcurl "${PVE_API}/cluster/resources?type=vm" | jq -c '.data[]')"

while read -r vm; do
  vmid=$(echo "$vm" | jq -r '.vmid')
  node=$(echo "$vm" | jq -r '.node')
  status=$(echo "$vm" | jq -r '.status')
  # Tags may not appear in this endpoint in older PVE; fallback to per-VM config:
  tags=$(pcurl "${PVE_API}/nodes/${node}/qemu/${vmid}/config" | jq -r '.data.tags // ""')

  [[ "$tags" != *"ephemeral-ai"* ]] && continue
  [[ "$status" != "stopped" ]] && continue

  # Check how long it's been stopped (using ctime as a heuristic)
  ctime=$(echo "$vm" | jq -r '.cgroup // 0') # not always present; adapt to your env/logs
  # For demo, just delete stopped ephemeral workers:
  echo "Deleting VMID ${vmid} on ${node}..."
  pcurl -X DELETE "${PVE_API}/nodes/${node}/qemu/${vmid}" > /dev/null
done <<< "$VM_LIST"

Tip:

• For production, use pools to restrict where tokens can create/delete VMs.

• Use a proper CA and drop -k.

• Prefer SSH keys over passwords via cloud‑init.

---

4) Scale workers with Python (proxmoxer)

When a job queue grows (e.g., pending inference requests), add more workers; when it shrinks, remove them. proxmoxer makes the Proxmox API ergonomic:

#!/usr/bin/env python3
import os, time
from proxmoxer import ProxmoxAPI

PVE_HOST = os.environ["PVE_HOST"]
PVE_NODE = os.environ["PVE_NODE"]
TEMPLATE_VMID = int(os.environ.get("TEMPLATE_VMID", "9000"))
TOKEN_ID = os.environ["PVE_TOKEN_ID"]
TOKEN_SECRET = os.environ["PVE_TOKEN_SECRET"]

proxmox = ProxmoxAPI(
    PVE_HOST, user=TOKEN_ID.split('!')[0], token_name=TOKEN_ID.split('!')[1],
    token_value=TOKEN_SECRET, verify_ssl=False  # set True with proper CA
)

def nextid():
    return int(proxmox.cluster.nextid.get())

def spawn_worker(name_prefix="aiw", storage="local-lvm"):
    newid = nextid()
    proxmox.nodes(PVE_NODE).qemu(TEMPLATE_VMID).clone.post(
        newid=newid, name=f"{name_prefix}-{newid}", storage=storage, full=1
    )
    proxmox.nodes(PVE_NODE).qemu(newid).config.post(
        ciuser="ubuntu", ipconfig0="ip=dhcp", tags="ephemeral-ai"
    )
    proxmox.nodes(PVE_NODE).qemu(newid).status.start.post()
    return newid

def count_ephemeral_running():
    vms = proxmox.cluster.resources.get(type='vm')
    c = 0
    for vm in vms:
        if vm.get('status') == 'running':
            cfg = proxmox.nodes(vm['node']).qemu(vm['vmid']).config.get()
            if 'tags' in cfg and 'ephemeral-ai' in cfg['tags']:
                c += 1
    return c

def scale_to(n):
    current = count_ephemeral_running()
    if current < n:
        for _ in range(n - current):
            print("Spawning worker:", spawn_worker())
            time.sleep(2)
    elif current > n:
        vms = proxmox.cluster.resources.get(type='vm')
        stopped = []
        running = []
        for vm in vms:
            cfg = proxmox.nodes(vm['node']).qemu(vm['vmid']).config.get()
            if 'tags' in cfg and 'ephemeral-ai' in cfg['tags']:
                (running if vm['status']=='running' else stopped).append(vm)
        # Prefer stopping/deleting stopped ones first
        victims = stopped + running
        for vm in victims[:(current-n)]:
            proxmox.nodes(vm['node']).qemu(vm['vmid']).status.stop.post()
            proxmox.nodes(vm['node']).qemu(vm['vmid']).delete()
            print("Removed worker:", vm['vmid'])

if __name__ == "__main__":
    desired = 4  # Example: derive from queue depth
    scale_to(desired)

Wire this script to your metrics (e.g., pending jobs in a message queue) and run it periodically via cron or a CI/CD job.

---

5) Add GPU passthrough for model inference (optional)

If your node has an NVIDIA/AMD GPU, you can pass it through to a VM and run modern inference servers (e.g., TensorRT, vLLM, text-generation-webui). High‑level steps:

Host prerequisites (Proxmox node):

• Enable IOMMU in your bootloader and load VFIO modules for your GPU.

• Blacklist the host GPU driver if necessary.

• Reboot and verify the GPU is bound to vfio‑pci.

Attach the GPU to the template or a specific VM:

# Identify GPU PCI address on the Proxmox host:
lspci | grep -E "VGA|3D|Display"
# Example GPU at 0000:65:00.0

# Add passthrough to a VM (adjust VMID and address):
qm set 1001 --hostpci0 0000:65:00,pcie=1

Inside the guest VM:

• Install vendor drivers and CUDA (for NVIDIA) or ROCm (for AMD) as appropriate.

• Deploy your inference stack (Docker often simplifies this).

Tip:

• Keep one GPU‑enabled template and one CPU‑only template to avoid driver drift.

• Start small (one GPU VM), validate performance and stability, then scale out.

---

Real‑world example: “AI lab on demand”

• Your CI/CD queue spikes with incoming inference jobs.

• A scheduled Python scaler (proxmoxer) sees queue depth > 100 and scales to 10 workers.

• Workers are clones of a golden image with Docker and model runtime preinstalled.

• After 30 minutes of low traffic, the scaler drains and deletes all ephemeral-ai instances.

• You pay nothing extra, and your lab remains clean and reproducible.

---

Conclusion and next steps

You don’t need a cloud bill or a complex orchestrator to run serious AI workloads. With Proxmox VE, a golden cloud‑init template, and a few Bash/Python scripts, you can build an elastic AI lab that’s fast, secure, and repeatable.

Your next steps:

• Set up the template and token from this guide.

• Automate a single end‑to‑end run: clone -> configure -> run inference -> destroy.

• Add GPU passthrough if your workloads need it.

• Gradually wire in auto‑scaling based on your queue metrics.

If this was helpful, turn it into a repo for your team (scripts + docs), or add Ansible roles to make your AI lab one command away.