Artificial Intelligence High Availability for Virtual Machines (on Linux)

AI that answers customers, scores transactions, or runs critical analytics can’t go down. Yet many AI stacks still rely on single VMs with a single GPU and a single point of failure. This post shows you a practical way to make your AI workloads highly available using Linux, KVM/libvirt, and a battle‑tested HA stack—so your model endpoints keep serving even when a host fails.

We’ll cover why VM-based HA for AI is valid, and then walk through 4 actionable steps you can use today, including exact installation commands for apt, dnf, and zypper.

---

Why HA for AI VMs is worth it

• Uptime meets AI SLAs: Your models aren’t useful if they’re not reachable. HA reduces downtime during host crashes, upgrades, and kernel/GPU driver changes.

• VMs are still the ops baseline: When you need OS isolation, stable drivers, or strict change control, VMs fit better than containers—and VM HA is a solved problem on Linux.

• GPU realities: Live-migrating a VM with a passed-through GPU is still limited in the real world. HA at the VM and service layer means you can fail over (or fail across) with minimal customer impact.

• On‑prem/private cloud: If your AI stack runs in regulated or data-resident environments, Linux + libvirt + Pacemaker gives you cloud-like resilience on your own metal.

---

The plan at a glance

We’ll build: 1) A KVM/libvirt base on each host 2) Shared storage and bridged networking for migratable VMs 3) A Pacemaker/Corosync cluster to manage AI VMs and handle failover 4) A floating Virtual IP and load balancer to keep endpoints stable

Optional: performance tuning (hugepages, CPU pin, NUMA) and failover testing.

Note: Replace hostnames/IPs with your own. Commands assume sudo/root.

---

1) Prepare the hosts: KVM/libvirt (and basics)

Install virtualization packages and start libvirtd on each host.

• Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager guestfs-tools qemu-utils
sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt $(whoami)
newgrp libvirt

• RHEL/CentOS/AlmaLinux/Rocky/Fedora (dnf):

sudo dnf install -y @virtualization qemu-kvm libvirt virt-install virt-manager guestfs-tools qemu-img bridge-utils
sudo systemctl enable --now libvirtd

• openSUSE/SLES (zypper):

sudo zypper install -y qemu-kvm libvirt libvirt-daemon virt-install virt-manager guestfs-tools qemu-tools bridge-utils
sudo systemctl enable --now libvirtd

Create a network bridge for VM traffic (example with NetworkManager):

nmcli connection add type bridge ifname br0 con-name br0
nmcli connection add type bridge-slave ifname eno1 master br0
nmcli connection up br0

Tip for AI VMs:

• Prefer hosts with identical CPU flags and matching microcode to maximize live migration compatibility.

• For GPU workloads, consider designing active-active VMs across GPU-capable hosts instead of relying on live migration of GPU passthrough.

---

2) Shared storage and stable networking

To migrate or restart VMs across hosts, keep disks on shared storage and attach via the same path on each node. Two common options:

• NFS (simple, widely supported)

• iSCSI (block-level, good performance)

Install client tools:

• Ubuntu/Debian (apt):

# NFS
sudo apt install -y nfs-common
# iSCSI
sudo apt install -y open-iscsi

• RHEL/CentOS/AlmaLinux/Rocky/Fedora (dnf):

# NFS
sudo dnf install -y nfs-utils
# iSCSI
sudo dnf install -y iscsi-initiator-utils

• openSUSE/SLES (zypper):

# NFS
sudo zypper install -y nfs-client
# iSCSI
sudo zypper install -y open-iscsi

Mount an NFS export that holds VM disks:

sudo mkdir -p /vmstore
echo "nfs-server.example.com:/exports/vmstore /vmstore nfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab
sudo mount -a

Create/convert VM disks on shared storage:

qemu-img create -f qcow2 /vmstore/ai1.qcow2 200G

Networking tips:

• Put VM NICs on a bridge (br0) bound to a physical NIC/bond/VLAN.

• If you run a frontend load balancer, give it a routable VIP that can float between nodes.

---

3) Build the HA cluster (Pacemaker/Corosync) to manage AI VMs

We’ll manage VMs as cluster resources so they start elsewhere if a host fails. Enable fencing (STONITH) to avoid split‑brain.

Install cluster packages:

• Ubuntu/Debian (apt):

sudo apt install -y pacemaker corosync pcs fence-agents resource-agents

• RHEL/CentOS/AlmaLinux/Rocky/Fedora (dnf):

sudo dnf install -y pacemaker corosync pcs fence-agents-all resource-agents

• openSUSE/SLES (zypper):

sudo zypper install -y pacemaker corosync crmsh fence-agents resource-agents

A) Fedora/RHEL/Debian/Ubuntu workflow using pcs:

# On all nodes
sudo systemctl enable --now pcsd
echo "Set a password for the 'hacluster' user"
sudo passwd hacluster

# From one node, authenticate and create the cluster
sudo pcs host auth node1 node2 node3 -u hacluster -p 'YourStrongPW'
sudo pcs cluster setup --name ai-ha node1 node2 node3
sudo pcs cluster start --all
sudo pcs property set stonith-enabled=true

# Example fencing (IPMI). Replace with your details.
sudo pcs stonith create ipmi-fence fence_ipmilan ipaddr=192.0.2.10 login=admin passwd='IPMIpass' lanplus=1 pcmk_host_list="node1 node2 node3"

# Create a VM resource referencing a libvirt domain XML
sudo pcs resource create ai-vm ocf:heartbeat:VirtualDomain \
  hypervisor="qemu:///system" \
  config="/etc/libvirt/qemu/ai1.xml" \
  migration_transport=ssh \
  op start timeout=180s op stop timeout=180s \
  meta allow-migrate=true target-role=Started

B) openSUSE/SLES workflow using crmsh:

# Initialize cluster (first node)
sudo crm cluster init -y

# Join from other nodes
sudo crm cluster join -c node1 -y

# Enable fencing property
sudo crm configure property stonith-enabled=true

# Example fencing (IPMI)
sudo crm configure primitive ipmi-fence stonith:fence_ipmilan \
  params ipaddr=192.0.2.10 login=admin passwd=IPMIpass lanplus=1 \
  meta target-role=Started

# Create a VM resource
sudo crm configure primitive ai-vm ocf:heartbeat:VirtualDomain \
  params hypervisor="qemu:///system" config="/etc/libvirt/qemu/ai1.xml" migration_transport=ssh \
  op start timeout=180s op stop timeout=180s \
  meta allow-migrate=true target-role=Started

Libvirt domain example highlights (place at /etc/libvirt/qemu/ai1.xml):

<domain type='kvm'>
  <name>ai1</name>
  <memory unit='MiB'>262144</memory>
  <vcpu placement='static'>24</vcpu>
  <memoryBacking><hugepages/></memoryBacking>
  <cpu mode='host-passthrough'/>
  <os><type arch='x86_64' machine='pc-q35-8.2'>hvm</type></os>
  <devices>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source file='/vmstore/ai1.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <source bridge='br0'/>
      <model type='virtio-net'/>
    </interface>
    <graphics type='vnc' port='-1' autoport='yes'/>
  </devices>
</domain>

Reload and define the VM on all nodes (same XML path):

sudo virsh define /etc/libvirt/qemu/ai1.xml

Live migration test (optional, requires shared storage and SSH trust between libvirt on nodes):

virsh migrate --live ai1 qemu+ssh://node2/system

---

4) Keep your endpoint stable: VIP + Load Balancer

Run at least two small “LB VMs” (which can be clustered too) with Keepalived to float a VIP, and HAProxy to route to your AI VMs. This way, client traffic never changes IP.

Install keepalived and haproxy:

• Ubuntu/Debian (apt):

sudo apt install -y keepalived haproxy

• RHEL/CentOS/AlmaLinux/Rocky/Fedora (dnf):

sudo dnf install -y keepalived haproxy

• openSUSE/SLES (zypper):

sudo zypper install -y keepalived haproxy

Keepalived example (/etc/keepalived/keepalived.conf) on both LB nodes:

vrrp_script chk_haproxy {
  script "pidof haproxy"
  interval 2
  weight 10
}

vrrp_instance VI_1 {
  interface eth0
  virtual_router_id 51
  priority 150     # on the primary, lower (e.g., 100) on the secondary
  advert_int 1
  state MASTER     # BACKUP on the secondary
  virtual_ipaddress {
    198.51.100.50/24 dev eth0
  }
  track_script {
    chk_haproxy
  }
}

sudo systemctl enable --now keepalived

HAProxy example (/etc/haproxy/haproxy.cfg):

global
  log /dev/log local0
defaults
  log global
  mode http
  timeout connect 5s
  timeout client  30s
  timeout server  30s

frontend ai_in
  bind :80
  default_backend ai_back

backend ai_back
  option httpchk GET /healthz
  http-check expect status 200
  balance roundrobin
  server ai1 10.0.0.11:8000 check
  server ai2 10.0.0.12:8000 check

sudo systemctl enable --now haproxy

Point clients to the VIP (198.51.100.50). As long as at least one LB is healthy, traffic flows to healthy AI VMs.

Real-world note:

• For GPU-bound inference, run two or more AI VMs on separate GPU hosts and let HAProxy steer around failures.

• For stateful sessions, store session/model state in a shared datastore (Redis, object storage, or a replicated DB) so any VM can serve requests.

---

5) Operate and test: health checks, failover, and tuning

• Health checks: Your AI service should expose a fast /healthz that reflects GPU/model readiness, not just process up.

• Failover drill (pcs):

sudo pcs status
sudo pcs resource move ai-vm node2
sudo pcs resource clear ai-vm

• Fail a host on purpose (maintenance):

sudo pcs node standby node1
# ... do maintenance ...
sudo pcs node unstandby node1

• Performance tuning ideas for AI VMs:

  • Pin vCPUs and use hugepages for predictable latency:

  # Host: reserve hugepages (example: 16GB total)
  echo 8192 | sudo tee /proc/sys/vm/nr_hugepages
  

  Ensure <memoryBacking><hugepages/></memoryBacking> in the VM XML.

  • NUMA pinning for memory locality (example snippet in XML):

  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  

  • Isolate host CPUs used by the VM with kernel args (isolcpus, nohz_full) and systemd CPUAffinity for libvirtd/qemu if needed.

• Backups:

  • Keep VM XML and cloud-init seeds in git.
  • Snapshot qcow2 regularly or back up block volumes at the storage layer.

Security and correctness:

• Always enable fencing (STONITH). Without it, you risk two nodes thinking they own the same VM/disk.

• Align CPU models across hosts; use host-passthrough carefully across different generations.

---

Example: resilient AI inference for recommendations

• 3 KVM hosts: node1/node2 with GPUs, node3 CPU-only for non-GPU fallback

• Shared NFS at /vmstore for disks and model artifacts

• Pacemaker manages ai1 (GPU) and ai2 (GPU) across node1/node2; ai3 (CPU) on node3 as a degraded fallback

• 2 small LB VMs (keepalived + haproxy) present VIP 198.51.100.50

• HAProxy routes to ai1/ai2 and only uses ai3 if both GPUs fail health checks

Outcome: host failure or GPU driver upgrade reboots do not take the endpoint down.

---

Conclusion and next steps

High availability for AI on VMs is both practical and powerful: shared storage, a proven Linux HA stack, and a floating VIP keep your models online through failures and maintenance. Start small—three hosts, one shared datastore, and a pair of LBs—and iterate.

Try this next:

• Build a 3-node lab and complete steps 1–4.

• Add real health checks to your AI service and practice failovers.

• Tune with hugepages/NUMA and measure latency.

• Consider storage upgrades (iSCSI or Ceph RBD) and GPU-aware scheduling across multiple AI VMs.

If you’d like a follow-up deep dive on GPU passthrough, mediated devices (mdev/MIG), or live migration caveats for AI, let me know.