Posted on
• Artificial Intelligence

Artificial Intelligence Load Balancing on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Load Balancing on Linux: Fast, Fair and Fail‑Safe

AI apps don’t just need to be “smart.” They need to be fast, consistent, and resilient—especially at scale. When an AI model is deployed behind an API, your p50 might look fine, but the p99 tails can quietly wreck user experience and unit economics. The good news: Linux already ships with powerful building blocks you can use today—from L7 proxies to NUMA-aware placement and kernel L4 load balancers—to tame concurrency and keep latency low.

This post shows practical, Bash-first ways to balance AI inference or training microservices on Linux. You’ll get actionable configurations, with install commands for apt, dnf, and zypper where needed.

Why this matters (and works)

  • AI inference is just a service: it speaks HTTP/gRPC and competes for CPU, memory, and I/O like any other microservice.

  • Tail latency is often caused by poor placement (cross-NUMA hops), CPU contention, head-of-line blocking, and overload without backpressure.

  • Linux provides predictable, low-latency mechanisms for:

    • Front-door balancing (e.g., HAProxy)
    • Kernel-level L4 dispatch (IPVS)
    • NUMA/CPU pinning and cgroups (stable throughput)
    • Highly available VIPs (Keepalived/VRRP)

Below are 5 concrete steps you can apply today.

1) Front-door L7 load balancing with HAProxy

Use HAProxy to evenly distribute incoming requests to multiple model servers (containers or processes) and add basic health checks and queueing.

Install HAProxy:

  • Debian/Ubuntu (apt):

    sudo apt update && sudo apt install -y haproxy

  • Fedora/RHEL/CentOS (dnf):

    sudo dnf install -y haproxy

  • openSUSE/SLE (zypper):

    sudo zypper install -y haproxy

Minimal /etc/haproxy/haproxy.cfg example (HTTP inference on ports 8001–8003):

global
  maxconn 20000
  log /dev/log local0
  daemon

defaults
  mode http
  timeout connect 2s
  timeout client  60s
  timeout server  60s
  option httplog
  option dontlognull

frontend ai_inference
  bind :8080
  default_backend ai_backends

backend ai_backends
  balance leastconn
  option httpchk GET /health
  http-response add-header X-Backend %[srv_name]
  server m1 127.0.0.1:8001 check maxconn 500
  server m2 127.0.0.1:8002 check maxconn 500
  server m3 127.0.0.1:8003 check maxconn 500

Enable and test:

sudo systemctl enable --now haproxy
curl -i http://127.0.0.1:8080/v1/predict
curl -I http://127.0.0.1:8080/health

Tips:

  • Use leastconn for steadier p99 under variable latency.

  • Set realistic maxconn per server to avoid overload collapse; HAProxy will queue smartly.

2) CPU and NUMA-aware placement for consistent latency

AI inference often suffers when threads bounce across NUMA nodes. Keep your model workers local to a socket and cache domain.

Install numactl:

  • apt:

    sudo apt update && sudo apt install -y numactl

  • dnf:

    sudo dnf install -y numactl

  • zypper:

    sudo zypper install -y numactl

Inspect topology:

lscpu --extended
numactl --hardware

Pin model servers to specific CPUs/NUMA nodes. Example using systemd-run (works with cgroups v2, no extra packages needed). Replace your_model_server with your binary/command:

  • Pin one instance to cores 0–7 on NUMA node 0:
sudo systemd-run --unit=ai1 \
  --property=AllowedCPUs=0-7 \
  --property=AllowedMemoryNodes=0 \
  --property=CPUQuota=90% \
  --collect \
  bash -lc 'exec your_model_server --port 8001'
  • Pin another to cores 8–15 on NUMA node 0:
sudo systemd-run --unit=ai2 \
  --property=AllowedCPUs=8-15 \
  --property=AllowedMemoryNodes=0 \
  --property=CPUQuota=90% \
  --collect \
  bash -lc 'exec your_model_server --port 8002'
  • Pin third to NUMA node 1:
sudo systemd-run --unit=ai3 \
  --property=AllowedCPUs=16-23 \
  --property=AllowedMemoryNodes=1 \
  --property=CPUQuota=90% \
  --collect \
  bash -lc 'exec your_model_server --port 8003'

Alternative with numactl (process-level):

numactl --cpunodebind=0 --membind=0 taskset -c 0-7 your_model_server --port 8001

Why it helps:

  • Keeps threads and memory local, improving cache hit rates and cutting cross-socket latency.

  • CPU quotas prevent single instances from starving others under burst load.

3) Kernel L4 load balancing with IPVS (ultra-low overhead)

For TCP/UDP services (HTTP/gRPC included), Linux Virtual Server (IPVS) provides fast, in-kernel load balancing that beats user-space hops for raw throughput and low latency.

Install ipvsadm:

  • apt:

    sudo apt update && sudo apt install -y ipvsadm

  • dnf:

    sudo dnf install -y ipvsadm

  • zypper:

    sudo zypper install -y ipvsadm

Example: TCP VIP on port 9000 dispatching to two local backends (9001, 9002) with round-robin:

sudo modprobe ip_vs
sudo ipvsadm -C
sudo ipvsadm -A -t 0.0.0.0:9000 -s rr
sudo ipvsadm -a -t 0.0.0.0:9000 -r 127.0.0.1:9001 -m
sudo ipvsadm -a -t 0.0.0.0:9000 -r 127.0.0.1:9002 -m
sudo ipvsadm -Ln

Send traffic to :9000 and IPVS will spread it across the two backends. Replace -m (NAT) with -g (DR) for direct routing in multi-host setups.

Cleanup:

sudo ipvsadm -C

When to use IPVS:

  • You need kernel-speed L4 load balancing (e.g., many concurrent gRPC streams).

  • You already handle HTTP semantics (health checks, retries) elsewhere.

Use HAProxy at the edge and IPVS internally for a hybrid approach.

4) Backpressure and observability: prevent overload before it starts

Even perfect balancing fails if you don’t cap concurrency and watch saturation.

Install basic tools:

  • htop:

    • apt: sudo apt update && sudo apt install -y htop
    • dnf: sudo dnf install -y htop
    • zypper: sudo zypper install -y htop

  • sysstat (mpstat, pidstat, iostat):

    • apt: sudo apt update && sudo apt install -y sysstat
    • dnf: sudo dnf install -y sysstat
    • zypper: sudo zypper install -y sysstat

Quick saturation checks:

mpstat -P ALL 1
pidstat -t -p $(pgrep -d, your_model_server) 1
htop

Apply backpressure:

  • Cap per-instance CPU so queues form at the load balancer, not inside the model server:
sudo systemd-run --unit=ai1 --property=CPUQuota=80% bash -lc 'exec your_model_server --port 8001'
  • Tune accept backlog (temporary, runtime only):
sudo sysctl -w net.core.somaxconn=8192
sudo sysctl -w net.core.netdev_max_backlog=250000
  • In HAProxy, cap and queue:
backend ai_backends
  balance leastconn
  queue-timeout 30s
  server m1 127.0.0.1:8001 check maxconn 500
  server m2 127.0.0.1:8002 check maxconn 500
  server m3 127.0.0.1:8003 check maxconn 500

Rule of thumb:

  • Keep CPU utilization per NUMA node below ~80% under steady load.

  • Watch run queue length and context switches; rising values signal head-of-line blocking.

5) High availability with a floating VIP (Keepalived)

Eliminate single points of failure by running two LBs (e.g., HAProxy nodes) and floating a virtual IP (VIP) between them using VRRP.

Install Keepalived:

  • apt:

    sudo apt update && sudo apt install -y keepalived

  • dnf:

    sudo dnf install -y keepalived

  • zypper:

    sudo zypper install -y keepalived

Primary node /etc/keepalived/keepalived.conf:

vrrp_script chk_haproxy {
  script "/usr/bin/pgrep haproxy"
  interval 2
  fall 2
  rise 2
}

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 200
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass changeme
  }
  virtual_ipaddress {
    10.0.0.100/24 dev eth0
  }
  track_script {
    chk_haproxy
  }
}

Backup node: same file but state BACKUP and priority 100.

Enable:

sudo systemctl enable --now keepalived
ip addr show dev eth0 | grep 10.0.0.100

Point clients to 10.0.0.100:8080. If the primary fails or HAProxy stops, the VIP moves to the backup almost instantly.

Real-world pattern you can deploy today

  • Pin 2–4 model server processes per NUMA node with systemd-run and AllowedCPUs/AllowedMemoryNodes.

  • Front with HAProxy (L7), limit maxconn per server to what your model handles, and set leastconn.

  • For east-west traffic across many pods/nodes, use IPVS to fan in/out at L4.

  • Add Keepalived to float a VIP between 2 load-balancer nodes.

  • Watch p95/p99 while gradually raising concurrency; keep a margin to absorb spikes.

Conclusion and next steps

Linux gives you the primitives you need to balance AI workloads without exotic tooling: HAProxy at the edge, IPVS in the kernel, NUMA-aware placement, and VRRP for HA. Start small, measure p99, and iterate.

Your next steps: 1) Install HAProxy, pin a few model servers to distinct CPU sets, and enable health checks. 2) Add CPU quotas and backlog tuning to stabilize tail latency. 3) Layer in IPVS and Keepalived as your traffic and availability needs grow.

Have a specific topology or latency target in mind? Share your constraints and current stack, and I’ll help you design a low-latency, HA load-balancing plan tuned for your Linux environment.

Related posts