Posted on
• Artificial Intelligence

Artificial Intelligence Hosting Checklists

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

The Linux Admin’s AI Hosting Checklist: From Bare Metal to Inference in 30 Minutes

If your model runs perfectly on a developer laptop but chokes in production, your problem is probably the host—not the code. AI workloads hammer GPUs, saturate memory bandwidth, and expose security gaps you won’t notice until 3 a.m. This checklist turns “it works on my machine” into “it works on any Linux box,” with repeatable steps, battle-tested defaults, and copy-paste Bash you can hand to ops.

What you’ll get:

  • Why hosting AI on Linux needs its own checklist

  • 3–5 actionable, ops-friendly checklists with commands

  • A real-world, 30-minute quick start: run an LLM API container and verify performance

  • Install instructions for apt, dnf, and zypper where cited

Why an AI hosting checklist is worth your time

  • Performance is nonlinear: one missing kernel toggle or NUMA mismatch can cost 20–40% throughput on inference.

  • GPU drivers, containers, and Python stacks are a dependency tangle; checklists cut time-to-first-token dramatically.

  • Security by default prevents data leaks and supply-chain surprises (and keeps auditors away).

  • Observability and capacity planning stop “mystery slowness” and monthly-bill nightmares.

1) Baseline Hardware and Kernel Readiness

Confirm the box can actually do the job before installing anything.

  • CPU vector extensions and cores
lscpu | egrep -i 'model name|socket|core|thread|avx|sse'
numactl --hardware
  • Memory and huge pages (optional but helpful for some inference engines)
free -h
grep -i huge /proc/meminfo
# Enable at runtime and persist:
sudo sysctl -w vm.nr_hugepages=1024
echo 'vm.nr_hugepages=1024' | sudo tee /etc/sysctl.d/99-hugepages.conf
  • Storage and I/O (prefer NVMe for models/checkpoints; verify TRIM)
lsblk -o NAME,MODEL,SIZE,TYPE,MOUNTPOINT
sudo fstrim -av
  • GPU presence (NVIDIA example)
nvidia-smi || echo "NVIDIA driver not found"
lspci | egrep -i 'nvidia|amd|infiniband'

Note: Use the vendor’s official documentation to install supported GPU drivers and toolkits for your distro and kernel. Don’t rely on random PPAs or outdated guides.

2) OS Packages and Runtimes (apt, dnf, zypper)

Install a minimal, reproducible toolchain for AI hosting.

  • Base tools, Python, performance, and monitoring CLI

APT (Debian/Ubuntu)

sudo apt update
sudo apt install -y git curl wget python3 python3-venv python3-pip build-essential \
  htop btop nvtop numactl tuned firewalld
sudo systemctl enable --now firewalld
sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance

DNF (Fedora/RHEL-like)

sudo dnf -y install git curl wget python3 python3-pip @development-tools \
  htop btop nvtop numactl tuned firewalld
sudo systemctl enable --now firewalld
sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance

Zypper (openSUSE/SLE)

sudo zypper refresh
sudo zypper install -y git curl wget python3 python3-venv python3-pip -t pattern devel_basis \
  htop btop nvtop numactl tuned firewalld
sudo systemctl enable --now firewalld
sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance
  • Container runtime (choose one; Podman is daemonless and distro-native)

APT

# Podman
sudo apt install -y podman
# Docker (from distro)
sudo apt install -y docker.io
sudo systemctl enable --now docker

DNF

# Podman
sudo dnf -y install podman
# Docker (Moby engine in Fedora)
sudo dnf -y install moby-engine docker-compose-plugin
sudo systemctl enable --now docker

Zypper

# Podman
sudo zypper install -y podman
# Docker
sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
  • NFS client (if you mount shared datasets/models)

APT

sudo apt install -y nfs-common

DNF

sudo dnf -y install nfs-utils

Zypper

sudo zypper install -y nfs-client
  • Quick checks
podman run --rm quay.io/podman/hello
# or
docker run --rm hello-world

3) Secure-by-Default Posture

Least privilege and simple guardrails beat postmortems.

  • Create a service account and data dirs
sudo useradd -r -m -s /usr/sbin/nologin ai
sudo mkdir -p /srv/ai/{models,data,logs,run}
sudo chown -R ai:ai /srv/ai
  • Open only what you need (example: allow HTTPS)
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-all
  • Keep MAC on (don’t disable SELinux/AppArmor)
# SELinux (Fedora/RHEL)
sestatus
# AppArmor (Debian/Ubuntu)
aa-status || sudo aa-status
  • Patch, reboot policy, and SSH hygiene
# Patch
sudo unattended-upgrades --dry-run || true
# SSH: disable password auth
sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl reload sshd 2>/dev/null || sudo systemctl reload ssh

4) Data, Storage, and Networking Layout

Plan for throughput and recovery, not just “it runs.”

  • Filesystem and mount options (example: XFS with noatime)
# Example fstab line (edit for your device):
# UUID=xxxx  /srv/ai  xfs  defaults,noatime  0 2
  • Mount remote datasets (NFS example)
sudo mkdir -p /mnt/datasets
echo 'nfs.example.com:/export/datasets  /mnt/datasets  nfs4  rsize=1048576,wsize=1048576,hard,noatime  0 0' | sudo tee -a /etc/fstab
sudo mount -a
  • Separate model cache from OS root
sudo mkdir -p /srv/ai/models
sudo chown -R ai:ai /srv/ai/models

5) Observability, Performance, and Cost Control

Give yourself a dashboard and a lever to pull during incidents.

  • Lightweight host metrics (Prometheus node exporter via container)
podman run -d --name node-exporter --restart=always --net=host \
  --pid=host --ipc=host \
  quay.io/prometheus/node-exporter:latest

# Or Docker:
docker run -d --name node-exporter --restart=always --net=host \
  --pid=host --ipc=host \
  quay.io/prometheus/node-exporter:latest
  • GPU quick watch (NVIDIA)
watch -n 2 nvidia-smi
  • Journal size and log retention
sudo mkdir -p /etc/systemd/journald.conf.d
echo -e "[Journal]\nSystemMaxUse=1G\nMaxFileSec=1month" | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald
  • Simple backup of critical state
sudo rsync -aHAX --delete /srv/ai/ /mnt/backup/ai/

Real-World Quick Start: Host a Local LLM API with Ollama

Ollama provides an easy, self-hosted API for LLMs. We’ll run it in a container, persist data, and open a port. Works CPU-only; if you have GPUs configured properly, it can use them.

  • Make a persistent volume and pick a runtime
sudo mkdir -p /srv/ai/ollama
sudo chown -R 1000:1000 /srv/ai/ollama  # default container user
  • Podman
podman run -d --name ollama --restart=always \
  -p 11434:11434 \
  -v /srv/ai/ollama:/root/.ollama \
  ollama/ollama:latest
  • Docker
docker run -d --name ollama --restart=always \
  -p 11434:11434 \
  -v /srv/ai/ollama:/root/.ollama \
  ollama/ollama:latest
  • Open the firewall port (if needed)
sudo firewall-cmd --add-port=11434/tcp --permanent
sudo firewall-cmd --reload
  • Pull a small model and test
# Pull and start a model (inside the container)
podman exec -it ollama ollama run llama3:8b-instruct
# or
docker exec -it ollama ollama run llama3:8b-instruct
  • From another terminal, send a prompt
curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3:8b-instruct",
  "prompt": "Explain shell pipelines in one paragraph."
}' | jq -r '.response' | sed ':a;N;$!ba;s/\n/ /g'
  • Optional: create a systemd service for Podman-managed container
podman generate systemd --new --name ollama | sudo tee /etc/systemd/system/ollama.service
sudo systemctl daemon-reload
sudo systemctl enable --now ollama

Tip: For production, place Ollama behind a reverse proxy (nginx, Caddy, or Traefik) with TLS and auth. Keep tokens and secrets out of the image—pass them as env vars or files mounted at runtime.

Common Pitfalls You Just Avoided

  • Running models from $HOME on a tiny ext4 root and wondering why performance tanks

  • Disabling SELinux/AppArmor “just for now” and forgetting

  • Using a random CUDA/ROCm guide that doesn’t match your kernel

  • Shipping a container with mutable state and no volume mapping

  • Debugging “slow GPU” when the real issue is wrong NUMA or no huge pages

Conclusion and Next Steps (CTA)

You now have a practical AI hosting checklist for Linux: verify hardware, install clean runtimes, lock down the surface area, mount fast storage, and observe everything. Your next step:

  • Turn this checklist into an Ansible role or Bash script for repeatable provisioning.

  • Add GPU/vendor specifics from official docs and bake them into an image.

  • Stand up a minimal monitoring stack (Prometheus + Grafana) and baseline performance.

When you’re ready, iterate: pin versions, capture performance before/after each change, and promote the host config to your “golden” image. Reliable AI hosting is a process—this checklist is your starting line.

Related posts