Artificial Intelligence Linux Standards: A Practical, Bash‑First Guide

AI models run great on Linux—until they don’t. One team’s “works on my machine” model becomes another team’s driver mismatch, missing library, or runaway process problem. The answer isn’t magic; it’s standards. A small set of Linux-native conventions can turn ad‑hoc AI deployments into repeatable, observable, and secure systems.

This post lays out a practical set of “AI Linux Standards” you can adopt today—grounded in tools you already know: containers, systemd, cgroups, FHS paths, and neutral model formats. You’ll get the why, the how, and copy‑paste commands for apt, dnf, and zypper.

Why standardize AI on Linux?

• Portability beats snowflakes: Neutral model formats (like ONNX) and OCI containers outlive CUDA/ROCm/Python drift.

• Operability matters: Systemd + cgroups v2 provide tame-by-default resource limits and predictable restarts.

• Security by convention: Non-root, least privilege, and standard filesystem paths keep surprises to a minimum.

• Reproducibility wins trust: Pinned dependencies and immutable container digests debunk “it changed itself.”

Below are five actionable standards to adopt across your AI projects.

---

1) Standardize the runtime with OCI containers (Docker/Podman)

If every AI job runs in an OCI image, the base OS, drivers, and Python stack stop being “works on my laptop” risks. Favor rootless Podman when possible; Docker is fine too.

Install containers (pick Podman or Docker):

• apt (Debian/Ubuntu):

sudo apt update
# Podman
sudo apt install -y podman
# Docker
sudo apt install -y docker.io
sudo systemctl enable --now docker

• dnf (Fedora/RHEL family):

sudo dnf -y upgrade
# Podman
sudo dnf install -y podman
# Docker (Moby Engine in Fedora)
sudo dnf install -y moby-engine
sudo systemctl enable --now docker

• zypper (openSUSE/SLES):

sudo zypper refresh
# Podman
sudo zypper install -y podman
# Docker
sudo zypper install -y docker
sudo systemctl enable --now docker

A minimal, portable inference image:

# Dockerfile
FROM python:3.11-slim

# Create non-root user
RUN useradd -m -u 10001 appuser
USER appuser

WORKDIR /opt/ml/app
ENV PATH="/opt/ml/app/.venv/bin:${PATH}"

# Dependencies (CPU-only inference)
RUN python -m venv .venv && \
    . .venv/bin/activate && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir onnxruntime numpy

# Copy model and script
COPY models/squeezenet.onnx /opt/ml/models/squeezenet.onnx
COPY inference.py /opt/ml/app/inference.py

# Read-only model dir is a good practice
VOLUME ["/opt/ml/models:ro"]

CMD ["python", "inference.py", "--model", "/opt/ml/models/squeezenet.onnx"]

Run it:

• Podman (rootless):

podman build -t onnx-cpu .
podman run --rm --cpus=2 --memory=2g -v $PWD/models:/opt/ml/models:ro localhost/onnx-cpu

• Docker:

docker build -t onnx-cpu .
docker run --rm --cpus=2 --memory=2g -v $PWD/models:/opt/ml/models:ro onnx-cpu

Key standards:

• Always run as non-root in the container.

• Pass models/configs via volumes at runtime.

• Constrain CPU/memory for predictability.

---

2) Use ONNX as your neutral model exchange format

ONNX is the de‑facto open format for moving models between frameworks and hardware. Train in PyTorch or TensorFlow, export to ONNX, and run anywhere with ONNX Runtime.

Prepare Python and tools:

• apt:

sudo apt update
sudo apt install -y python3 python3-venv python3-pip

• dnf:

sudo dnf install -y python3 python3-pip

• zypper:

sudo zypper install -y python3 python3-pip

Optional developer tools (handy for native builds):

• apt:

sudo apt install -y build-essential cmake ninja-build git curl

• dnf:

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y cmake ninja-build git curl

• zypper:

sudo zypper install -y cmake ninja git curl
sudo zypper install -t pattern -y devel_basis

Create a venv and test ONNX Runtime (CPU):

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install onnxruntime numpy

Download a sample model and run it:

mkdir -p models
wget -O models/squeezenet.onnx \
  https://github.com/onnx/models/raw/main/vision/classification/squeezenet/model/squeezenet1.0-12.onnx

python - <<'PY'
import onnxruntime as ort, numpy as np, os
model = "models/squeezenet.onnx"
sess = ort.InferenceSession(model, providers=["CPUExecutionProvider"])
x = np.random.rand(1,3,224,224).astype(np.float32)
y = sess.run(None, {"data": x})
print("Output shapes:", [a.shape for a in y])
PY

Optional: export from PyTorch to ONNX (requires extra deps):

. .venv/bin/activate
pip install torch torchvision onnx
python - <<'PY'
import torch, torchvision
model = torchvision.models.resnet18(weights=None).eval()
x = torch.randn(1,3,224,224)
torch.onnx.export(model, x, "models/resnet18.onnx",
                  input_names=["input"], output_names=["logits"],
                  opset_version=17, dynamic_axes=None)
print("Exported to models/resnet18.onnx")
PY

Key standards:

• Keep your “source of truth” in ONNX alongside training artifacts.

• Use ONNX Runtime for baseline CPU inference; add GPU later if needed.

• Store models under a standard path (see next section).

---

3) Follow the Filesystem Hierarchy Standard (FHS) for AI assets

Stop scattering models, logs, and configs. Use conventional Linux locations so SREs know where to look.

Recommended layout:

• Binaries/app code: /opt/ml/app

• Models: /opt/ml/models (mounted read‑only in prod)

• Config: /etc/ml/<app>.conf

• State/data: /var/lib/ml/<app>

• Logs: /var/log/ml/<app>.log

Create a dedicated user and directories:

sudo useradd --system --create-home --home-dir /srv/ml --shell /usr/sbin/nologin mlsvc || true
sudo install -d -o mlsvc -g mlsvc /opt/ml/{app,models} /var/lib/ml/myservice /var/log/ml
sudo touch /etc/ml/myservice.conf
sudo chown mlsvc:mlsvc /etc/ml/myservice.conf /var/lib/ml/myservice /var/log/ml

A minimal systemd unit for an inference script:

# /etc/systemd/system/ml-infer.service
[Unit]
Description=ONNX Inference Service
After=network-online.target
Wants=network-online.target

[Service]
User=mlsvc
WorkingDirectory=/opt/ml/app
Environment=MODEL_PATH=/opt/ml/models/squeezenet.onnx
ExecStart=/opt/ml/app/.venv/bin/python /opt/ml/app/inference.py --model ${MODEL_PATH}
Restart=on-failure
# Resource controls (see next section)
MemoryMax=2G
CPUQuota=200%

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now ml-infer.service
sudo journalctl -u ml-infer.service -f

Key standards:

• One service user per app.

• Logs go to journal and /var/log/ml.

• Models and configs are not co‑located with code.

---

4) Constrain and schedule with cgroups v2 + systemd

AI jobs can be resource hogs. Use cgroups v2 controls consistently—whether you run bare-metal Python, systemd services, or containers.

Quick checks:

mount | grep cgroup2 || true
ls /sys/fs/cgroup

Ad‑hoc constrained run:

sudo systemd-run --unit=onnx-bench -p MemoryMax=2G -p CPUQuota=150% \
  /usr/bin/python3 /opt/ml/app/inference.py --model /opt/ml/models/squeezenet.onnx

Service-level constraints (in the unit file):

MemoryMax=4G
CPUQuota=300%
IOReadBandwidthMax=/dev/nvme0n1 200M

Container constraints:

• Podman:

podman run --rm --cpus=2 --memory=4g -v $PWD/models:/opt/ml/models:ro localhost/onnx-cpu

• Docker:

docker run --rm --cpus=2 --memory=4g -v $PWD/models:/opt/ml/models:ro onnx-cpu

Key standards:

• Every job has explicit CPU/memory ceilings.

• Prefer systemd units for long‑lived services; containers for immutable packaging.

---

5) Reproducibility and supply chain hygiene

Pin what you run, or it will drift.

Python locking with pip‑tools:

. .venv/bin/activate
pip install --upgrade pip pip-tools
echo 'onnxruntime==1.*' > requirements.in
pip-compile requirements.in
pip-sync requirements.txt

Pin container images by digest:

# Get digest
docker pull python:3.11-slim
docker inspect --format='{{index .RepoDigests 0}}' python:3.11-slim
# Use image@sha256:<digest> in Dockerfile FROM or Kubernetes manifests

Record build metadata:

git rev-parse --short HEAD > BUILD_INFO
date -u +"%Y-%m-%dT%H:%M:%SZ" >> BUILD_INFO

Key standards:

• Lock Python deps with a solver (pip‑tools/poetry/uv).

• Reference images by digest, not moving tags.

• Embed build provenance for traceability.

---

Real‑world example: a tiny CPU inference script

inference.py:

import argparse, onnxruntime as ort, numpy as np, time

p = argparse.ArgumentParser()
p.add_argument("--model", required=True)
args = p.parse_args()

sess = ort.InferenceSession(args.model, providers=["CPUExecutionProvider"])
x = np.random.rand(1,3,224,224).astype(np.float32)

for i in range(5):
    t0 = time.time()
    y = sess.run(None, {"data": x})
    dt = (time.time() - t0) * 1000
    print(f"inference {i}: {dt:.2f} ms; outputs {[a.shape for a in y]}")

Run in venv:

. .venv/bin/activate
python inference.py --model models/squeezenet.onnx

Run in a constrained container:

podman run --rm --cpus=2 --memory=2g -v $PWD/models:/opt/ml/models:ro localhost/onnx-cpu

---

Conclusion and Call to Action

Linux already gives you the building blocks of “AI standards.” Put them to work:

1) Package everything in OCI images (non-root).
2) Export models to ONNX and keep a CPU‑first runtime path.
3) Use FHS paths so ops isn’t a scavenger hunt.
4) Apply cgroups v2 limits everywhere.
5) Pin dependencies and image digests.

Your next step:

• Pick one service and implement all five standards this week.

• Commit a base container, a sample systemd unit, and an internal README that codifies these rules.

• Automate checks in CI (lint for user=root in Dockerfile, scan for unpinned apt/pip installs, etc.).

Small, consistent standards beat heroic debugging every time.