Beginner’s Guide to Artificial Intelligence Linux Administration

AI workloads are exploding, and so are the headaches when “it works on my laptop” meets production Linux. Drivers, Python environments, containers, GPU memory, long-running jobs—your job is to make all of this reliable, repeatable, and secure. This guide gives you a practical, distro-agnostic path to stand up AI-ready Linux systems with minimal drama.

What you’ll get:

• Why AI administration is different (and worth doing right)

• A baseline setup that works across Ubuntu/Debian, Fedora/RHEL, and openSUSE

• 3–5 actionable steps with commands for apt, dnf, and zypper

• Optional GPU enablement and a simple way to serve models as services

Note: Use sudo with package commands. Replace versions to match your distro where appropriate.

Why this matters

AI changes the operational game because:

• Toolchains are heavy and finicky: GPU drivers, CUDA/ROCm, BLAS/MKL, compilers.

• Reproducibility is non-negotiable: experiments must be repeatable across machines.

• Contention and cost matter: one runaway job can starve everyone else.

• Services, not scripts: models become APIs that must start on boot and be monitored.

• Security is critical: exposed model endpoints can leak data or become attack vectors.

Doing the basics right saves days of troubleshooting and keeps your team moving.

1) Baseline system prep for AI workloads

Update your system, install dev tools, Python 3 + venv, Git, and basic monitors. Also add OpenBLAS for faster CPU math.

Ubuntu/Debian (apt):

sudo apt update && sudo apt -y upgrade
sudo apt -y install build-essential python3 python3-venv python3-pip git curl wget \
                    htop nvtop libopenblas-dev

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf -y upgrade
sudo dnf -y install gcc gcc-c++ make python3 python3-pip python3-virtualenv git curl wget \
                    htop nvtop openblas-devel

openSUSE Leap/Tumbleweed (zypper):

sudo zypper refresh && sudo zypper -y update
sudo zypper -y install gcc gcc-c++ make python3 python3-pip git curl wget \
                       htop nvtop openblas-devel

Quick sanity checks:

python3 -V
pip3 -V
git --version
htop --version

2) Reproducible Python environments (the right way)

Use venv per project, pin dependencies, and keep requirements.txt.

Create a project and venv:

mkdir -p ~/ai-project && cd ~/ai-project
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools

Install common CPU-friendly AI libraries:

pip install numpy pandas scikit-learn jupyter

PyTorch (CPU-only, reliable across hosts):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Lock your environment:

pip freeze > requirements.txt

Later, reproduce it:

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Tip: Prefer CPU builds unless you truly need GPU; they’re simpler to maintain. Move to GPU once your pipeline is stable.

3) Optional: GPU enablement (NVIDIA)

First, confirm hardware:

lspci -k | grep -A3 -E "VGA|3D"

Then install the proprietary NVIDIA driver per distro.

Ubuntu/Debian (apt):

sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot
# After reboot
nvidia-smi

Fedora/RHEL/CentOS Stream (dnf) via RPM Fusion:

sudo dnf -y install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
                    https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf -y upgrade
sudo dnf -y install akmod-nvidia xorg-x11-drv-nvidia-cuda
sudo reboot
# After reboot
nvidia-smi

openSUSE Leap/Tumbleweed (zypper) via NVIDIA repo:

• For Leap:

sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/leap/$releasever/ nvidia
sudo zypper --gpg-auto-import-keys refresh
sudo zypper -y install --from nvidia nvidia-driver-G06
sudo reboot
# After reboot
nvidia-smi

• For Tumbleweed:

sudo zypper addrepo --refresh https://download.nvidia.com/opensuse/tumbleweed/ nvidia
sudo zypper --gpg-auto-import-keys refresh
sudo zypper -y install --from nvidia nvidia-driver-G06
sudo reboot
# After reboot
nvidia-smi

Note: G06 supports most current GPUs; older cards may require a G05 package name.

If you later run GPU workloads in containers, also install the NVIDIA Container Toolkit:

• Ubuntu/Debian (apt): sudo apt -y install nvidia-container-toolkit

• Fedora/RHEL (dnf): sudo dnf -y install nvidia-container-toolkit

• openSUSE (zypper): sudo zypper -y install nvidia-container-toolkit Then configure it for your container runtime per its official docs.

4) Containers that don’t bite (Podman)

Containers isolate dependencies and make sharing environments sane. Podman works rootless and ships on all major distros.

Install Podman:

Ubuntu/Debian (apt):

sudo apt update
sudo apt -y install podman

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf -y install podman

openSUSE Leap/Tumbleweed (zypper):

sudo zypper -y install podman

Smoke test (CPU):

podman run --rm python:3.11-slim python -c "import numpy as np; print(np.__version__)"

Build and run a local app:

cat > Dockerfile <<'EOF'
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "app.py"]
EOF

podman build -t my-ai-app .
podman run --rm -p 8000:8000 my-ai-app

GPU in containers (optional, NVIDIA):

• Ensure host driver + NVIDIA Container Toolkit installed.

• Run with --hooks-dir or the runtime configured by the toolkit; consult its docs for your distro/runtime.

5) Operate, monitor, and protect your AI jobs

Monitoring

• CPU/memory/processes:

htop

• GPU:

nvidia-smi
nvtop

• Logs:

journalctl -u my-ai-service -f

Constrain runaway jobs with systemd-run and cgroups v2:

# Limit to 2 CPUs and 8 GiB RAM for this process and its children
systemd-run --scope -p CPUQuota=200% -p MemoryMax=8G \
  bash -lc 'source .venv/bin/activate && python train.py --epochs 10'

Basic firewalling

Ubuntu/Debian (ufw):

sudo apt -y install ufw
sudo ufw allow 8000/tcp
sudo ufw enable
sudo ufw status

Fedora/RHEL/CentOS Stream and openSUSE (firewalld):

sudo dnf -y install firewalld   # Fedora/RHEL
# or
sudo zypper -y install firewalld  # openSUSE

sudo systemctl enable --now firewalld
sudo firewall-cmd --add-port=8000/tcp --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

Bonus: Serve a small model as a systemd service

Minimal FastAPI app (CPU example):

mkdir -p ~/ai-service && cd ~/ai-service
python3 -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn[standard] scikit-learn joblib
cat > app.py <<'PY'
from fastapi import FastAPI
from sklearn.linear_model import LinearRegression
import numpy as np

app = FastAPI()
model = LinearRegression().fit(np.array([[0],[1],[2]]), np.array([0,1,4]))

@app.get("/predict")
def predict(x: float):
    return {"y": float(model.predict(np.array([[x]]) )[0])}
PY

# Run once to test:
uvicorn app:app --host 0.0.0.0 --port 8000

Create a systemd unit:

sudo bash -c 'cat > /etc/systemd/system/ai-service.service <<SYSTEMD
[Unit]
Description=AI Model API (FastAPI)
After=network.target

[Service]
User=%i
WorkingDirectory=/home/%i/ai-service
Environment=PATH=/home/%i/ai-service/.venv/bin
ExecStart=/home/%i/ai-service/.venv/bin/uvicorn app:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
SYSTEMD'

Enable and start (replace $USER if running for another user):

sudo systemctl daemon-reload
sudo systemctl enable ai-service.service
sudo systemctl start ai-service.service
sudo systemctl status ai-service.service

Open the firewall (see previous section), then test:

curl http://localhost:8000/predict?x=3.5

Real-world tips

• Keep a “golden” image or Ansible playbook that applies the baseline setup and installs project requirements.

• Favor CPU builds for development; switch to GPU only when needed for scale or latency.

• Use separate UNIX users or containers per team/project to reduce dependency conflicts.

• Put large datasets on a fast filesystem and monitor disk I/O; consider ionice for heavy preprocessing.

• Schedule long runs off-hours with systemd timers or your scheduler of choice (e.g., SLURM) as you mature.

Conclusion and next steps

You now have a clean, repeatable baseline for AI on Linux: solid Python environments, optional GPU support, containers for isolation, and ops basics (monitoring, cgroups, firewalling). Your next moves:

• Turn this into an automated bootstrap (Ansible).

• Add observability (node_exporter + Prometheus + Grafana).

• Standardize a container workflow per team (Podman + registries).

• Pilot one production model behind systemd and a firewall.

Bookmark this checklist, adapt it to your distro, and ship your first reliable AI node this week.