Artificial Intelligence DevOps Case Studies: A Bash-First Playbook for Linux

If your model works on your laptop but pages you at 2 a.m. in production, you don’t have a model problem—you have a DevOps problem. In AI, shipping value requires repeatable pipelines, portable runtimes, scalable inference, and observable systems. This article walks through practical, Bash-friendly case studies that show how real teams moved from flaky experiments to reliable AI services on Linux.

What you’ll get:

• Why AI DevOps is a must-have, not a nice-to-have

• 3 real-world case studies with command-line steps

• Install commands for apt, dnf, and zypper

• Copy-paste code snippets to start today

Why AI DevOps matters on Linux

• Reproducibility or bust: Without data and environment versioning, you can’t debug or trust results.

• Shipping beats perfect: Containers and CI/CD make “works here” also work everywhere else.

• Scale or stall: A nightly batch job might be fine until a customer signs a bigger contract.

• Visibility saves sleep: Metrics and logs turn “what happened?” into “we saw it coming.”

Below are three case studies you can reproduce with Bash on Ubuntu/Debian (apt), Fedora/RHEL (dnf), and openSUSE (zypper).

---

Case Study 1: Reproducible training with DVC + MLflow (goodbye “works on my machine”)

Goal: Version your data and experiments so you can rewind, compare, and re-run exactly.

1) Install prerequisites

# Ubuntu/Debian (apt)
sudo apt update
sudo apt install -y git git-lfs python3 python3-venv python3-pip make curl jq

# Fedora/RHEL (dnf)
sudo dnf -y install git git-lfs python3 python3-pip make curl jq

# openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y git git-lfs python3 python3-pip make curl jq

2) Initialize a project with DVC and MLflow

git init ai-repro && cd ai-repro
git lfs install

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install dvc mlflow scikit-learn pandas joblib
dvc init

3) Track data, define a training step, and log metrics

mkdir -p raw
curl -L -o raw/iris.csv https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv
dvc add raw/iris.csv
git add .gitignore raw/iris.csv.dvc .dvc
git commit -m "Track raw data with DVC"

cat > train.py <<'PY'
import os, json, mlflow, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
import joblib

mlflow.set_tracking_uri("mlruns")
mlflow.set_experiment("iris")

df = pd.read_csv("raw/iris.csv")
X = df.drop(columns=["species"])
y = df["species"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=int(os.getenv("SEED","42")))

with mlflow.start_run():
    n_estimators=int(os.getenv("N_ESTIMATORS","100"))
    clf=RandomForestClassifier(n_estimators=n_estimators, random_state=0)
    clf.fit(Xtr, ytr)
    preds=clf.predict(Xte)
    f1=f1_score(yte, preds, average="macro")
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_metric("f1_macro", f1)
    os.makedirs("model", exist_ok=True)
    joblib.dump(clf, "model/model.pkl")
    mlflow.log_artifact("model/model.pkl")
    print(json.dumps({"f1_macro":f1}))
PY

# Add a DVC stage to make the pipeline reproducible
dvc stage add -n train -d raw/iris.csv -d train.py -o model \
  -p N_ESTIMATORS,SEED ". .venv/bin/activate && python train.py"

git add train.py dvc.yaml dvc.lock .gitignore
git commit -m "Baseline training pipeline with MLflow + DVC"

4) Run, compare, and view the UI

# Baseline run
N_ESTIMATORS=100 SEED=42 dvc repro

# Tweak a hyperparam and re-run
N_ESTIMATORS=200 dvc repro

# Launch MLflow UI
. .venv/bin/activate
mlflow ui --backend-store-uri mlruns -h 0.0.0.0 -p 5000
# Visit http://localhost:5000 to compare runs

What teams learn:

• Git tracks code; DVC tracks data; MLflow tracks runs and metrics.

• You get a time machine for your experiments and a clean audit trail.

---

Case Study 2: Containerized model serving with Podman/Docker (+ optional GPU)

Goal: Package and serve a model behind a simple REST API, with optional NVIDIA GPU acceleration.

1) Install a container runtime

Podman (rootless by default; great for dev and prod):

# Ubuntu/Debian (apt)
sudo apt update
sudo apt install -y podman

# Fedora/RHEL (dnf)
sudo dnf -y install podman

# openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y podman

Docker (if you already use it):

# Ubuntu/Debian (apt)
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

# Fedora/RHEL (dnf) - Moby Engine
sudo dnf -y install moby-engine
sudo systemctl enable --now docker

# openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y docker
sudo systemctl enable --now docker

Optional: NVIDIA Container Toolkit for GPU access

Ubuntu/Debian (apt):

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=auto
sudo systemctl restart docker || true
sudo systemctl restart podman || true

Fedora/RHEL (dnf):

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /etc/pki/rpm-gpg/NVIDIA-Container-Toolkit-Keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=auto
sudo systemctl restart docker || true
sudo systemctl restart podman || true

openSUSE (zypper):

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /etc/pki/rpm-gpg/NVIDIA-Container-Toolkit-Keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=auto
sudo systemctl restart docker || true
sudo systemctl restart podman || true

2) Write a minimal FastAPI server and containerize it

mkdir -p ai-serve && cd ai-serve

# Example model artifact
python3 - <<'PY'
import joblib
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
X, y = load_iris(return_X_y=True)
clf=RandomForestClassifier().fit(X,y)
import os; os.makedirs("model", exist_ok=True)
joblib.dump(clf, "model/model.pkl")
print("Wrote model/model.pkl")
PY

cat > app.py <<'PY'
from fastapi import FastAPI
import joblib, os
from pydantic import BaseModel
import numpy as np

app=FastAPI()
model_path=os.getenv("MODEL_PATH","/model/model.pkl")
model=joblib.load(model_path)

class Item(BaseModel):
    features: list

@app.get("/healthz")
def health():
    return {"ok":True}

@app.post("/predict")
def predict(item: Item):
    X = np.array(item.features).reshape(1,-1)
    yhat = model.predict(X)[0]
    return {"prediction": str(yhat)}
PY

cat > requirements.txt <<'REQ'
fastapi uvicorn joblib scikit-learn numpy
REQ

cat > Dockerfile <<'DF'
FROM python:3.11-slim
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
COPY model/ /model/
EXPOSE 8000
CMD ["uvicorn","app:app","--host","0.0.0.0","--port","8000"]
DF

Build and run (CPU):

# Podman
podman build -t ai-serve:0.1 .
podman run --rm -p 8000:8000 ai-serve:0.1

# Docker
docker build -t ai-serve:0.1 .
docker run --rm -p 8000:8000 ai-serve:0.1

Run with GPU (requires NVIDIA toolkit set up):

# Docker
docker run --rm --gpus all -p 8000:8000 ai-serve:0.1

# Podman (using OCI hooks; environment variables recommended)
podman run --rm -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -p 8000:8000 ai-serve:0.1

Smoke test:

curl -s http://localhost:8000/healthz
curl -s -X POST http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"features":[5.1,3.5,1.4,0.2]}'

Zero-downtime tip:

• Run blue and green containers on different ports (e.g., 8000 and 8001), front them with Nginx/HAProxy, then switch the upstream. Health-check /healthz before the cutover. This pattern avoids killing the old container until the new one is proven healthy.

What teams learn:

• Container images make your inference server portable and testable.

• Adding /healthz and versioned tags enables safe rollouts.

---

Case Study 3: Scalable batch inference with Ray (no Kubernetes required)

Goal: Spread batch scoring across multiple nodes with minimal orchestration.

1) Install Python, pip, tmux

# Ubuntu/Debian (apt)
sudo apt update
sudo apt install -y python3 python3-pip tmux

# Fedora/RHEL (dnf)
sudo dnf -y install python3 python3-pip tmux

# openSUSE (zypper)
sudo zypper refresh
sudo zypper install -y python3 python3-pip tmux

2) Create a virtualenv and install Ray

python3 -m venv ~/rayenv
. ~/rayenv/bin/activate
pip install --upgrade pip
pip install "ray[default]" pandas scikit-learn

3) Start a small Ray cluster and run jobs

On the head node:

ray start --head --port=6379

On each worker node (replace HEAD_IP):

ray start --address='HEAD_IP:6379'

Submit a simple batch job:

cat > batch_infer.py <<'PY'
import ray, time
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

ray.init(address="auto")

@ray.remote
def score(batch):
    X, y = load_iris(return_X_y=True)
    clf=RandomForestClassifier().fit(X,y)  # pretend we loaded it; demo only
    return clf.predict(batch).tolist()

if __name__ == "__main__":
    X, _ = load_iris(return_X_y=True)
    batches=[X[i:i+10] for i in range(0, len(X), 10)]
    start=time.time()
    results = ray.get([score.remote(b) for b in batches])
    print("Predictions:", sum(len(r) for r in results))
    print("Elapsed:", round(time.time()-start,3), "s")
PY

python3 batch_infer.py

Tear down:

ray stop

What teams learn:

• Horizontal scale doesn’t have to start with Kubernetes.

• Ray abstracts scheduling so you can parallelize Python without rewriting your pipeline.

---

Lessons from the trenches

• Pin the stack: Lock Python deps in requirements.txt and container base images by digest.

• Promote via artifacts: Train once, promote the exact model file through staging to prod.

• Bake health and version endpoints: Automate readiness and rollbacks.

• Keep GPUs optional: CPU-first deployments are simpler; turn on GPUs only when profiling proves the need.

• Logs and metrics or it didn’t happen: Even a simple /metrics endpoint or request logger can halve your MTTR.

Call to action

• Pick one case study above and replicate it in a scratch repo today.

• Wrap it in a Makefile so new teammates can “make up” and “make run”.

• Add CI to build the container and launch a smoke test on every commit.

• When you’re ready, extend with proper observability (Prometheus/Grafana) and CD.

Got stuck or want this as a starter template? Tell me your distro and constraints (GPU? air-gapped? rootless?), and I’ll tailor the Bash playbook.