Learning AI DevOps (MLOps) the Linux Bash Way

If you can ship a web service, you can ship a model. The catch? Models drift, data changes, and a “works-on-my-machine” training run is worthless in production. This post shows how to learn Artificial Intelligence DevOps (aka MLOps) using the tools you already trust on Linux: Bash, Git, containers, and a few focused CLI utilities.

We’ll set up a minimal but realistic workflow you can reproduce end-to-end from a shell:

• Reproducible environments

• Data and experiment versioning

• Containerized training and serving

• CI/CD and basic observability

By the end, you’ll have scripts and files you can drop into any repo to level up your AI delivery pipeline.

---

Why AI DevOps is worth your time

• AI systems break differently. Bugs hide in data, not just code. You need versioned datasets, tracked experiments, and reproducible builds.

• Speed comes from automation. Consistent environments, container images, and CI runners save hours per week.

• Reliability is observability. Metrics, health checks, and rollbacks make your model services boring (in a good way).

• Governance isn’t optional. Audit trails for data and experiments are a must for teams and regulated environments.

---

Prerequisites: install core tooling

Pick Docker or Podman for containers. Run the matching block for your distro. The rest (Python, Git, Make, etc.) is common to most Linux workflows.

Note:

• Use sudo where needed.

• After installing Docker, log out/in or run newgrp docker.

Ubuntu/Debian (apt)

Update and install core CLI tools:

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip python3-dev \
  git git-lfs make jq curl \
  build-essential
git lfs install

Choose ONE container stack:

Docker:

sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Re-login or: newgrp docker

Podman:

sudo apt install -y podman podman-compose
# Optional (rootless needs no daemon); for socket-compat:
systemctl --user enable --now podman.socket

Fedora/RHEL/CentOS Stream (dnf)

Core tools:

sudo dnf -y update
sudo dnf install -y \
  python3 python3-pip python3-devel \
  git git-lfs make jq curl \
  gcc gcc-c++
git lfs install

Choose ONE container stack:

Docker (moby-engine):

sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Re-login or: newgrp docker

Podman:

sudo dnf install -y podman podman-compose
# Optional user socket:
systemctl --user enable --now podman.socket

openSUSE Leap/Tumbleweed (zypper)

Core tools:

sudo zypper refresh
sudo zypper install -y \
  python3 python3-pip python3-virtualenv python3-devel \
  git git-lfs make jq curl \
  gcc gcc-c++
git lfs install

Choose ONE container stack:

Docker:

sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Re-login or: newgrp docker

Podman:

sudo zypper install -y podman podman-compose
# Optional user socket:
systemctl --user enable --now podman.socket

---

Project scaffold

Start a clean repo:

mkdir ai-devops-demo && cd ai-devops-demo
git init

Recommended tree (we’ll generate these as we go):

.
├── data/
├── models/
├── scripts/
├── docker/
├── requirements.txt
├── dvc.yaml
├── params.yaml
└── Makefile

---

1) Reproducible environments with Python venv + Make

Keep dependencies simple and frozen. Build from scratch anywhere—local, CI, container.

Create a virtual environment:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip

Seed dependencies:

cat > requirements.txt <<'EOF'
mlflow==2.14.3
dvc[ssh]==3.51.2
pandas==2.2.2
scikit-learn==1.5.1
fastapi==0.115.0
uvicorn[standard]==0.30.6
prometheus-client==0.20.0
EOF

pip install -r requirements.txt

Add a Makefile to standardize commands:

cat > Makefile <<'EOF'
VENV=.venv
PY=$(VENV)/bin/python
PIP=$(VENV)/bin/pip

.PHONY: venv install freeze clean

venv:
    @test -d $(VENV) || python3 -m venv $(VENV)

install: venv
    $(PIP) install --upgrade pip
    $(PIP) install -r requirements.txt

freeze:
    $(PIP) freeze | sort > requirements.lock

clean:
    rm -rf $(VENV) .dvc/ dvc.lock mlruns __pycache__
EOF

Usage:

make install
make freeze

Why this matters:

• Deterministic installs with requirements.lock

• One-liners for CI and local parity

---

2) Data and experiment versioning with Git LFS, DVC, and MLflow

You’ll track large files in Git LFS, structure pipelines with DVC, and log metrics/artifacts with MLflow.

Initialize Git LFS and DVC:

git lfs track "data/*.csv" "models/*.pkl"
git add .gitattributes && git commit -m "Track large files with LFS"

dvc init
git add .dvc .gitignore && git commit -m "Init DVC"

Add minimal params and sample data:

cat > params.yaml <<'EOF'
train:
  test_size: 0.2
  random_state: 42
EOF

mkdir -p data scripts models
curl -L -o data/raw.csv https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv
git add params.yaml data/raw.csv && git commit -m "Params and sample data"

Training script (logs to MLflow and saves a model):

cat > scripts/train.py <<'PY'
import os, json
import mlflow
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
import joblib
import yaml

with open("params.yaml") as f:
    params = yaml.safe_load(f)["train"]

df = pd.read_csv("data/raw.csv")
X = df.drop(columns=["species"])
y = df["species"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=params["test_size"], random_state=params["random_state"], stratify=y
)

clf = LogisticRegression(max_iter=1000)
with mlflow.start_run():
    clf.fit(X_train, y_train)
    preds = clf.predict(X_test)
    acc = accuracy_score(y_test, preds)
    mlflow.log_param("model", "logreg")
    mlflow.log_metric("accuracy", acc)
    os.makedirs("models", exist_ok=True)
    joblib.dump(clf, "models/model.pkl")
    mlflow.log_artifact("models/model.pkl")
    with open("metrics.json", "w") as m:
        json.dump({"accuracy": acc}, m)
    mlflow.log_artifact("metrics.json")
    print(f"accuracy={acc:.4f}")
PY

Wire it up with DVC:

cat > dvc.yaml <<'YAML'
stages:
  train:
    cmd: .venv/bin/python scripts/train.py
    deps:
      - scripts/train.py
      - data/raw.csv
      - params.yaml
    outs:
      - models/model.pkl
    metrics:
      - metrics.json:
          cache: false
YAML

Run and version:

. .venv/bin/activate
dvc repro
git add dvc.yaml dvc.lock models/.gitignore metrics.json
git commit -m "Add training stage with DVC + MLflow"

Explore experiments:

mlflow ui --host 0.0.0.0 --port 5000
# Open http://localhost:5000

Why this matters:

• DVC guarantees reproducible pipelines bound to data and code versions.

• MLflow leaves a trace of each run: metrics, params, and artifacts.

---

3) Containerize training and serving (Docker or Podman)

Containers give you a consistent runtime for both training and inference. Build once, run anywhere.

Dockerfile:

mkdir -p docker
cat > docker/Dockerfile <<'DOCKER'
FROM python:3.11-slim

ENV PIP_NO_CACHE_DIR=1
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    git curl jq build-essential && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --upgrade pip && pip install -r requirements.txt

COPY . .
CMD ["bash", "-lc", ". .venv/bin/activate 2>/dev/null || python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt && dvc repro && mlflow ui --host 0.0.0.0 --port 5000"]
DOCKER

Build and run (Docker):

docker build -t ai-devops-demo -f docker/Dockerfile .
docker run --rm -p 5000:5000 -v "$(pwd)":/app -w /app ai-devops-demo

Build and run (Podman):

podman build -t ai-devops-demo -f docker/Dockerfile .
podman run --rm -p 5000:5000 -v "$(pwd)":/app -w /app ai-devops-demo

Why this matters:

• Same bits in CI and prod

• Minimal drift across developer machines

---

4) CI/CD and a tiny model service with metrics

Automate checks in CI and expose Prometheus metrics in prod.

Add a tiny FastAPI service that loads the trained model and exports metrics:

cat > scripts/serve.py <<'PY'
import joblib
from fastapi import FastAPI
from pydantic import BaseModel
from prometheus_client import Counter, Gauge, generate_latest, CONTENT_TYPE_LATEST
from fastapi.responses import Response

app = FastAPI()
model = joblib.load("models/model.pkl")

PRED_COUNT = Counter("predictions_total", "Total predictions")
MODEL_LOAD = Gauge("model_loaded", "Model loaded (1=ready)")
MODEL_LOAD.set(1)

class Features(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float

@app.get("/healthz")
def healthz():
    return {"status": "ok"}

@app.post("/predict")
def predict(x: Features):
    import numpy as np
    PRED_COUNT.inc()
    arr = np.array([[x.sepal_length, x.sepal_width, x.petal_length, x.petal_width]])
    pred = model.predict(arr)[0]
    return {"species": str(pred)}

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)
PY

Run locally:

. .venv/bin/activate
uvicorn scripts.serve:app --host 0.0.0.0 --port 8000
# curl -X POST localhost:8000/predict -H 'content-type: application/json' \
#   -d '{"sepal_length":5.1,"sepal_width":3.5,"petal_length":1.4,"petal_width":0.2}'
# curl localhost:8000/metrics

Optional: compose Prometheus + Grafana for metrics dashboards.

docker-compose.yml:

cat > docker/compose.yml <<'YAML'
services:
  app:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    command: bash -lc ". .venv/bin/activate || (python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt); uvicorn scripts.serve:app --host 0.0.0.0 --port 8000"
    ports: ["8000:8000"]
    volumes:
      - ..:/app
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports: ["9090:9090"]
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
YAML

Prometheus configuration:

cat > docker/prometheus.yml <<'YAML'
global:
  scrape_interval: 5s
scrape_configs:
  - job_name: "model-service"
    static_configs:
      - targets: ["app:8000"]
    metrics_path: /metrics
YAML

Start the stack:

Docker:

docker compose -f docker/compose.yml up --build

Podman:

podman-compose -f docker/compose.yml up --build

Basic CI (GitHub Actions) to test training on push:

mkdir -p .github/workflows
cat > .github/workflows/ci.yml <<'YAML'
name: ci
on: [push, pull_request]
jobs:
  test-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
      - run: . .venv/bin/activate && dvc repro
      - run: . .venv/bin/activate && python scripts/train.py
YAML

Why this matters:

• A health-checked service with /metrics is ready for real ops

• CI keeps training from silently breaking after code changes

---

Real-world tips

• Pin, then audit: use requirements.lock in production; review updates on a schedule.

• Track data storage wisely: DVC remotes (S3, SSH, etc.) keep your Git repo light. Start with a local remote, then move to cloud.

• Prefer Compose for local orchestration: it matches k8s patterns later without the complexity on day one.

• Keep Bash-friendly: wrap commands in Make targets so anyone can run make train, make serve.

---

Your next steps (CTA)

• Clone this scaffold into your next ML repo

• Replace the sample dataset and model with your own

• Bring your team: add CI, a DVC remote, and a simple Grafana dashboard

• Iterate: when you can retrain and redeploy with one command, you’re doing AI DevOps

Have questions or want a follow-up guide on GPU training, Kubernetes, or secure model registries? Tell me what you’re building, and we’ll go there next.