Posted on
• Artificial Intelligence

Artificial Intelligence DevOps Projects to Build

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

5 Hands‑On AI DevOps Projects You Can Build on Linux (Bash‑First)

If you’ve ever trained a great model that never made it past your laptop, you’re not alone. The hard truth: model quality is only half the battle—shipping, scaling, and maintaining it is where AI becomes real. That’s where DevOps meets ML (aka MLOps), and where Linux and Bash shine.

In this guide, you’ll get five practical, buildable AI DevOps projects you can run from your terminal. You’ll learn reproducible pipelines, containerized inference, experiment tracking, CI/CD, and monitoring—using tools you can install with apt, dnf, or zypper.

Why this matters:

  • Reproducibility: Know exactly which data/code created a model.

  • Velocity: Automate training, packaging, and deployment.

  • Reliability: Catch drift and regressions before your users do.

  • Cost: Use the right infra and scale only when needed.

Jump in, copy the commands, and adapt them to your stack.

Prerequisites: Install Core Tools

Pick either Podman (rootless by default; recommended on Fedora/RHEL/openSUSE) or Docker. You’ll also need Git, Python, and Make.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip make jq

# Option A: Podman (recommended for rootless containers)
sudo apt install -y podman podman-compose

# Option B: Docker (community package + compose plugin)
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
# Optional: run docker without sudo (log out/in after)
sudo usermod -aG docker "$USER"

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y git python3 python3-virtualenv python3-pip make jq

# Option A: Podman (recommended on Fedora/RHEL)
sudo dnf install -y podman podman-compose

# Option B: Docker (Moby engine + compose plugin; availability varies by distro/version)
sudo dnf install -y moby-engine docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y git python3 python3-venv python3-pip make jq

# Option A: Podman
sudo zypper install -y podman podman-compose

# Option B: Docker
sudo zypper install -y docker docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

Notes:

  • Use either docker compose ... or podman-compose ... consistently.

  • If you prefer a Python toolchain per project:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip

Project 1: Reproducible ML Pipeline with Git + DVC + Bash

Value: Ensure the exact data/code/params used to produce any model can be recovered and rebuilt on any machine.

Setup:

mkdir ml-pipeline && cd ml-pipeline
git init
python3 -m venv .venv && . .venv/bin/activate
pip install --upgrade pip
pip install "dvc[s3,ssh]" scikit-learn pandas joblib
dvc init

Create data and track it with DVC:

mkdir -p data/raw data/processed models
# put your dataset at data/raw/dataset.csv
dvc add data/raw/dataset.csv
git add data/.gitignore data/raw/dataset.csv.dvc .dvc .gitignore
git commit -m "Init DVC and track raw data"

Define a pipeline with stages (dvc.yaml):

cat > dvc.yaml <<'YAML'
stages:
  preprocess:
    cmd: bash scripts/preprocess.sh
    deps:
      - data/raw/dataset.csv
      - scripts/preprocess.sh
    outs:
      - data/processed/train.csv
      - data/processed/test.csv
  train:
    cmd: bash scripts/train.sh
    deps:
      - data/processed/train.csv
      - scripts/train.sh
    outs:
      - models/model.joblib
  eval:
    cmd: bash scripts/eval.sh
    deps:
      - data/processed/test.csv
      - models/model.joblib
      - scripts/eval.sh
    metrics:
      - metrics.json:
          cache: false
YAML

Add Bash stage scripts:

mkdir -p scripts
cat > scripts/preprocess.sh <<'BASH'
set -euo pipefail
python - <<'PY'
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("data/raw/dataset.csv")
train, test = train_test_split(df, test_size=0.2, random_state=42)
train.to_csv("data/processed/train.csv", index=False)
test.to_csv("data/processed/test.csv", index=False)
PY
BASH

cat > scripts/train.sh <<'BASH'
set -euo pipefail
python - <<'PY'
import pandas as pd, joblib
from sklearn.linear_model import LogisticRegression
df = pd.read_csv("data/processed/train.csv")
X = df.drop(columns=["target"]); y = df["target"]
m = LogisticRegression(max_iter=200).fit(X, y)
joblib.dump(m, "models/model.joblib")
PY
BASH

cat > scripts/eval.sh <<'BASH'
set -euo pipefail
python - <<'PY'
import pandas as pd, json, joblib
from sklearn.metrics import f1_score
df = pd.read_csv("data/processed/test.csv")
X = df.drop(columns=["target"]); y = df["target"]
m = joblib.load("models/model.joblib")
pred = m.predict(X)
metrics = {"f1": float(f1_score(y, pred))}
print(json.dumps(metrics))
open("metrics.json","w").write(json.dumps(metrics, indent=2))
PY
BASH
chmod +x scripts/*.sh

Run the pipeline:

dvc repro
cat metrics.json
git add dvc.yaml scripts metrics.json models/model.joblib.dvc data/processed/*.dvc
git commit -m "Add pipeline and initial model"

Optional: Configure a remote storage (local or cloud) to share data/artifacts:

dvc remote add -d storage s3://your-bucket/your-prefix
# or local directory:
# dvc remote add -d storage ../dvc-remote
dvc push

What you learned:

  • Data and artifacts are versioned without bloating Git.

  • One command (dvc repro) rebuilds the exact model.

Project 2: Containerized Model Service with FastAPI + Compose

Value: Ship your model as a stateless microservice you can run anywhere.

Create the service:

mkdir -p serving && cd serving
python3 -m venv .venv && . .venv/bin/activate
pip install fastapi uvicorn[standard] joblib scikit-learn

App code (app.py):

cat > app.py <<'PY'
from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI(title="Model Service")

class Features(BaseModel):
    values: list[float]

model = joblib.load("models/model.joblib")

@app.post("/predict")
def predict(f: Features):
    x = np.array(f.values).reshape(1, -1)
    y = model.predict(x).tolist()
    return {"prediction": y}
PY

Dockerfile:

cat > Dockerfile <<'DOCKER'
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py /app/
# Expect model volume mounted at /app/models
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
DOCKER

Requirements:

pip freeze | grep -E 'fastapi|uvicorn|joblib|scikit-learn' > requirements.txt

Compose (Docker):

cat > compose.yaml <<'YAML'
services:
  model:
    build: .
    image: model-service:latest
    ports:
      - "8000:8000"
    volumes:
      - ./models:/app/models:ro
YAML

Build and run:

  • With Docker:
docker compose up --build -d
curl -X POST localhost:8000/predict -H 'content-type: application/json' \
  -d '{"values":[0.1,0.2,0.3,0.4]}'
  • With Podman:
podman-compose -f compose.yaml up --build -d
curl -X POST localhost:8000/predict -H 'content-type: application/json' \
  -d '{"values":[0.1,0.2,0.3,0.4]}'

Tip: Keep models external (mounted volume) so you can hot‑swap a new model without rebuilding the image.

Project 3: Experiment Tracking + Model Registry with MLflow (Postgres + MinIO)

Value: Centralize metrics, artifacts, and model versions so your team can compare runs and promote models with a click (or API).

Compose stack:

mkdir -p mlflow && cd mlflow
cat > compose.yaml <<'YAML'
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: mlflow
      POSTGRES_DB: mlflow
    volumes: [ "pgdata:/var/lib/postgresql/data" ]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mlflow"]
      interval: 5s
      timeout: 5s
      retries: 10

  minio:
    image: quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
    ports: ["9000:9000", "9001:9001"]
    volumes: [ "minio:/data" ]

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    depends_on: [ postgres, minio ]
    environment:
      MLFLOW_TRACKING_URI: http://0.0.0.0:5000
    command:
      [
        "mlflow","server",
        "--backend-store-uri","postgresql+psycopg2://mlflow:mlflow@postgres/mlflow",
        "--artifacts-destination","s3://mlflow",
        "--host","0.0.0.0","--port","5000"
      ]
    ports: ["5000:5000"]
    environment:
      AWS_ACCESS_KEY_ID: minio
      AWS_SECRET_ACCESS_KEY: minio123
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000

volumes:
  pgdata: {}
  minio: {}
YAML

Start:

  • Docker:
docker compose up -d
  • Podman:
podman-compose -f compose.yaml up -d

Create a bucket in MinIO (UI at http://localhost:9001 or via API):

curl -s -X PUT "http://localhost:9000/mlflow" \
  -u minio:minio123

Log experiments from Python:

python - <<'PY'
import os, mlflow, mlflow.sklearn
os.environ["MLFLOW_TRACKING_URI"] = "http://127.0.0.1:5000"
os.environ["MLFLOW_S3_ENDPOINT_URL"] = "http://127.0.0.1:9000"
os.environ["AWS_ACCESS_KEY_ID"]="minio"
os.environ["AWS_SECRET_ACCESS_KEY"]="minio123"

import sklearn.datasets, sklearn.linear_model, sklearn.metrics
X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
with mlflow.start_run(run_name="logreg-baseline"):
    model = sklearn.linear_model.LogisticRegression(max_iter=200).fit(X, y)
    pred = model.predict(X)
    f1 = sklearn.metrics.f1_score(y, pred)
    mlflow.log_metric("f1", f1)
    mlflow.sklearn.log_model(model, "model", registered_model_name="BreastCancerLR")
print("Logged to MLflow at http://127.0.0.1:5000")
PY

Browse MLflow UI at http://localhost:5000 to compare runs and register models.

Project 4: CI/CD for ML with GitHub Actions

Value: Every push triggers tests, data/pipeline checks, image builds, and staged deploys—so “it works on my machine” stops being a blocker.

Sample workflow (.github/workflows/ml-ci.yaml):

mkdir -p .github/workflows
cat > .github/workflows/ml-ci.yaml <<'YAML'
name: ml-ci
on:
  push:
    branches: [ "main" ]
  pull_request:

jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install deps
        run: |
          python -m pip install --upgrade pip
          pip install dvc[ssh,s3] -r requirements.txt

      - name: Reproduce pipeline and verify metrics
        run: |
          dvc repro
          jq . metrics.json
          # simple guard: fail if f1 < 0.90
          python - <<'PY'
import json, sys
m=json.load(open("metrics.json"))
assert m["f1"] >= 0.90, f"F1 too low: {m['f1']}"
PY

      - name: Build container
        run: |
          echo "IMAGE=ghcr.io/${{ github.repository_owner }}/model-service:${{ github.sha }}" >> $GITHUB_ENV
          docker build -t $IMAGE -f Dockerfile .

      - name: Log in to GHCR
        if: github.ref == 'refs/heads/main'
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Push image
        if: github.ref == 'refs/heads/main'
        run: docker push $IMAGE
YAML

Add environment‑specific deploy jobs (e.g., staging) after the image push. If you’re using Docker Compose on a VM, a simple SSH deploy step can pull the new image and restart the service.

Project 5: Production Monitoring with Prometheus + Grafana + Custom Exporter

Value: Watch model health in real time: latency, error rates, and basic drift signals.

Compose stack:

mkdir -p monitoring && cd monitoring
cat > prometheus.yml <<'YAML'
global:
  scrape_interval: 5s
scrape_configs:
  - job_name: 'model'
    static_configs:
      - targets: ['exporter:9100']
YAML

cat > compose.yaml <<'YAML'
services:
  exporter:
    build: ./exporter
    ports: [ "9100:9100" ]
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes: [ "./prometheus.yml:/etc/prometheus/prometheus.yml:ro" ]
    ports: [ "9090:9090" ]
  grafana:
    image: grafana/grafana:10.4.5
    ports: [ "3000:3000" ]
YAML

Custom metrics exporter (minimal drift proxy):

mkdir -p exporter && cat > exporter/Dockerfile <<'DOCKER'
FROM python:3.11-slim
WORKDIR /app
RUN pip install prometheus-client numpy
COPY exporter.py /app/
EXPOSE 9100
CMD ["python","exporter.py"]
DOCKER

cat > exporter/exporter.py <<'PY'
from prometheus_client import start_http_server, Gauge
import random, time
# Example metrics: latency, 5m moving avg drift (toy)
latency_ms = Gauge('model_latency_ms', 'P50 inference latency (ms)')
drift = Gauge('input_drift_score', 'Toy drift score 0..1')

if __name__ == "__main__":
    start_http_server(9100)
    while True:
        latency_ms.set(20 + random.random()*5)
        drift.set(random.random()*0.2)
        time.sleep(5)
PY

Start:

  • Docker:
docker compose up --build -d
  • Podman:
podman-compose -f compose.yaml up --build -d

Dashboards:

  • Prometheus: http://localhost:9090

  • Grafana: http://localhost:3000 (default admin/admin; add Prometheus data source at http://prometheus:9090)

Integrate with your FastAPI service by exposing a /metrics endpoint (use prometheus-client) and add it to Prometheus’ scrape configs.

Why These Projects Work (and Scale)

  • They’re modular. You can replace FastAPI with your framework, DVC with your orchestrator, or MLflow with another registry.

  • They’re cloud‑agnostic. Compose and rootless containers run locally, on VMs, or inside CI.

  • They’re testable. Every project has clear CLI entry points you can wire into CI.

  • They’re upgradeable. Swap local Compose for Kubernetes later without changing core patterns.

What to Build Next (CTA)

  • Pick one project and ship it in the next hour:

    • Project 1 if your data and results aren’t reproducible.
    • Project 2 if your team needs a working API now.
    • Project 3 if experiments are scattered across laptops.
    • Project 4 if merges keep breaking models.
    • Project 5 if you’ve been paged at 2 a.m. without useful metrics.

  • Add docs: a README with commands your teammates can paste.

  • Automate one more step each week (tests, lint, security scan, blue/green deploy).

When you’re ready to scale, port the Compose services to Helm charts, add a feature store, and wire canary releases. But the Bash you wrote today will still be the glue.

Happy shipping.

Related posts