Posted on
• Artificial Intelligence

Artificial Intelligence DevOps Pipelines

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Bash-first AI DevOps Pipelines on Linux: A Practical Guide

AI models don’t fail because of math—they fail because of messy handoffs, missing data lineage, and “it worked on my machine” deployments. If you’re a Linux user who prefers Bash and reproducible builds, this guide shows you how to build an Artificial Intelligence DevOps pipeline you can trust: versioned data, automated training, containerized serving, and CI hooks—all with simple shell commands.

What you’ll get:

  • Why AI DevOps pipelines matter and what they solve

  • A Bash-first, portable toolkit: Git, Python, DVC, MLflow, Podman/Docker, Make

  • 3–5 actionable steps to stand up a real, reproducible pipeline

  • Copy-pasteable commands for apt, dnf, and zypper

Why AI DevOps Pipelines Matter

  • Reproducibility: Same data, same environment, same result. Pipelines formalize every step and dependency.

  • Speed and safety: Automate training, testing, and packaging so you can move fast without breaking prod.

  • Observability: Track metrics and artifacts (models, datasets) with traceability for audits and rollbacks.

  • Portability: Containerized workflows run anywhere—your laptop, CI, or the cloud.

Prerequisites: Install the Basics

Install these once. They’re available from standard repos across major distros.

Packages:

  • git, python3, python3-venv, python3-pip, make

  • curl, jq

  • podman, buildah, skopeo (rootless containers; Docker users can swap commands)

  • git-lfs (for large files in Git)

  • shellcheck (for Bash linting)

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  git python3 python3-venv python3-pip make \
  curl jq \
  podman buildah skopeo \
  git-lfs shellcheck

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y \
  git python3 python3-venv python3-pip make \
  curl jq \
  podman buildah skopeo \
  git-lfs shellcheck

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  git python3 python3-venv python3-pip make \
  curl jq \
  podman buildah skopeo \
  git-lfs ShellCheck

Note for Docker users:

  • Docker isn’t in all default repos. If you prefer Docker, follow Docker’s official repo install docs, then replace podman with docker in the commands below.

Step 1: Scaffold a Minimal, Reproducible Project

Initialize a repository and Python environment.

mkdir -p ai-pipeline/{scripts,data,models}
cd ai-pipeline
git init
python3 -m venv venv
. venv/bin/activate
python -m pip install --upgrade pip
pip install \
  scikit-learn numpy pandas \
  dvc mlflow \
  fastapi uvicorn[standard] \
  black isort flake8 pytest pre-commit

Enable Git LFS and pre-commit:

git lfs install
cat > .pre-commit-config.yaml << 'EOF'
repos:

- repo: https://github.com/psf/black
  rev: 24.3.0
  hooks: [{id: black}]

- repo: https://github.com/pycqa/isort
  rev: 5.13.2
  hooks: [{id: isort}]

- repo: https://github.com/pycqa/flake8
  rev: 7.0.0
  hooks: [{id: flake8}]

- repo: https://github.com/koalaman/shellcheck-precommit
  rev: v0.10.0
  hooks: [{id: shellcheck}]
EOF
pre-commit install

Create a simple data fetch script:

cat > scripts/fetch_data.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
out="${1:-data/raw.csv}"
mkdir -p "$(dirname "$out")"
# Synthetic dataset for demo
python - <<PY > "$out"
import numpy as np, pandas as pd
from sklearn.datasets import make_classification
X,y = make_classification(n_samples=1000, n_features=20, random_state=42)
cols = [f"f{i}" for i in range(X.shape[1])]
df = pd.DataFrame(X, columns=cols)
df["target"] = y
df.to_csv("$out", index=False)
print(f"Wrote {out}")
PY
echo "OK: $out"
EOF
chmod +x scripts/fetch_data.sh

Training script:

cat > scripts/train.py << 'EOF'
import argparse, joblib, os
import mlflow
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--data", required=True)
    p.add_argument("--model", required=True)
    args = p.parse_args()

    df = pd.read_csv(args.data)
    X = df.drop(columns=["target"]).values
    y = df["target"].values
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)

    mlflow.set_tracking_uri("file://" + os.path.abspath("mlruns"))
    with mlflow.start_run():
        clf = LogisticRegression(max_iter=1000, n_jobs=-1)
        clf.fit(Xtr, ytr)
        acc = clf.score(Xte, yte)
        mlflow.log_metric("accuracy", acc)
        os.makedirs(os.path.dirname(args.model), exist_ok=True)
        joblib.dump(clf, args.model)
        mlflow.log_artifact(args.model)

    print(f"Trained model saved to {args.model}")

if __name__ == "__main__":
    main()
EOF

Evaluation script:

cat > scripts/evaluate.py << 'EOF'
import argparse, json
import joblib, pandas as pd
from sklearn.metrics import accuracy_score

p = argparse.ArgumentParser()
p.add_argument("--data", required=True)
p.add_argument("--model", required=True)
p.add_argument("--metrics", required=True)
args = p.parse_args()

df = pd.read_csv(args.data)
X = df.drop(columns=["target"]).values
y = df["target"].values
clf = joblib.load(args.model)
pred = clf.predict(X)
acc = accuracy_score(y, pred)
metrics = {"accuracy_full_dataset": float(acc)}
with open(args.metrics, "w") as f:
    json.dump(metrics, f)
print(json.dumps(metrics, indent=2))
EOF

Step 2: Track Data and Models with DVC

Initialize DVC and define a pipeline. DVC treats data, models, and metrics as first-class versioned citizens.

dvc init
git add .dvc .gitignore
git commit -m "Initialize DVC"

Create dvc.yaml:

cat > dvc.yaml << 'EOF'
stages:
  data:
    cmd: bash scripts/fetch_data.sh data/raw.csv
    deps:
    - scripts/fetch_data.sh
    outs:
    - data/raw.csv
  train:
    cmd: venv/bin/python scripts/train.py --data data/raw.csv --model models/model.pkl
    deps:
    - scripts/train.py
    - data/raw.csv
    outs:
    - models/model.pkl
  eval:
    cmd: venv/bin/python scripts/evaluate.py --data data/raw.csv --model models/model.pkl --metrics metrics.json
    deps:
    - scripts/evaluate.py
    - data/raw.csv
    - models/model.pkl
    metrics:
    - metrics.json
EOF

Reproduce the pipeline end-to-end:

. venv/bin/activate
dvc repro

Optional: Configure a remote (S3, SSH, etc.) to store data/models outside Git:

dvc remote add -d storage s3://my-bucket/ai-pipeline
dvc push

Step 3: Orchestrate with Make (Bash-friendly)

A Makefile gives you one-liners for setup, training, testing, and packaging.

cat > Makefile << 'EOF'
SHELL := /usr/bin/env bash
CONTAINER ?= podman

.PHONY: help
help:
    @echo "Targets:"
    @echo "  venv        - Create Python venv and install deps"
    @echo "  fmt         - Format Python code"
    @echo "  lint        - Lint Python and shell scripts"
    @echo "  repro       - Run full DVC pipeline"
    @echo "  build       - Build container image"
    @echo "  run         - Run API server container"
    @echo "  test        - Run unit tests"

venv:
    python3 -m venv venv
    venv/bin/pip install --upgrade pip
    venv/bin/pip install -r requirements.txt || true

fmt:
    venv/bin/black .
    venv/bin/isort .

lint:
    venv/bin/flake8 .
    find scripts -type f -name "*.sh" -print0 | xargs -0 -r shellcheck

repro:
    . venv/bin/activate && dvc repro

build:
    $(CONTAINER) build -t ai-pipeline:latest -f Dockerfile .

run:
    $(CONTAINER) run --rm -p 8000:8000 ai-pipeline:latest

test:
    venv/bin/pytest -q
EOF

Lock your Python dependencies:

cat > requirements.txt << 'EOF'
scikit-learn
numpy
pandas
dvc
mlflow
fastapi
uvicorn[standard]
black
isort
flake8
pytest
joblib
EOF

Step 4: Containerize a Minimal Inference API

Add a small FastAPI app to serve predictions. This allows CI/CD to ship a runnable artifact.

cat > app.py << 'EOF'
import joblib
import numpy as np
from fastapi import FastAPI
from pydantic import BaseModel

class Features(BaseModel):
    values: list[float]

app = FastAPI()
model = joblib.load("models/model.pkl")

@app.get("/healthz")
def healthz():
    return {"status": "ok"}

@app.post("/predict")
def predict(f: Features):
    x = np.array([f.values])
    y = model.predict(x).tolist()
    return {"prediction": y[0]}
EOF

Containerfile (works with Podman or Docker):

cat > Dockerfile << 'EOF'
FROM python:3.11-slim
RUN useradd -m app
WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
USER app
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host=0.0.0.0", "--port=8000"]
EOF

Build and run (Podman; swap to docker if you prefer):

podman build -t ai-pipeline:latest .
podman run --rm -p 8000:8000 ai-pipeline:latest

Test:

curl -s http://localhost:8000/healthz
curl -s -X POST http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"values":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,2.0]}'

Step 5: Automate with CI (GitHub Actions Example)

Add a CI workflow that formats, lints, reproduces the pipeline, and builds the container image on each push.

mkdir -p .github/workflows
cat > .github/workflows/ci.yml << 'EOF'
name: ci
on:
  push:
  pull_request:
jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: System deps
      run: |
        sudo apt-get update
        sudo apt-get install -y python3-venv python3-pip git-lfs shellcheck podman
        git lfs install
    - name: Python deps
      run: |
        python3 -m venv venv
        venv/bin/pip install --upgrade pip
        venv/bin/pip install -r requirements.txt
    - name: Lint & Format
      run: |
        venv/bin/black --check .
        venv/bin/isort --check-only .
        venv/bin/flake8 .
        find scripts -type f -name "*.sh" -print0 | xargs -0 -r shellcheck
    - name: Reproduce pipeline
      run: |
        . venv/bin/activate
        dvc repro
    - name: Build container
      run: |
        podman build -t ai-pipeline:ci .
EOF

Real-World Patterns You Can Reuse Today

  • Nightly retraining with artifact retention:

    • Use DVC with an S3/SSH remote. Schedule a nightly dvc repro && dvc push in CI. Keep N best models by metric using a simple script that reads metrics.json.

  • Governance and auditability:

    • Tie MLflow runs to Git SHA and DVC data version. You’ll always know which data/model/code produced which metric.

  • Fast rollback:

    • Tag container images with both latest and a version (YYYYMMDD.sha). If metrics degrade, deploy the prior tag.

  • Local-to-prod parity:

    • Podman rootless builds and runs mimic production containers without extra daemon complexity.

Putting It All Together (Quickstart Recap)

  • Install prerequisites (apt/dnf/zypper commands above).

  • Create venv, install Python deps, and initialize DVC.

  • Add scripts for data, train, and evaluate. Reproduce with dvc repro.

  • Containerize the app and run via podman run -p 8000:8000.

  • Wire up CI to lint, test, repro, and build on every push.

If you want a single-shot run locally:

. venv/bin/activate
dvc repro
podman build -t ai-pipeline:latest .
podman run --rm -p 8000:8000 ai-pipeline:latest

Conclusion and Call to Action

You now have a Bash-first AI DevOps pipeline: versioned data and models (DVC), tracked metrics (MLflow), reproducible orchestration (Make), and portable deployment (Podman/Docker). This foundation scales from your laptop to CI and production with minimal changes.

Your next steps:

  • Add real datasets and connect a DVC remote (S3/SSH).

  • Expand tests and add monitoring (drift alerts, latency SLOs).

  • Automate promotion: only ship models that beat baseline metrics.

Clone this pattern into your next AI project and ship models you can trust—confidently, repeatedly, and with a single command.

Related posts