Posted on
• Artificial Intelligence

Building Open Source Artificial Intelligence Workflows

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Building Open Source Artificial Intelligence Workflows (with Bash)

You don’t need a proprietary platform to build robust AI. With Linux, Bash, and a handful of battle-tested open source tools, you can ship reproducible, portable AI workflows that your whole team can run anywhere. The problem most teams face isn’t modeling—it’s gluing together data, experiments, training, and deployment in a way that’s reliable and repeatable. This guide shows you how to do exactly that, using the command line you already know.

What you’ll get:

  • A clear, minimal stack for open AI workflows

  • Actionable steps and real examples

  • Installation commands for apt, dnf, and zypper

  • Bash- and Makefile-driven automation you can copy into your repo

Why build AI workflows this way?

  • Reproducibility: Pin dependencies, version data, containerize runtime. Anyone can rerun your work.

  • Transparency: Open tools (Git, DVC, MLflow, Make) are auditable and vendor-neutral.

  • Portability: Same commands work on dev laptops, CI, or servers.

  • Cost control: Use free tools, scale up with containers only when needed.

1) Install the essentials and scaffold your project

We’ll use Git for code, Python + venv for dependencies, Make for orchestration.

Install base tools:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git python3 python3-pip python3-venv make build-essential
  • Fedora (dnf):
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git python3 python3-pip make
  • openSUSE (zypper):
sudo zypper install -y git python3 python3-pip make gcc
# Recommended developer pattern (installs build tools):
sudo zypper install -y -t pattern devel_basis

Create a repo and a Python virtual environment:

mkdir open-ai-workflow && cd open-ai-workflow
git init

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Add minimal dependencies:

cat > requirements.txt << 'EOF'
dvc
mlflow
scikit-learn
joblib
EOF

pip install -r requirements.txt

Pro tip: Add a .gitignore:

cat > .gitignore << 'EOF'
.venv/
__pycache__/
*.pyc
models/
mlruns/
data/
EOF

2) Version your data and experiments (Git + DVC + MLflow)

DVC tracks datasets and model artifacts alongside Git without bloating your repo.

Initialize DVC:

dvc init
git add .dvc .gitignore
git commit -m "Initialize DVC"

Create a place for raw data:

mkdir -p data/raw
# Put files here as needed, or generate with a script later

Track data with DVC:

dvc add data/raw
git add data/raw.dvc .gitignore
git commit -m "Track raw data with DVC"

Spin up MLflow locally to explore metrics:

mlflow ui --port 5000
# Open http://127.0.0.1:5000

3) Write small, testable training scripts

Keep scripts pure and CLI-driven so they compose well in Bash/Make. Here’s a minimal training example using scikit-learn and MLflow.

train.py:

#!/usr/bin/env python3
import argparse, os, joblib, mlflow
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--C", type=float, default=1.0)
    parser.add_argument("--max-iter", type=int, default=200)
    parser.add_argument("--model-dir", type=str, default="models")
    args = parser.parse_args()

    os.makedirs(args.model_dir, exist_ok=True)

    X, y = load_wine(return_X_y=True)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

    with mlflow.start_run():
        clf = LogisticRegression(C=args.C, max_iter=args.max_iter, n_jobs=-1)
        clf.fit(Xtr, ytr)
        preds = clf.predict(Xte)
        acc = accuracy_score(yte, preds)

        mlflow.log_param("C", args.C)
        mlflow.log_param("max_iter", args.max_iter)
        mlflow.log_metric("accuracy", acc)

        out_path = os.path.join(args.model_dir, "model.pkl")
        joblib.dump(clf, out_path)
        mlflow.log_artifact(out_path)
        print(f"Saved model to {out_path} | accuracy={acc:.4f}")

if __name__ == "__main__":
    main()

Make it executable and test it:

chmod +x train.py
source .venv/bin/activate
./train.py --C 0.5 --max-iter 400

You now have:

  • Re-runnable training

  • Metrics in MLflow (open UI to compare runs)

  • A versionable artifact in models/

4) Automate the workflow with Make and Bash

Use Make as your “one-liner API.” It documents and automates steps for teammates and CI.

Makefile:

SHELL := /usr/bin/env bash
.ONESHELL:
.SHELLFLAGS := -eu -o pipefail -c

PY := .venv/bin/python
PIP := .venv/bin/pip

.PHONY: help bootstrap data train eval mlflow-ui clean repro

help:
    @echo "Targets:"
    @echo "  make bootstrap  - create venv and install deps"
    @echo "  make data       - prepare or pull data via DVC"
    @echo "  make train      - train the model"
    @echo "  make eval       - (example) evaluate or test"
    @echo "  make mlflow-ui  - run MLflow UI on :5000"
    @echo "  make repro      - end-to-end (data->train->eval)"
    @echo "  make clean      - clean artifacts"

bootstrap:
    test -d .venv || python3 -m venv .venv
    source .venv/bin/activate
    $(PIP) install --upgrade pip
    $(PIP) install -r requirements.txt

data:
    mkdir -p data/raw
    # If using remote storage: dvc pull
    # If generating data, call a script here.

train: bootstrap
    $(PY) train.py --C 0.8 --max-iter 300

eval: bootstrap
    # Placeholder: run evaluation scripts, load models/models.pkl, etc.
    @echo "Eval step not implemented yet."

mlflow-ui: bootstrap
    . .venv/bin/activate && mlflow ui --port 5000

repro: data train eval

clean:
    rm -rf models/ mlruns/

Run it:

make data
make train
make mlflow-ui

Everything important now has a single command. This is gold for onboarding and CI.

5) Containerize for runtime parity (Docker/Podman)

Containers lock in dependencies and make your workflow runnable on any machine or CI agent.

Dockerfile:

FROM python:3.11-slim

ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -r requirements.txt

COPY train.py /app/
COPY Makefile /app/

# Non-root for safety (optional)
RUN useradd -ms /bin/bash runner
USER runner

CMD ["make", "train"]

Build and run with Docker:

docker build -t open-ai-workflow:latest .
docker run --rm -it -p 5000:5000 open-ai-workflow:latest

Build and run with Podman:

podman build -t open-ai-workflow:latest .
podman run --rm -it -p 5000:5000 open-ai-workflow:latest

Install Docker/Podman:

  • Debian/Ubuntu (apt):
# Docker (community package)
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker

# Podman (rootless by default)
sudo apt install -y podman
  • Fedora (dnf):
# Podman (recommended default on Fedora)
sudo dnf install -y podman

# Docker (Moby) from Fedora repos
sudo dnf install -y moby-engine
sudo systemctl enable --now docker
  • openSUSE (zypper):
sudo zypper install -y docker podman
sudo systemctl enable --now docker

Note: If you use private registries or need GPU support, consult your distro’s docs for auth/NVIDIA runtime setup.

Real-world flow: tie it together

  • Develop locally with venv and Make:
make bootstrap
make train
make mlflow-ui    # view metrics in the browser
  • Version data and models:
dvc add data/raw
git add data/raw.dvc
git commit -m "Track data"

git add models/ mlruns/ -f   # or rely on MLflow artifacts only; tailor to your policy
git commit -m "Save model and runs (demo)"
  • Containerize and hand off to CI:
docker build -t registry.example.com/open-ai-workflow:$(git rev-parse --short HEAD) .
docker push registry.example.com/open-ai-workflow:$(git rev-parse --short HEAD)

Optional: a minimal GitHub Actions workflow to run training on push (ubuntu has Docker preinstalled, but you can also run natively with Python):

.github/workflows/train.yml

name: Train
on: [push, pull_request]
jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: |
          python -m venv .venv
          . .venv/bin/activate
          pip install -r requirements.txt
          ./train.py --C 0.7 --max-iter 350

Common variations and upgrades

  • Storage backends for DVC: add S3/GCS/SSH remotes and run dvc push/pull.

  • Heavier frameworks: for CPU-only PyTorch:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
  • Parameter sweeps: make targets or a small Bash loop calling train.py with different flags.

Example sweep.sh:

#!/usr/bin/env bash
set -euo pipefail
for C in 0.1 0.5 1.0 2.0; do
  ./train.py --C "$C" --max-iter 300
done

Conclusion / Call to Action

Open source AI workflows don’t have to be complex. With Git, DVC, MLflow, Make, and containers, you can build a transparent, reproducible pipeline your team can trust—and you can do it all from Bash.

Next steps: 1) Copy the Makefile, train.py, and Dockerfile into a fresh repo. 2) Replace the example model with your task, wire in DVC remotes, and start logging metrics. 3) Add one CI job to run make repro on every PR.

If you get stuck, start small: one script, one dataset, one Make target. Ship something reproducible today.

Related posts