Posted on
• Artificial Intelligence

Artificial Intelligence Workflow Projects

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

AI Workflow Projects on Linux: From Notebook to Repeatable Pipelines with Bash

You don’t need a fleet of cloud services or heavyweight orchestration to ship machine learning reliably. With Bash, Make, and containers, you can build reproducible AI workflows that run on any Linux box—laptop to server—using tools you already trust.

The problem: Notebooks are great for exploration, but they often fail when you need to run the same steps tomorrow, on another machine, or inside a CI pipeline. The value: Turn your one-off experiments into predictable, versioned, testable workflows with a few small scripts and standard Linux utilities.

This post shows you a concrete, end-to-end example: fetching data with curl and jq, preparing features with Bash, training with Python, packaging with containers, and automating with cron.

Why a Bash-first AI workflow is a great fit

  • Deterministic and transparent: Each step is a plain command you can read, audit, and re-run.

  • Reproducible anywhere: If it runs in a terminal, it runs in CI and in containers.

  • Minimal dependencies: Uses standard Linux tools; no need for complex orchestrators to start.

  • Composable: Makefiles and scripts let you chain tasks, pass artifacts, and track results.

Prerequisites: Install core tools

Install system packages with your distro’s package manager. Choose either Podman (rootless by default) or Docker as your container engine.

Core tools (git, make, Python, pip, jq, curl):

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y git make python3 python3-venv python3-pip jq curl
  • Fedora/RHEL (dnf):
sudo dnf install -y git make python3 python3-pip jq curl
  • openSUSE (zypper):
sudo zypper install -y git make python3 python3-pip python3-virtualenv jq curl

Container engine (choose one):

  • Podman

    • apt:
    sudo apt install -y podman
    • dnf:
    sudo dnf install -y podman
    • zypper:
    sudo zypper install -y podman

  • Docker

    • apt:
    sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
    • dnf (Fedora provides Docker via moby-engine):
    sudo dnf install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
    • zypper:
    sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Cron (for scheduled automation):

  • apt:
sudo apt install -y cron
sudo systemctl enable --now cron
  • dnf:
sudo dnf install -y cronie
sudo systemctl enable --now crond
  • zypper:
sudo zypper install -y cron
sudo systemctl enable --now cron

Project goal and dataset

We’ll build a tiny forecasting pipeline that:

  • Fetches hourly temperature data from the Open-Meteo API (no key needed)

  • Prepares a simple supervised dataset (predict next hour’s temperature from the current hour)

  • Trains a linear regression model

  • Packages inference into a container

  • Automates retraining with cron

Directory structure:

ai-workflow/
├── artifacts/            # trained model
├── data/                 # raw and processed data
├── metrics/              # training metrics
├── scripts/              # bash helpers (fetch/prepare)
├── src/                  # python code (train/predict)
├── Makefile
├── requirements.txt
└── Dockerfile

Step 1: Scaffold the project

Create folders and files:

mkdir -p ai-workflow/{artifacts,data/raw,data/processed,metrics,scripts,src}
cd ai-workflow

requirements.txt:

pandas
scikit-learn
joblib

Makefile:

SHELL := /usr/bin/env bash
ENGINE ?= podman
VENV := .venv
PY := $(VENV)/bin/python

.PHONY: setup data prepare train evaluate predict container clean

setup:
    python3 -m venv $(VENV)
    $(PY) -m pip install -U pip
    $(PY) -m pip install -r requirements.txt

data:
    bash scripts/fetch.sh data/raw/weather.json

prepare: data
    bash scripts/prepare.sh data/raw/weather.json data/processed/features.csv

train: setup prepare
    $(PY) src/train.py data/processed/features.csv artifacts/model.pkl metrics/metrics.json

evaluate:
    jq . metrics/metrics.json

predict: train
    echo 21.0 | $(PY) src/predict.py -

container: train
    $(ENGINE) build -t ai-workflow:latest .

clean:
    rm -rf $(VENV) artifacts metrics data

Step 2: Ingest and prepare with Bash + curl + jq

scripts/fetch.sh:

#!/usr/bin/env bash
set -euo pipefail

out="${1:-data/raw/weather.json}"
mkdir -p "$(dirname "$out")"

# Override via env: LAT=... LON=...
lat="${LAT:-52.52}"   # Berlin
lon="${LON:-13.41}"

curl -fsSL \
  "https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${lon}&hourly=temperature_2m&past_days=2&forecast_days=1&timezone=UTC" \
  -o "$out"
echo "Wrote $out"

scripts/prepare.sh:

#!/usr/bin/env bash
set -euo pipefail

in="${1:-data/raw/weather.json}"
out="${2:-data/processed/features.csv}"
mkdir -p "$(dirname "$out")"

# Build a two-column CSV: current_temp, next_hour_temp
jq -r '
  .hourly.temperature_2m as $t
  | ["temp_t","temp_t_plus_1"],
    (range(0; ($t|length - 1)) | [$t[.], $t[.+1]])
  | @csv
' "$in" > "$out"

echo "Wrote $out"

Make them executable:

chmod +x scripts/*.sh

Test the ingestion:

make data
make prepare
head -n 5 data/processed/features.csv

Step 3: Train and evaluate the model

src/train.py:

import json, os, sys
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import joblib

def main(in_csv, model_out, metrics_out):
    df = pd.read_csv(in_csv)
    X = df["temp_t"].values.reshape(-1, 1)
    y = df["temp_t_plus_1"].values

    n = int(0.8 * len(df))
    X_train, X_test = X[:n], X[n:]
    y_train, y_test = y[:n], y[n:]

    model = LinearRegression().fit(X_train, y_train)
    preds = model.predict(X_test)
    mae = float(np.mean(np.abs(preds - y_test)))

    os.makedirs(os.path.dirname(model_out), exist_ok=True)
    joblib.dump(model, model_out)

    os.makedirs(os.path.dirname(metrics_out), exist_ok=True)
    with open(metrics_out, "w") as f:
        json.dump({"mae": mae, "n_train": int(len(X_train)), "n_test": int(len(X_test))}, f)

    print(f"Saved model to {model_out}")
    print(f"Metrics: MAE={mae:.3f}")

if __name__ == "__main__":
    in_csv, model_out, metrics_out = sys.argv[1], sys.argv[2], sys.argv[3]
    main(in_csv, model_out, metrics_out)

src/predict.py:

import os, sys
import numpy as np
import joblib

# Usage:
#   echo 21.0 | python src/predict.py -
#   python src/predict.py 21.0
# MODEL_PATH can override the model location
model_path = os.environ.get("MODEL_PATH", "artifacts/model.pkl")
model = joblib.load(model_path)

def read_input():
    if len(sys.argv) > 1 and sys.argv[1] != "-":
        return float(sys.argv[1])
    return float(sys.stdin.read().strip())

x = np.array([[read_input()]])
y = model.predict(x)[0]
print(y)

Run the pipeline:

make train
make evaluate
echo 22.5 | .venv/bin/python src/predict.py -

Step 4: Package inference in a container

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ /app/src/
COPY artifacts/model.pkl /app/artifacts/model.pkl
ENTRYPOINT ["python", "src/predict.py"]

Build and run with Podman:

make container        # uses ENGINE=podman by default
echo 23.0 | podman run -i --rm ai-workflow:latest -

Or with Docker:

make container ENGINE=docker
echo 23.0 | docker run -i --rm ai-workflow:latest -

Tip: To use a different trained model at runtime, mount it and set MODEL_PATH:

echo 23.0 | podman run -i --rm -e MODEL_PATH=/m/model.pkl -v "$PWD/artifacts/model.pkl":/m/model.pkl:ro ai-workflow:latest -

Step 5: Automate retraining with cron

Add a cron job to fetch fresh data and retrain nightly. First, ensure cron is installed and running (see “Prerequisites” above).

Edit your user crontab:

crontab -e

Add a line (runs at 02:30 UTC daily):

30 2 * * * cd /path/to/ai-workflow && make train >> logs/cron.log 2>&1

Create the logs folder:

mkdir -p logs

Optional: Pin Python and system packages in CI, and have cron only kick off a containerized retrain to guarantee identical environments.

Real-world tips and extensions

  • Version everything: data snapshots, code, and artifacts. Tag releases after each successful training run.

  • Make targets composable: add test, lint, quality for quick checks; wire them into CI.

  • Track metrics: write JSONL per run with timestamp and git SHA to compare models.

  • Promote models: gate deployment steps on metric thresholds (e.g., MAE improved by X%).

  • Scale incrementally: when ready, wrap Make targets in a simple CI pipeline or use a lightweight queue. You can keep the same scripts.

Conclusion and next steps (CTA)

You’ve just built an end-to-end AI workflow using only Linux-native tools: Bash, Make, Python, and containers. It’s fast to run locally, easy to reason about, and ready for CI or cron.

Next steps:

  • Customize LAT/LON in scripts/fetch.sh to train on your region: LAT=40.71 LON=-74.00 make train

  • Add features (hour-of-day, moving averages) in scripts/prepare.sh and retrain

  • Expose predictions via a tiny FastAPI app and extend the Dockerfile

  • Wire make targets into GitHub Actions or GitLab CI for continuous training

Small, composable pieces beat complicated stacks when you’re moving fast. Start here, iterate, and scale only when you need to.

Related posts