Posted on
• Artificial Intelligence

Artificial Intelligence Lab Automation

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Automate Your AI Lab With Bash: Reproducible, Faster, and Hands-Off

If you spend more time babysitting experiments than doing science, you’re not alone. Ad-hoc environments, flaky data pulls, and manual runs chew up hours and make results hard to trust. The fix? Treat your AI lab like code and automate it with Bash.

In this guide, you’ll standardize environments, version datasets, batch experiments, containerize workflows, and auto-generate reports. Everything is CLI-first, Linux-native, and reproducible.

  • Who this is for: Researchers, data scientists, and MLEs on Linux.

  • What you’ll get: A working “labctl” Bash skeleton you can drop into any AI project.

Why Bash for AI Lab Automation?

  • Portability: Ship commands that work across Debian/Ubuntu, Fedora/RHEL, and openSUSE.

  • Transparency: No hidden GUI state; everything’s in scripts and logs.

  • Composability: Glue Python, containers, schedulers, and cloud CLIs together easily.

  • Reproducibility: Deterministic setup and run steps; audits become simple.

  • Speed: One-liners for batch runs, parallelism, and quick iteration.

Prerequisites: Install Core Tools

Commands below cover apt (Debian/Ubuntu), dnf (Fedora/RHEL/CentOS), and zypper (openSUSE). Choose your distro’s block.

System packages: Python 3, virtual env support, pip, Git, Git LFS, curl, jq, GNU parallel, tmux, plus either Docker or Podman.

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip \
  git git-lfs curl jq parallel tmux \
  docker.io podman
git lfs install
# Optional: enable Docker rootless-less convenience
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Fedora/RHEL/CentOS (dnf):

sudo dnf install -y \
  python3 python3-pip \
  git git-lfs curl jq parallel tmux \
  podman moby-engine
git lfs install
# Optional: start Docker (moby-engine) if you prefer Docker over Podman
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
# Note: On RHEL/CentOS you may need EPEL for git-lfs/parallel:
# sudo dnf install -y epel-release

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y \
  python3 python3-pip \
  git git-lfs curl jq parallel tmux \
  docker podman
git lfs install
# Optional: enable Docker service
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Tip: The Python venv module is built-in. On Debian/Ubuntu you specifically need the python3-venv package (installed above).

Step 1: Standardize Your Lab With One Entry Point

Create a single executable, “labctl”, that handles environment setup, data pulls, runs, and reports.

Project skeleton:

ai-lab/
├── labctl
├── requirements.txt
├── data/             # raw and processed data (tracked with git-lfs)
├── experiments/      # run logs, artifacts
├── models/           # model checkpoints/artifacts (lfs)
├── scripts/          # helper scripts
└── reports/          # auto-generated metrics

Sample requirements.txt (edit as needed):

numpy
pandas
scikit-learn
jinja2

labctl (make it executable: chmod +x labctl):

#!/usr/bin/env bash
set -euo pipefail

LAB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV="$LAB_DIR/.venv"
LOGS="$LAB_DIR/experiments"
DATA="$LAB_DIR/data"
REPORTS="$LAB_DIR/reports"

usage() {
  cat <<EOF
Usage: ./labctl <command>

Commands:
  setup         Create venv, install deps, pre-flight checks
  data          Pull/update datasets (git-lfs)
  run           Run batch experiments (GNU parallel)
  eval          Aggregate metrics (jq)
  report        Build markdown report
  clean         Remove venv and caches
EOF
}

cmd_setup() {
  mkdir -p "$LOGS" "$DATA" "$REPORTS"
  # venv
  if [[ ! -d "$VENV" ]]; then
    python3 -m venv "$VENV"
  fi
  # shellcheck disable=SC1090
  source "$VENV/bin/activate"
  python -m pip install --upgrade pip setuptools wheel
  if [[ -f "$LAB_DIR/requirements.txt" ]]; then
    pip install -r "$LAB_DIR/requirements.txt"
  fi
  # Silence GNU parallel citation prompt
  yes | parallel --citation >/dev/null 2>&1 || true
  echo "[setup] Done"
}

cmd_data() {
  pushd "$LAB_DIR" >/dev/null
  git lfs install
  git lfs pull || true
  popd >/dev/null
  echo "[data] Pulled large files via git-lfs (if any)"
}

# Example: simulate an experiment that writes metrics.json
run_one() {
  local seed="$1"
  local out="$LOGS/seed_${seed}"
  mkdir -p "$out"
  # In real life you'd call python train.py --seed "$seed" ...
  python - <<"PY" "$seed" "$out"
import json, os, random, sys, time
seed = int(sys.argv[1]); out = sys.argv[2]
random.seed(seed); time.sleep(0.2)
metrics = {
  "seed": seed,
  "accuracy": round(0.80 + random.random()*0.15, 4),
  "loss": round(0.5 + random.random()*0.5, 4),
  "timestamp": time.time()
}
with open(os.path.join(out, "metrics.json"), "w") as f:
    json.dump(metrics, f)
print(f"Wrote {out}/metrics.json")
PY
}

export -f run_one

cmd_run() {
  # shellcheck disable=SC1090
  source "$VENV/bin/activate"
  mkdir -p "$LOGS"
  SEEDS="${SEEDS:-0 1 2 3 4 5 6 7}"
  echo "$SEEDS" | tr ' ' '\n' | parallel -j "$(nproc)" run_one {}
  echo "[run] Completed batch"
}

cmd_eval() {
  mkdir -p "$REPORTS"
  jq -s '
    map(select(.accuracy != null)) as $m |
    {
      n: ($m|length),
      avg_accuracy: ($m|map(.accuracy)|add/($m|length)),
      avg_loss: ($m|map(.loss)|add/($m|length))
    }' "$LOGS"/*/metrics.json > "$REPORTS/summary.json"
  echo "[eval] Wrote $REPORTS/summary.json"
}

cmd_report() {
  if [[ ! -f "$REPORTS/summary.json" ]]; then
    echo "No summary.json. Run ./labctl eval first." >&2; exit 1
  fi
  {
    echo "# AI Lab Report"
    echo
    echo "Generated: $(date -Is)"
    echo
    echo "## Summary"
    jq -r '"- Runs: \(.n)\n- Avg Accuracy: \(.avg_accuracy)\n- Avg Loss: \(.avg_loss)"' "$REPORTS/summary.json"
    echo
    echo "## Per-run Metrics"
    for f in "$LOGS"/*/metrics.json; do
      jq -r '"- \(.seed): acc=\(.accuracy) loss=\(.loss)"' "$f"
    done
  } > "$REPORTS/latest.md"
  echo "[report] Wrote $REPORTS/latest.md"
}

cmd_clean() {
  rm -rf "$VENV" "$LOGS"/* "$REPORTS"/*
  echo "[clean] Removed venv and artifacts"
}

main() {
  cmd="${1:-}"; shift || true
  case "${cmd:-}" in
    setup) cmd_setup "$@";;
    data) cmd_data "$@";;
    run) cmd_run "$@";;
    eval) cmd_eval "$@";;
    report) cmd_report "$@";;
    clean) cmd_clean "$@";;
    *) usage; exit 1;;
  esac
}
main "$@"

Try it:

./labctl setup
./labctl data
./labctl run
./labctl eval
./labctl report
cat reports/latest.md

Result: a clean venv, batched runs across all cores, JSON metrics, and a human-readable report.

Step 2: Version Datasets and Models With Git LFS

Keep large binaries out of normal Git history while staying reproducible.

Initialize in your repo:

git lfs install
git lfs track "data/**/*"
git lfs track "models/**/*"
git add .gitattributes
git commit -m "Track data/ and models/ with Git LFS"

Pulling the latest (wired into labctl data):

./labctl data

Use remotes (GitHub/GitLab) that support LFS or self-host a Git server with LFS.

Step 3: Batch Experiments With GNU parallel and tmux

Run many variants at once cleanly.

  • GNU parallel fans out your seeds/hyperparams.

  • tmux keeps sessions alive on SSH disconnects.

Example: Running seeds in a detached tmux session:

tmux new -d -s labrun './labctl run && ./labctl eval && ./labctl report'
tmux attach -t labrun   # reattach any time

Override seeds and concurrency as env vars:

SEEDS="0 1 2 3 4 5 6 7 8 9" ./labctl run

Tip: For GPU-aware scheduling, you can extend run_one to pick a free CUDA device by parsing nvidia-smi, or integrate a job scheduler (Slurm) later. Bash plays nicely with both.

Step 4: Containerize for Parity (Docker or Podman)

Containers ensure your lab runs the same on laptops, workstations, or CI.

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN chmod +x labctl
ENTRYPOINT ["./labctl"]

Build and run (Docker):

docker build -t ai-lab:latest .
docker run --rm -it -v "$PWD:/app" ai-lab:latest setup
docker run --rm -it -v "$PWD:/app" ai-lab:latest run

Build and run (Podman):

podman build -t ai-lab:latest .
podman run --rm -it -v "$PWD:/app" ai-lab:latest setup
podman run --rm -it -v "$PWD:/app" ai-lab:latest run

Mounting the project directory keeps artifacts and reports on your host.

Step 5: Continuous Evaluation and Reporting

Make your lab self-check at night and email or commit reports.

Cron example (runs daily at 01:30):

crontab -e
# Add:
30 1 * * * cd /path/to/ai-lab && ./labctl run && ./labctl eval && ./labctl report >> cron.log 2>&1

Optionally, push reports to a docs branch:

git checkout -B reports
git add reports/latest.md reports/summary.json
git commit -m "Nightly lab report"
git push -u origin reports

You now have living documentation of your lab’s health and progress.

Real-World Use Cases

  • Research teams: Reproduce baselines across OSes with one command.

  • MLEs: Spin up CI that runs the same labctl steps in containers.

  • Academia: Hand students a starter lab with zero “it works on my machine” issues.

  • Regulated environments: Scripted, logged, and auditable end-to-end.

Common Extensions

  • Hyperparameter grids: Pipe a CSV of params to parallel and pass as args.

  • Remote storage: Sync data/ and models/ to S3/GCS with rclone or DVC.

  • GPU isolation: Wrap runs with CUDA_VISIBLE_DEVICES selection logic in Bash.

  • Notebooks: Pre-generate notebooks from templates with Jinja2 and run with papermill in CI.

Conclusion and Next Steps

Automation is leverage. With a single Bash entrypoint you’ve made your AI lab reproducible, scalable, and observable. Now:

1) Drop the labctl skeleton into your current project.
2) Replace the simulated run with your actual train/eval scripts.
3) Containerize and hook into CI or cron.

When you’re ready to go deeper, add GPU scheduling, cloud remotes, and CI badges to your reports. Your future self (and teammates) will thank you.

Related posts