Posted on
• Artificial Intelligence

Artificial Intelligence GitLab CI Pipelines

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence GitLab CI Pipelines: From Notebook to Repeatable Linux Automation

If you’ve ever heard “it works on my machine” right before a model demo fails, you’ve already met the core problem AI teams face: reproducibility. AI stacks are dependency-heavy, data-hungry, and GPU-sensitive. The fix is boring—in the best way possible: codify the workflow with GitLab CI so you can go from notebook to repeatable pipeline on any Linux box or runner.

This article shows you how to stand up a robust, Bash-friendly GitLab CI pipeline for AI projects, with practical YAML and shell snippets, GPU notes, and distro-specific install steps for apt, dnf, and zypper.

Why GitLab CI for AI?

  • Repeatability: Pin environments, cache models, and make training deterministic.

  • Speed: CI caches and parallel stages slash iteration time.

  • Traceability: Every run is logged, versioned, and easy to roll back.

  • Scale: Add runners (CPU/GPU) as your workloads grow.

  • Governance: Secrets, approvals, and audit trails are first-class citizens.

What you’ll build

  • A CI pipeline that lints and tests Python, then trains a model (CPU or GPU), and packages artifacts.

  • Smart caching for Python deps and model weights.

  • Optional GPU acceleration via NVIDIA Container Toolkit.

  • Bash-first, minimal tooling, distro-agnostic instructions.

Prerequisites and installation

You need:

  • A Linux machine or VM with sudo

  • A GitLab project (SaaS or self-managed)

  • Docker Engine (or Podman for basic CPU jobs)

  • Optional: An NVIDIA GPU for accelerated training

Base tools

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y git curl python3 python3-venv python3-pip jq

Fedora/RHEL/CentOS Stream (dnf):

sudo dnf install -y git curl python3 python3-venv python3-pip jq

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y git curl python3 python3-venv python3-pip jq

Container runtime (Docker or Podman)

Docker Engine

  • apt:
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
  • dnf:
sudo dnf install -y moby-engine
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker
  • zypper:
sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Podman (CPU-only pipelines without Docker-in-Docker)

  • apt:
sudo apt update
sudo apt install -y podman
  • dnf:
sudo dnf install -y podman
  • zypper:
sudo zypper install -y podman

GitLab Runner

  • apt:
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt install -y gitlab-runner
  • dnf:
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash
sudo dnf install -y gitlab-runner
sudo systemctl enable --now gitlab-runner
  • zypper:
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash
sudo zypper install -y gitlab-runner
sudo systemctl enable --now gitlab-runner

Register your runner (Docker executor is the most flexible):

sudo gitlab-runner register
# URL: https://gitlab.com/ or your self-managed URL
# Registration token: from your project/group
# Description: ai-docker-runner
# Tags: cpu,linux (add gpu later if needed)
# Executor: docker
# Default image: python:3.11-slim

Optional persistent cache volume (improves speed):

  • Edit /etc/gitlab-runner/config.toml and add to your runner’s docker config:
volumes = ["/cache"]
  • Restart:
sudo systemctl restart gitlab-runner

Optional: NVIDIA GPU support for containers

Install NVIDIA Container Toolkit so Docker can pass the GPU into containers.

  • apt:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
 | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
 | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
  • dnf:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
 | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
  • zypper:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo \
 | sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Give your GitLab Runner access to GPUs by setting gpus = "all":

  • Edit /etc/gitlab-runner/config.toml for a “gpu” runner:
[[runners]]
  name = "gpu-docker"
  executor = "docker"
  tags = ["gpu","linux"]
  [runners.docker]
    image = "pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime"
    gpus = "all"
    volumes = ["/cache"]
  • Restart:
sudo systemctl restart gitlab-runner

Repository layout (minimal)

.
├── .gitlab-ci.yml
├── requirements.txt
├── train.py
└── inference.py

requirements.txt:

numpy
scikit-learn
joblib

train.py (CPU by default; uses GPU if PyTorch+cudevices are present, e.g., in the GPU job image):

#!/usr/bin/env python3
import os, json, time, pathlib
from pathlib import Path

def save_meta(path, meta: dict):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(meta, indent=2))

def cpu_train():
    from sklearn.datasets import load_iris
    from sklearn.linear_model import LogisticRegression
    from sklearn.model_selection import train_test_split
    import joblib

    X, y = load_iris(return_X_y=True, as_frame=False)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)
    m = LogisticRegression(max_iter=200)
    m.fit(Xtr, ytr)
    acc = m.score(Xte, yte)
    Path("artifacts").mkdir(exist_ok=True)
    joblib.dump(m, "artifacts/model.joblib")
    save_meta(Path("artifacts/metrics.json"), {"accuracy": acc, "device": "cpu"})
    print(f"CPU model accuracy: {acc:.4f}")

def gpu_train():
    import torch

    device = "cuda" if torch.cuda.is_available() else "cpu"
    x = torch.randn(1024, 512, device=device)
    w = torch.randn(512, 256, device=device, requires_grad=True)
    y = x @ w
    loss = y.pow(2).mean()
    loss.backward()
    torch.save({"W": w.detach().cpu()}, "artifacts/model.pt")
    save_meta(Path("artifacts/metrics.json"), {"loss": float(loss.item()), "device": device})
    print(f"GPU path ran on: {device}, loss={loss.item():.6f}")

def main():
    t0 = time.time()
    Path("artifacts").mkdir(exist_ok=True)
    try:
        import torch  # noqa
        gpu_train()
    except Exception as e:
        print(f"Falling back to CPU training: {e}")
        cpu_train()
    print(f"Done in {time.time() - t0:.2f}s")

if __name__ == "__main__":
    main()

inference.py (toy CLI for packaged model):

#!/usr/bin/env python3
import sys, joblib, json
from pathlib import Path
m = joblib.load("artifacts/model.joblib")
print(json.dumps({"pred": m.predict([[float(v) for v in sys.argv[1:5]]]).tolist()}))

The GitLab CI pipeline (.gitlab-ci.yml)

This pipeline:

  • Caches Python and model caches between jobs.

  • Lints/tests quickly in a slim image.

  • Trains on CPU by default and optionally on GPU via a separate job.

  • Publishes model artifacts for later promotion.

stages:
  - verify
  - train
  - package

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  HF_HOME: "$CI_PROJECT_DIR/.cache/huggingface"
  PIP_NO_INPUT: "1"
  PIP_DISABLE_PIP_VERSION_CHECK: "1"
  PYTHONDONTWRITEBYTECODE: "1"

cache:
  key: "pip-${CI_COMMIT_REF_SLUG}"
  paths:
    - .cache/pip
    - .cache/huggingface
  policy: pull-push

verify:
  stage: verify
  image: python:3.11-slim
  before_script:
    - python -V
    - python -m pip install --upgrade pip
    - pip install flake8 pytest -r requirements.txt
  script:
    - flake8 .
    - pytest -q || true  # optional if no tests yet
  rules:
    - if: $CI_COMMIT_BRANCH

train:cpu:
  stage: train
  image: python:3.11-slim
  needs: ["verify"]
  before_script:
    - python -m pip install --upgrade pip
    - pip install -r requirements.txt
  script:
    - python train.py
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - artifacts/
  rules:
    - if: $CI_COMMIT_BRANCH
  tags:
    - cpu

# Optional GPU job; requires a runner tagged "gpu" and NVIDIA toolkit
train:gpu:
  stage: train
  image: pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime
  needs: ["verify"]
  before_script:
    - pip install -r requirements.txt
  script:
    - python train.py
    - python -c "import torch; print('CUDA?', torch.cuda.is_available())"
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - artifacts/
  rules:
    - if: '$ENABLE_GPU == "1"'
  tags:
    - gpu

# Create a signed tarball of artifacts for promotion
package:
  stage: package
  image: bash:latest
  needs: ["train:cpu"]
  script:
    - mkdir -p dist
    - tar czf dist/model-${CI_COMMIT_SHORT_SHA}.tar.gz artifacts
    - sha256sum dist/model-${CI_COMMIT_SHORT_SHA}.tar.gz | tee dist/SHA256SUMS
  artifacts:
    expire_in: 14 days
    paths:
      - dist/
  rules:
    - if: $CI_COMMIT_TAG

Notes:

  • Enable GPU path per pipeline by setting CI/CD variable ENABLE_GPU=1.

  • For private model hubs, store tokens as masked variables (e.g., HF_TOKEN) and let your training scripts read from os.environ without echoing secrets.

4 actionable practices that pay off immediately

1) Make pipelines hermetic and cache aggressively

  • Pin images (python:3.11-slim, pytorch with a specific CUDA tag).

  • Cache Python wheels and model weights:

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  HF_HOME: "$CI_PROJECT_DIR/.cache/huggingface"
cache:
  paths: [.cache/pip, .cache/huggingface]
  • In Bash scripts, fail fast for reliability:
set -euo pipefail
IFS=$'\n\t'

2) Split fast feedback from heavy compute

  • Keep verify jobs <2 minutes by using slim images and avoiding heavyweight installs.

  • Run train jobs on tagged runners (cpu, gpu) so you don’t starve basic CI tasks.

3) Treat models as artifacts

  • Always save model files and a small metrics.json:
python - <<'PY'
import json, os, pathlib
pathlib.Path("artifacts").mkdir(exist_ok=True)
json.dump({"git_sha": os.environ.get("CI_COMMIT_SHA")}, open("artifacts/build_meta.json","w"))
PY
  • Use artifacts to promote models through environments without rebuilding.

4) Secure your AI supply chain

  • Store tokens (HF_TOKEN, WANDB_API_KEY) as protected, masked variables.

  • Prefer non-root containers and drop capabilities by default.

  • Consider dependency proxy and private PyPI to speed and control package sources.

Optional: containerize your inference

If you want a tiny image for inference (CPU), add a Dockerfile and build on tags. This assumes a runner with Docker available.

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY artifacts/ artifacts/
RUN pip install --no-cache-dir joblib scikit-learn
COPY inference.py .
ENTRYPOINT ["python","inference.py"]

CI job (add to .gitlab-ci.yml):

containerize:
  stage: package
  image: docker:26
  services: ["docker:26-dind"]
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  needs: ["train:cpu"]
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" --password-stdin
    - docker build -t "$CI_REGISTRY_IMAGE:$(echo $CI_COMMIT_TAG | tr / -)" .
    - docker push "$CI_REGISTRY_IMAGE:$(echo $CI_COMMIT_TAG | tr / -)"
  rules:
    - if: $CI_COMMIT_TAG

Real-world tip: warm up the GPU runner

First job on a fresh GPU runner can be slow due to cold caches. Add a short warm-up step to your GPU job:

python - <<'PY'
import torch
x = torch.randn(4096,4096, device='cuda')
y = x @ x
torch.cuda.synchronize()
print("Warmed up CUDA.")
PY

Conclusion and next step (CTA)

AI is fragile until you script it. With a GitLab CI pipeline, you make your model lifecycle boringly reliable: lint, test, train, and package on any Linux runner—with or without GPUs.

Your next steps: 1) Install Docker and GitLab Runner (apt/dnf/zypper commands above). 2) Register a CPU runner; optionally add a GPU runner with NVIDIA toolkit. 3) Drop the provided .gitlab-ci.yml, train.py, and requirements.txt into your repo. 4) Push a branch, watch the pipeline go green, then tag a release to package.

When you’re ready, extend this with dataset versioning (DVC), experiment tracking (MLflow/W&B), and container scanning. If you want a follow-up post on multi-arch images, spot/preemptible GPU runners, or Hugging Face cache best practices, say the word.

Related posts