Posted on
• Artificial Intelligence

Artificial Intelligence DevSecOps on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence DevSecOps on Linux: A Bash-First, Hands-On Guide

AI is moving fast—and so are its risks. From poisoned datasets and vulnerable Python packages to insecure containers and unsigned model files, an ML pipeline has a sprawling attack surface. The good news: if you build on Linux, you already have everything you need to add DevSecOps discipline with simple, scriptable tools.

This guide shows how to stand up a practical AI DevSecOps workflow on Linux using Bash and widely available, open tooling. You’ll:

  • Set up a secure, reproducible environment

  • Scan code, dependencies, and containers early

  • Generate SBOMs, sign artifacts, and verify provenance

  • Automate it all locally and in CI

The value: fewer surprises in production, faster incident response, and a clear “paper trail” for audit and compliance—without slowing down data scientists.

Why AI DevSecOps on Linux?

  • AI attacks target the supply chain. Poisoned data, malicious pip packages, hidden backdoors in containers, and unsafe model serialization (e.g., pickled models) are all real-world threats.

  • Linux is the common denominator. It’s what powers most CI/CD, containers, and GPU stacks. Its package managers and shell tools make automation natural.

  • DevSecOps ≠ heavyweight. CLI-first scanners, pre-commit hooks, rootless containers, and cryptographic signing can be stitched together with Bash—fast.

Prerequisites: Install Once, Automate Forever

We’ll use:

  • podman (rootless containers)

  • python3 + venv + pipx (isolated Python and CLI tools)

  • pre-commit, bandit, semgrep, detect-secrets, pip-audit (code/dependency/security)

  • trivy (container vuln/SBOM)

  • cosign (sign/verify images and blobs)

  • shellcheck (lint Bash)

  • jq, make, curl (helpers)

Choose your package manager and run:

Debian/Ubuntu (apt)

sudo apt update
sudo apt install -y git podman python3 python3-venv python3-pip pipx jq shellcheck cosign curl make
pipx ensurepath

Install Trivy (apt via official repo):

sudo apt install -y wget gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --dearmor -o /usr/share/keyrings/trivy.gpg
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt update
sudo apt install -y trivy

Fedora/RHEL/CentOS (dnf)

sudo dnf -y install git podman python3 python3-virtualenv python3-pip pipx jq ShellCheck cosign curl make
pipx ensurepath

Install Trivy (dnf via official repo; replace “$(rpm -E %fedora)” with your version if needed, e.g., 39):

sudo dnf -y install dnf-plugins-core
sudo rpm --import https://aquasecurity.github.io/trivy-repo/rpm/public.key
sudo dnf config-manager --add-repo https://aquasecurity.github.io/trivy-repo/rpm/releases/$(rpm -E %fedora)/$basearch/
sudo dnf -y install trivy

openSUSE Leap/Tumbleweed (zypper)

sudo zypper refresh
sudo zypper install -y git podman python3 python3-virtualenv python3-pip pipx jq ShellCheck cosign curl make
pipx ensurepath

Install Trivy (zypper via official repo; replace “15” with your Leap version if needed):

sudo rpm --import https://aquasecurity.github.io/trivy-repo/rpm/public.key
sudo zypper ar -f https://aquasecurity.github.io/trivy-repo/rpm/releases/15/$basearch trivy
sudo zypper refresh
sudo zypper install -y trivy

Note: If pipx isn’t in your repo, install it via pip:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Now install CLI tools into isolated shims:

for pkg in pre-commit bandit semgrep detect-secrets pip-audit; do
  pipx install "$pkg"
done

Step 1: Make Your AI Environment Reproducible (and Safer)

  • Isolate Python environments (no sudo pip).

  • Pin dependencies to reduce supply-chain drift.

  • Prefer rootless containers (podman) for running notebooks, training jobs, and inference services.

Create and lock a Python environment:

python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Example minimal requirements.txt (pin versions):

numpy==1.26.4
pandas==2.2.2
scikit-learn==1.5.1
onnx==1.16.1

Build a non-root container image for training/inference:

cat > Containerfile <<'EOF'
FROM python:3.11-slim
# Create non-root user
RUN useradd -m -u 10001 appuser
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
USER appuser
CMD ["python", "serve.py"]
EOF

podman build -t ai-app:0.1 -f Containerfile .

Why this matters:

  • Non-root containers limit blast radius.

  • Pinning versions reduces “it worked yesterday” issues and aids reproducibility.

  • A Containerfile makes your environment portable across dev/CI/prod.

Step 2: Shift Left with Local, Fast Security Checks

Wire security into your edit/commit loop with pre-commit hooks. We’ll run:

  • shellcheck for Bash

  • bandit for Python security smells

  • semgrep for patterns/anti-patterns

  • detect-secrets to catch secrets before they land

  • pip-audit for Python vulns and SBOM output

Initialize:

pre-commit install

Add .pre-commit-config.yaml:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-merge-conflict
      - id: end-of-file-fixer
      - id: trailing-whitespace

  - repo: local
    hooks:
      - id: shellcheck
        name: shellcheck
        entry: shellcheck
        language: system
        types: [shell]

      - id: bandit
        name: bandit
        entry: bandit -q -r .
        language: system
        types: [python]

      - id: semgrep
        name: semgrep
        entry: semgrep --error --config=p/ci --exclude-dir .venv
        language: system
        types: [python]

      - id: detect-secrets
        name: detect-secrets
        entry: detect-secrets-hook
        language: system
        pass_filenames: true

Seed a secrets baseline (commit this once and review it!):

detect-secrets scan > .secrets.baseline
git add .secrets.baseline

Audit and create a Python SBOM:

pip-audit -r requirements.txt -f cyclonedx -o sbom-python.json

Pro tip: Run all hooks manually before a big PR:

pre-commit run --all-files

Step 3: Scan, SBOM, and Sign Your Containers and Models

Scan your container for known vulnerabilities and export an SBOM (CycloneDX):

trivy image --exit-code 1 --severity CRITICAL,HIGH ai-app:0.1
trivy image --format cyclonedx --output sbom-image.json ai-app:0.1

Sign the container image with cosign (key-pair example):

cosign generate-key-pair
# signs to the registry-attached signature (for local, compatible registries)
COSIGN_PASSWORD="" cosign sign --key cosign.key localhost/ai-app:0.1
# verify
cosign verify --key cosign.pub localhost/ai-app:0.1

Sign your model artifact (e.g., ONNX) as a blob:

python export_model.py --out model.onnx
cosign sign-blob --key cosign.key model.onnx > model.onnx.sig
cosign verify-blob --key cosign.pub --signature model.onnx.sig model.onnx

Why this matters:

  • Scans catch known vulns before prod.

  • SBOMs clarify “what’s inside” for incident response and compliance.

  • Signing proves integrity and origin of your images and model files.

Real-world example:

  • A team accidentally included a dev-only “latest” base image with OpenSSL CVEs. Trivy blocked the merge; switching to a pinned, patched base image fixed it in minutes—before any deploy.

Step 4: One-Command CI Script (Also Great Locally)

Bundle essential checks into a single Bash script you can run on your laptop and in CI. Fail on high/critical issues; publish SBOMs; sign artifacts when building release candidates.

Create ci.sh:

#!/usr/bin/env bash
set -euo pipefail

# 1) Lint shell
echo "[*] shellcheck"
shellcheck -x ci.sh || { echo "Shell lint failed"; exit 1; }

# 2) Python static + secrets
echo "[*] bandit"
bandit -q -r . || { echo "Bandit failed"; exit 1; }

echo "[*] semgrep"
semgrep --error --config=p/ci --exclude-dir .venv || { echo "Semgrep failed"; exit 1; }

echo "[*] detect-secrets"
detect-secrets scan --baseline .secrets.baseline --all-files || { echo "Secrets check failed"; exit 1; }

# 3) Python deps: audit + SBOM
echo "[*] pip-audit + SBOM"
pip-audit -r requirements.txt -f cyclonedx -o sbom-python.json || { echo "Dependency audit failed"; exit 1; }

# 4) Build, scan, and SBOM container
echo "[*] build container"
podman build -t ai-app:ci -f Containerfile .

echo "[*] trivy scan"
trivy image --exit-code 1 --severity CRITICAL,HIGH ai-app:ci

echo "[*] trivy SBOM (CycloneDX)"
trivy image --format cyclonedx --output sbom-image.json ai-app:ci

echo "[*] all checks passed"

Run it:

chmod +x ci.sh
./ci.sh

Optional: schedule weekly re-scans on the same tag (new CVEs appear daily). You can use cron:

( crontab -l 2>/dev/null; echo '30 3 * * 1 podman pull python:3.11-slim && trivy image --severity CRITICAL,HIGH python:3.11-slim || true' ) | crontab -

Or a user systemd timer for richer logging.

Putting It All Together (3–5 Key Takeaways)

1) Make it reproducible: pin Python deps, use venvs, and run rootless containers.
2) Shift left: pre-commit hooks catch issues before code ever reaches CI.
3) Scan everything: code, dependencies, containers; generate SBOMs for clarity.
4) Prove integrity: sign containers and model artifacts with cosign, verify before deploy.
5) Automate: a single Bash script run locally and in CI keeps drift small and velocity high.

Conclusion and Call To Action

You don’t need a heavy platform to practice DevSecOps for AI. On Linux, a handful of CLI tools—wired together with Bash—gives you immediate wins: fewer vulnerabilities, signed artifacts, and a clear, repeatable process your team can trust.

Your next steps:

  • Install the prerequisites for your distro (apt/dnf/zypper above).

  • Drop in the pre-commit config and run pre-commit install.

  • Add ci.sh to your repo and make it a required CI job.

  • Start signing: use cosign for your images and model files.

Small, consistent steps beat grand rewrites. Start today; your future self (and your incident pager) will thank you.

Related posts