Posted on
• Artificial Intelligence

Artificial Intelligence Cloud Infrastructure with Bash

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence Cloud Infrastructure with Bash

Want to spin up GPU-ready AI inference in the cloud without drowning in layers of tooling? Bash is still the most powerful glue you own. In this guide, you’ll learn how to build and operate AI cloud infrastructure using simple, auditable Bash scripts that run anywhere Linux runs.

We’ll cover why Bash is the right foundation, what to install, and 3–5 actionable steps you can copy-paste today to provision, bootstrap, and run an AI inference server (vLLM) on GPU-backed cloud instances. All package installation snippets include apt, dnf, and zypper variants.

Why Bash for AI cloud?

  • Ubiquity and zero lock-in: Bash is on every Linux server. No vendor-specific DSL or heavy agent required.

  • Composable: Orchestrate cloud-init, SSH, containers, object storage, and monitoring in a few lines.

  • Transparent ops: Bash makes your infra changes explicit and easy to audit.

  • Works with any cloud: If you can get an IP and an SSH key, you can manage it with Bash.

Prerequisites

  • Cloud VM(s), ideally GPU-backed with a vendor GPU image (e.g., NVIDIA drivers pre-installed).

  • Your SSH key on the instances.

  • A Linux workstation or jump host with Bash.

Below you’ll find installation commands for required tools. Use the matching block for your distro’s package manager.

1) Base CLI tools

  • curl, jq: HTTP + JSON parsing for automation

  • git: version your scripts

  • OpenSSH client: remote execution

  • rclone: object storage sync (S3/GCS/Azure/etc.)

apt:

sudo apt update
sudo apt install -y curl jq git openssh-client rclone

dnf:

sudo dnf install -y curl jq git openssh-clients rclone

zypper:

sudo zypper refresh
sudo zypper install -y curl jq git openssh rclone

2) Container runtime (Docker Engine)

Containerizing AI workloads makes dependency and GPU runtime setup predictable. Docker is a widely supported default (Fedora uses the moby-engine packages).

apt (Ubuntu/Debian):

sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

dnf (Fedora/RHEL via moby):

sudo dnf install -y moby-engine
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

zypper (openSUSE/SLES):

sudo zypper install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

Log out/in to pick up docker group membership.

Note: If you prefer Podman, it also works for CPU-only. GPU support with Podman requires additional configuration; Docker tends to be simpler for NVIDIA GPUs.

3) NVIDIA Container Toolkit (for GPU passthrough)

Needed to expose GPUs to containers via --gpus all. If your cloud image already includes this, you can skip.

apt (Ubuntu/Debian):

distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list >/dev/null
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

dnf (Fedora/RHEL):

distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --yes --dearmor -o /etc/pki/rpm-gpg/RPM-GPG-KEY-nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo >/dev/null
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

zypper (openSUSE/SLES):

distribution=$(. /etc/os-release; echo ${ID}${VERSION_ID})
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | \
  sudo tee /etc/zypp/repos.d/nvidia-container-toolkit.repo >/dev/null
sudo zypper refresh
sudo zypper install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility in containers:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If you see your GPU(s), you’re ready.

Core: 4 actionable steps to stand up AI infra with Bash

Step 1 — Bootstrap any cloud VM over SSH with a single Bash script

Use one script that detects the package manager, installs your basics, enables Docker, and preps directories.

#!/usr/bin/env bash
set -euo pipefail

# Usage: ./bootstrap-node.sh [user@]host
HOST="${1:?Usage: ./bootstrap-node.sh [user@]host}"

read -r -d '' REMOTE_SCRIPT <<'EOS'
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive

detector() {
  if command -v apt >/dev/null 2>&1; then echo apt
  elif command -v dnf >/dev/null 2>&1; then echo dnf
  elif command -v zypper >/dev/null 2>&1; then echo zypper
  else echo "unsupported"; exit 1; fi
}

pm=$(detector)
case "$pm" in
  apt)
    sudo apt update
    sudo apt install -y curl jq git openssh-client rclone docker.io
    sudo systemctl enable --now docker
    ;;
  dnf)
    sudo dnf install -y curl jq git openssh-clients rclone moby-engine
    sudo systemctl enable --now docker
    ;;
  zypper)
    sudo zypper refresh
    sudo zypper install -y curl jq git openssh rclone docker
    sudo systemctl enable --now docker
    ;;
esac

sudo usermod -aG docker "$USER" || true
sudo mkdir -p /opt/models
sudo chmod 777 /opt/models
echo "OK"
EOS

ssh -o StrictHostKeyChecking=accept-new "$HOST" "bash -s" <<<"$REMOTE_SCRIPT"
echo "Bootstrap complete on $HOST"

This gives you a repeatable, cloud-agnostic baseline for any instance.

Tip: You can parallelize across many hosts by backgrounding the SSH calls and waiting.

Step 2 — Optional: Cloud-init for day-0 provisioning

When you create instances, pass user-data so they come up ready (even before you SSH in). For Ubuntu/Debian images:

#cloud-config
users:
  - name: ai
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: [docker]
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-rsa AAAA...yourkey
package_update: true
packages:
  - curl
  - jq
  - git
  - rclone
  - docker.io
runcmd:
  - [systemctl, enable, --now, docker]
  - [bash, -lc, "mkdir -p /opt/models && chmod 777 /opt/models"]

On RHEL/Fedora/openSUSE images, swap packages accordingly (moby-engine or docker, openssh-clients vs openssh).

Step 3 — Run a GPU-backed AI inference server (vLLM) in one command

The vLLM project offers a fast OpenAI-compatible API server. Assuming you have an HF token and an NVIDIA GPU:

export HF_TOKEN="your_huggingface_token"
export MODEL="meta-llama/Llama-3.1-8B-Instruct"

docker pull vllm/vllm-openai:latest

docker run -d --name vllm \
  --gpus all \
  -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -e HUGGINGFACE_HUB_CACHE=/models \
  -v /opt/models:/models \
  vllm/vllm-openai:latest \
  --model "$MODEL" --max-model-len 4096

Test the endpoint:

curl -s http://SERVER_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Say hello from Bash"}],
    "max_tokens": 32
  }' | jq .

Real-world note: Mounting /opt/models caches weights between restarts and across deployments.

Step 4 — Scale out with simple Bash fan-out and object storage syncing

  • Keep a hosts.txt with one [user@]ip-or-dns per line.

  • Fan-out commands with SSH in parallel background jobs.

  • Share models via object storage using rclone for fast, reproducible rollouts.

Example: Start vLLM on multiple nodes:

#!/usr/bin/env bash
set -euo pipefail

HOSTS_FILE="${1:-hosts.txt}"
HF_TOKEN="${HF_TOKEN:?set HF_TOKEN in env}"
MODEL="${MODEL:-meta-llama/Llama-3.1-8B-Instruct}"

while read -r HOST; do
  [ -z "$HOST" ] && continue
  {
    ssh -o StrictHostKeyChecking=accept-new "$HOST" bash -lc " \
      docker pull vllm/vllm-openai:latest && \
      docker stop vllm 2>/dev/null || true && docker rm vllm 2>/dev/null || true && \
      docker run -d --name vllm --gpus all -p 8000:8000 \
        -e HF_TOKEN='$HF_TOKEN' -e HUGGINGFACE_HUB_CACHE=/models \
        -v /opt/models:/models \
        vllm/vllm-openai:latest \
        --model '$MODEL' --max-model-len 4096 \
    "
    echo "Started vLLM on $HOST"
  } &
done < "$HOSTS_FILE"
wait
echo "All hosts done."

Optional: Seed model cache via rclone (example for S3-compatible storage): 1) Configure once:

rclone config
# create remote named "ai-objects"

2) Sync a models directory to each node:

ssh "$HOST" "mkdir -p /opt/models"
rclone sync ai-objects:models /opt/models

This keeps cold-starts fast and bills low by avoiding repeated downloads.

Troubleshooting quick hits

  • docker: permission denied — Add your user to the docker group and relogin: sudo usermod -aG docker $USER.

  • GPU not visible in containers — Ensure NVIDIA drivers on host, install NVIDIA Container Toolkit, and restart Docker. Test with nvidia-smi inside a container.

  • Slow model downloads — Pre-seed /opt/models using rclone/s3 or bake a custom image with the model cache.

Conclusion and next steps

Bash remains the backbone of reliable, portable automation. With a few scripts, you can:

  • Bootstrap cloud VMs across providers

  • Enable GPU containers

  • Run a production-grade AI inference server

  • Scale out predictably with simple, auditable code

Your next step:

  • Save these snippets into a repo (e.g., infra/bootstrap-node.sh, infra/scale-out.sh).

  • Create a hosts.txt for your environment.

  • Run the bootstrap, then deploy vLLM, and hit your new AI endpoint.

Have a specific provider, cluster pattern, or CI/CD you want to wire in? Build on these scripts, or reach out with your requirements and I’ll tailor a Bash-first workflow for your stack.

Related posts