Posted on
• Artificial Intelligence

Offline Artificial Intelligence Workflows

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Offline Artificial Intelligence Workflows on Linux: Build, Run, and Ship Models Without the Internet

Air-gapped servers. Strict privacy policies. Flaky site links. If you depend on the cloud for every model and dependency, your AI pipeline can grind to a halt at the worst time. Offline AI workflows change that—letting you build, run, and ship intelligent systems entirely without a live internet connection.

This guide explains why offline AI matters and how to do it on Linux with Bash-first commands. You’ll learn a practical workflow to stage dependencies, cache models, run inference, and containerize for repeatability—complete with installation instructions for apt, dnf, and zypper.

Why offline AI is worth it

  • Privacy and compliance: Keep data and model artifacts inside the org boundary.

  • Reliability and latency: No waiting on the network; predictable performance at the edge.

  • Reproducibility: Same bits, same results—today and six months from now.

  • Cost control: Fewer surprise egress/ingress fees.

“Offline” usually means:

  • No outbound network at runtime.

  • All packages, models, and datasets are pre-fetched and verified.

  • Builds and runs remain deterministic without calling home.

Quick setup: system packages you’ll likely need

Install common build tools, Python, Git LFS, downloaders, and (optionally) containers.

Apt (Debian/Ubuntu):

sudo apt update
sudo apt install -y \
  git git-lfs python3 python3-venv python3-pip \
  build-essential cmake gcc g++ make \
  wget curl aria2 \
  podman docker.io tmux

Dnf (Fedora/RHEL/CentOS Stream):

sudo dnf install -y \
  git git-lfs python3 python3-pip \
  gcc gcc-c++ make cmake \
  wget curl aria2 \
  podman docker tmux

Zypper (openSUSE/SLE):

sudo zypper refresh
sudo zypper install -y \
  git git-lfs python3 python3-pip python3-virtualenv \
  gcc gcc-c++ make cmake \
  wget curl aria2 \
  podman docker tmux

Enable Git LFS once:

git lfs install

Tip: If Docker isn’t available or allowed, use Podman (rootless-friendly).

The core offline workflow (5 practical steps)

1) Build a portable Python environment (wheelhouse-first)

On an online “staging” machine:

mkdir -p ~/offline_ai/wheels
python3 -m venv ~/offline_ai/venv
source ~/offline_ai/venv/bin/activate

pip install -U pip wheel
# Freeze your dependencies
cat > requirements.txt <<'EOF'
transformers==4.41.2
huggingface_hub==0.23.4
tokenizers==0.15.2
onnxruntime==1.18.0
numpy==1.26.4
torch==2.3.1
EOF

# Download all wheels without installing system-wide
pip download -r requirements.txt -d ~/offline_ai/wheels

Transfer wheels/ and requirements.txt to the offline machine, then install:

python3 -m venv ~/offline_ai/venv
source ~/offline_ai/venv/bin/activate
pip install --no-index --find-links ~/offline_ai/wheels -r requirements.txt

Notes:

  • Pin exact versions in requirements.txt to ensure reproducibility.

  • For CUDA-enabled PyTorch or platform-specific wheels, download on a machine with matching architecture/GLIBC and GPU stack, or use containers (see Step 5).

2) Prefetch and cache models and datasets

On the online staging machine:

source ~/offline_ai/venv/bin/activate
pip install -U "huggingface_hub[cli]" hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1  # faster multi-connection downloads

# Example: download a small chat-friendly model and tokenizer
huggingface-cli download TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --local-dir ~/offline_ai/models/TinyLlama \
  --local-dir-use-symlinks False

# Example: download a GGUF quantized LLM for llama.cpp
huggingface-cli download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
  --include "*.gguf" \
  --local-dir ~/offline_ai/models/TinyLlama-GGUF \
  --local-dir-use-symlinks False

For Git LFS repos (some model repos use LFS):

cd ~/offline_ai/models
git clone https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
cd TinyLlama-1.1B-Chat-v1.0
git lfs fetch --all

Sync artifacts to the offline machine (via rsync, removable media, or your secure transfer method):

rsync -avh --progress ~/offline_ai/models/ user@offline-host:/opt/models/

On the offline machine, force local-only operation to avoid accidental network calls:

export HF_HOME=/opt/hf_cache
export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1

Pro tip: Store your cache (HF_HOME) on a fast SSD and back it up—your future self will thank you.

3) Offline LLM inference with llama.cpp (no Python required)

Build llama.cpp on the offline machine:

cd /opt
sudo git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

If you can’t access GitHub from the offline host, clone and transfer the repo tarball from the staging machine.

Run the model (using a GGUF file you pre-fetched):

/opt/llama.cpp/main \
  -m /opt/models/TinyLlama-GGUF/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
  -p "Explain offline AI workflows in 3 bullet points." \
  -n 128

Notes:

  • Choose a quantization that fits your RAM/CPU. Q4_K_M is a good starting point for small models.

  • For GPU acceleration, build with the appropriate flags (e.g., LLAMA_CUBLAS=1) on a machine with the correct toolchain and drivers.

4) Offline Transformers/ONNX inference (Python)

With your wheelhouse installed and models mirrored:

source ~/offline_ai/venv/bin/activate
export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1
python - <<'PY'
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("/opt/models/TinyLlama", local_files_only=True)
model = AutoModelForCausalLM.from_pretrained("/opt/models/TinyLlama", local_files_only=True)
input_ids = tok("Hello offline world!", return_tensors="pt").input_ids
out = model.generate(input_ids, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))
PY

ONNX Runtime example:

source ~/offline_ai/venv/bin/activate
python - <<'PY'
import onnxruntime as ort
import numpy as np

# Assume you pre-fetched a model.onnx
sess = ort.InferenceSession("/opt/models/my_onnx/model.onnx", providers=["CPUExecutionProvider"])
x = np.random.rand(1, 3, 224, 224).astype("float32")
inputs = {sess.get_inputs()[0].name: x}
outputs = sess.run(None, inputs)
print([o.shape for o in outputs])
PY

Hard-block network for a process (belt-and-suspenders):

unshare -n bash -c 'python my_infer.py'   # start with no network namespace

5) Reproducibility with containers (Podman/Docker), without the internet

On the online staging machine:

podman pull docker.io/pytorch/pytorch:2.3.1-cuda12.1-cudnn8-devel
podman save -o pytorch-2.3.1.tar docker.io/pytorch/pytorch:2.3.1-cuda12.1-cudnn8-devel

Transfer pytorch-2.3.1.tar to the offline machine and load:

podman load -i pytorch-2.3.1.tar

Run with mounted wheelhouse/models and no network:

podman run --rm --network=none \
  -v /opt/models:/opt/models:ro \
  -v /opt/wheels:/opt/wheels:ro \
  -e TRANSFORMERS_OFFLINE=1 -e HF_HUB_OFFLINE=1 \
  docker.io/pytorch/pytorch:2.3.1-cuda12.1-cudnn8-devel \
  bash -lc 'python -m venv /venv && source /venv/bin/activate && \
            pip install --no-index --find-links /opt/wheels -r /opt/wheels/requirements.txt && \
            python /opt/app/infer.py'

For Ollama users: prefetch on staging then move models offline.

# Online
ollama pull tinyllama
ollama export tinyllama > tinyllama.ollama

# Offline
ollama import < tinyllama.ollama
OLLAMA_MODELS=/opt/ollama_models ollama run tinyllama

Real-world patterns

  • Edge analytics on factory floors: Use ONNX Runtime with quantized CNNs for defect detection; cache models and Python wheels; run in Podman with --network=none.

  • Clinical settings with strict PHI rules: Use transformers offline for de-identification; set HF_HUB_OFFLINE=1, mirror models internally, and log artifact hashes for audit.

  • Gov/defense air-gapped labs: Build llama.cpp and quantized GGUF models; ship via signed media; reproduce builds using pinned commits and container images.

Common pitfalls and how to avoid them

  • Hidden downloads at runtime: Set TRANSFORMERS_OFFLINE=1 and HF_HUB_OFFLINE=1. Use local_files_only=True in code.

  • Architecture mismatch: Download wheels on a machine matching the target’s CPU/GPU stack, or rely on containers.

  • Git LFS gotchas: Always run git lfs fetch --all before going offline; otherwise you’ll have pointer files, not model weights.

  • Cache hygiene: Centralize caches (HF_HOME, pip wheelhouse) and back them up. Verify with file hashes before transfer.

Your next step (CTA)

Pick a small model and build your “AI briefcase”: 1) Create a wheelhouse on an online machine.
2) Mirror one model (TinyLlama or similar) and verify hashes.
3) Transfer to your offline host.
4) Run it with llama.cpp or Transformers—no internet needed.

Once you’ve got a basic offline inference demo, expand with containers, add CI to rebuild artifacts on version bumps, and document your reproducible runbook. If you want a reference checklist or a sample repo layout for staging/offline hosts, ask and I’ll share one tailored to your distro and hardware.

Related posts