Artificial Intelligence Pipeline Optimisation with Bash: Stream, Parallelise, Cache, Repeat

When your model is fast but your results are slow, the bottleneck isn’t AI—it’s your pipeline glue. Most AI teams lose more time in data prep, I/O, and orchestration than in model inference. The upside: optimizing the shell-based “plumbing” around your models often yields larger, cheaper wins than changing the model itself.

This guide shows how to speed up and stabilise AI pipelines using battle-tested Bash patterns. You’ll learn to measure the right things, stream instead of staging, parallelise safely, and cache aggressively—all with minimal dependencies and commands you can drop into your existing scripts.

Why pipeline optimisation matters

• Throughput > micro-optimisations: If you’re I/O bound, shaving 5% off a model runtime won’t matter. Eliminating temporary files and parallelising the right stage can double throughput.

• Cost and stability: Efficient pipelines do less disk I/O, use fewer temp files, and restart cleanly. That cuts incidents and cloud bills.

• Reproducibility: Small Bash patterns—locks, Makefiles, and content-addressable caches—turn flaky runs into dependable production jobs.

Prerequisites: install the common CLI tools

The examples below use only standard POSIX tools plus a few small utilities. Install them once and reuse everywhere.

• Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y parallel jq pv pigz lz4 sqlite3 python3 python3-pip make git curl time

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y parallel jq pv pigz lz4 sqlite python3 python3-pip make git curl time

• openSUSE/SLE (zypper):

sudo zypper refresh
sudo zypper install -y parallel jq pv pigz lz4 sqlite3 python3 python3-pip make git curl time

What each tool is for:

• parallel: simple and safe concurrency for CPUs and I/O

• jq: fast JSONL transformation

• pv: live progress and throughput

• pigz/lz4: multi-core (de)compression for streams

• sqlite3: small, embedded metadata/artifact index

• time: detailed resource usage of commands

---

A minimal reference pipeline shape

Let’s assume a common workflow:

• Input: data/raw.jsonl.gz (newline-delimited JSON, gzip-compressed)

• Clean/filter: jq selects the records and fields we need

• Inference: Python reads plain-text or JSONL from stdin and writes JSONL to stdout

• Output: compressed predictions out/preds.jsonl.gz

We’ll optimise around that with four high-impact techniques.

---

1) Measure before tuning

If you don’t measure, you’ll optimise the wrong thing. Start every script by timing the slowest stages and watching throughput.

• Use the external time binary for detail:

/usr/bin/time -v pigz -dc data/raw.jsonl.gz | jq -c . >/dev/null

• Add pv to surface throughput in any pipeline:

pv data/raw.jsonl.gz | pigz -dc | jq -c . >/dev/null

• Wrap a stage to log resource use:

log_time() { echo "== $*"; /usr/bin/time -v "$@"; }

log_time sh -c 'pigz -dc data/raw.jsonl.gz | jq -c ".text" | wc -l'

Look for:

• High “User time”: CPU-bound, you may benefit from parallelisation.

• High “System time” or low pv throughput: I/O bound, prefer streaming and fewer temp files.

• Large “Maximum resident set size”: memory-bound; use chunking and --pipe.

---

2) Stream, don’t stage

Temporary files kill performance. Linux pipes plus multi-core compression let you process large datasets without touching disk.

• Replace unzip-then-process with a streaming pipeline:

pigz -dc data/raw.jsonl.gz \
| jq -rc 'select(.lang=="en") | {id, text}' \
| pv \
| python3 infer.py \
| pigz -1 > out/preds.jsonl.gz

Notes:

• pigz -dc decompresses on all cores.

• jq filters and projects only what you need.

• pv shows live MB/s and records/s if lines are uniform.

• pigz -1 uses fast compression to avoid becoming the bottleneck (tune level as needed).

• If you must branch a stream to multiple consumers, use tee:

pigz -dc data/raw.jsonl.gz \
| jq -rc '.text' \
| tee run/text.clean.txt \
| python3 infer.py \
| pigz -1 > out/preds.jsonl.gz

---

3) Parallelise the right stage with GNU Parallel

Parallelising arbitrary shell loops is brittle. GNU Parallel makes it safe and easy—especially with stdin pipelines.

• Parallelise line-based processing (stdin → stdin tool), preserving order:

pigz -dc data/raw.jsonl.gz \
| jq -rc '.text' \
| pv \
| parallel --pipe -k -j"$(nproc)" --block 10M python3 infer.py \
| pigz -1 > out/preds.jsonl.gz

What this does:

• --pipe splits stdin into chunks and feeds each chunk to a separate job.

• --block 10M sizes chunks to balance CPU and memory (tune to your model’s tokeniser cost).

• -j $(nproc) uses all cores; start with half on busy hosts.

• -k keeps input order in the output (handy for aligning ids).

• Parallelise file shards (one file per job):

ls shards/*.jsonl.gz \
| parallel -j"$(nproc)" '
    pigz -dc {} \
    | jq -rc ".text" \
    | python3 infer.py \
    | pigz -1 > out/{/.}.preds.jsonl.gz
'

Real-world tip:

• If the model initialises slowly, ensure each job processes many records (bigger blocks) to amortise startup cost.

• If Python GIL or GPU is involved, you may want fewer jobs than CPUs (e.g., -j 2 with one GPU-bound worker per card).

---

4) Cache aggressively with content-addressable outputs

If a stage is pure (same input → same output), don’t recompute it. A tiny Bash cache saves hours on retries and tuning.

• Cache by input file + command signature:

cache_file() {
  # usage: cache_file inputfile -- cmd arg...
  local in="$1"; shift
  [ "$1" = "--" ] && shift
  mkdir -p cache
  local tag
  tag="$(printf '%s\0%s' "$(sha256sum "$in" | cut -d' ' -f1)" "$*" | sha256sum | cut -d' ' -f1)"
  local out="cache/$tag"
  if [ -s "$out" ]; then
    cat "$out"
    return 0
  fi
  local tmp="$out.tmp.$$"
  if cat "$in" | "$@" | tee "$tmp" >/dev/null; then
    mv "$tmp" "$out"
    cat "$out"
  else
    rm -f "$tmp"
    return 1
  fi
}

• Use it to cache tokenisation or embeddings:

cache_file data/clean.txt -- python3 embed.py | pigz -1 > out/embeddings.jsonl.gz

• Or to cache JSONL cleaning:

cache_file data/raw.jsonl.gz -- sh -c 'pigz -dc | jq -rc ".text"'

Tip:

• Bump the cache key when you change code: include a version token in the command, e.g., CMD_VERSION=v3 python3 infer.py.

For metadata and quick lookups (e.g., which shard ran with which model hash), a lightweight SQLite DB avoids spreadsheets:

sqlite3 run.db 'create table if not exists runs(id text primary key, cmd text, ts default current_timestamp);'
sqlite3 run.db 'insert or replace into runs(id, cmd) values ("'"$tag"'", "'"$*"'" );'

---

5) Make it reproducible and resumable

Small Bash patterns prevent double-runs, partial outputs, and inconsistent environments.

• Fail fast and trap errors:

set -euo pipefail
trap 'echo "Error on line $LINENO"; exit 1' ERR

• Lock to avoid concurrent runs colliding:

exec 9>run/pipeline.lock
flock -n 9 || { echo "Another run is in progress"; exit 0; }

• Orchestrate with a tiny Makefile so you can make -j and only rebuild what changed:

# Makefile
THREADS := $(shell nproc)

out/preds.jsonl.gz: data/raw.jsonl.gz
    pigz -dc $< \
    | jq -rc 'select(.lang=="en") | .text' \
    | parallel --pipe -k -j $(THREADS) --block 10M python3 infer.py \
    | pigz -1 > $@

.PHONY: all
all: out/preds.jsonl.gz

Then run:

make -j

• Keep environments sane. If you need Python packages, install them per-project:

python3 -m pip install --user --upgrade pip
python3 -m pip install -r requirements.txt

(Or use a venv/container if your distro’s Python is shared across apps.)

---

Mini case study: 40–70% wall-clock savings

Applying the four steps above to a 25 GB gzip-compressed JSONL corpus:

• Streaming (no temp files) improved throughput from ~90 MB/s to ~160 MB/s

• pigz for compression/decompression and pv for visibility prevented bottlenecks

• parallel --pipe with --block 16M -j 8 turned a 2h serial infer run into ~45 minutes

• A content-addressable cache meant retries after a config change resumed instantly for already-processed shards

Your exact gains will vary, but the pattern is robust: stream, parallelise where it counts, and never redo pure work.

---

Put it all together: a ready-to-adapt runner

Drop this into scripts/run.sh and tweak paths:

#!/usr/bin/env bash
set -euo pipefail

mkdir -p out run cache

# Single-instance lock
exec 9>run/pipeline.lock
flock -n 9 || { echo "Another run is in progress"; exit 0; }

THREADS="${THREADS:-$(nproc)}"
BLOCK="${BLOCK:-10M}"

echo "Threads: $THREADS  Block: $BLOCK"

# Measure input read speed
echo "== Measuring input throughput"
/usr/bin/time -v sh -c 'pv data/raw.jsonl.gz | pigz -dc >/dev/null'

# Main pipeline: stream + parallel + compress
echo "== Inference"
pigz -dc data/raw.jsonl.gz \
| jq -rc 'select(.lang=="en") | .text' \
| pv \
| parallel --pipe -k -j "$THREADS" --block "$BLOCK" python3 infer.py \
| pigz -1 > out/preds.jsonl.gz

echo "Done -> out/preds.jsonl.gz"

Run it:

bash scripts/run.sh

Tune it:

• Increase BLOCK if model startup dominates

• Reduce THREADS if you saturate disks or GPUs

• Add tee to persist intermediates only where needed

---

Conclusion and next steps

AI pipeline optimisation is mostly systems hygiene:

• Measure with /usr/bin/time -v and pv

• Stream everything; avoid temp files

• Use GNU Parallel for the right kind of concurrency

• Cache pure stages to make retries cheap

• Add locks and Makefiles for safe, resumable runs

Your next step: 1) Install the tools via apt/dnf/zypper (see above). 2) Convert one serial stage in your pipeline to a streaming + parallel pattern. 3) Add a content-addressable cache to your most expensive pure transform. 4) Measure again and iterate.

If you want a deeper dive, share a snippet of your current pipeline layout (no secrets) and the slowest command; I’ll suggest a targeted refactor.