Posted on
• Artificial Intelligence

Single-Agent vs Multi-Agent Systems

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Single-Agent vs Multi-Agent Systems: A Bash-Friendly Guide for Linux Users

Ever watched a “works-on-my-laptop” Bash script collapse under real-world load? Maybe it runs for hours, hogs a core, and becomes unmaintainable. The fix is not always “rewrite in Go” or “throw Kubernetes at it.” Often, the choice is simpler: decide whether your automation should be a single agent or a set of cooperating agents.

This post explains the difference, shows when each approach shines, and gives you practical Bash-first patterns (with ready-to-run snippets) to move from theory to working systems.

What do we mean by “agent”?

  • Single-agent: One process (often one script) performs the entire job sequentially or with simple pipelines.

  • Multi-agent: Multiple cooperating processes/workers each handle a slice of the workload, coordinated via a fan-out tool (like GNU Parallel) or a lightweight queue (like Redis).

Why this matters

  • Performance: Multi-agent designs can saturate CPU cores and hide IO latency.

  • Reliability: Workers can fail independently; the system can retry specific tasks.

  • Simplicity vs. scale: Single-agent scripts are easier to write and debug; multi-agent systems introduce coordination but unlock throughput.

  • Cost/portability: You can scale on one box with core Linux tools. No heavyweight orchestration required.

Install the tools we’ll use

The examples below use jq (for JSON parsing), GNU Parallel (for fan-out), and Redis (for a tiny job queue).

Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jq parallel redis-server
# Start Redis
sudo systemctl enable --now redis-server

Fedora (dnf):

sudo dnf install -y jq parallel redis
# Start Redis
sudo systemctl enable --now redis

openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y jq gnu_parallel redis
# Start Redis
sudo systemctl enable --now redis

Note: GNU Parallel may be named gnu_parallel on openSUSE.

A quick decision checklist

Use a single agent if:

  • The workload is small-to-medium and finishes in acceptable time.

  • The steps are tightly coupled and order-dependent.

  • Debuggability and simplicity outweigh throughput.

Use multiple agents if:

  • Tasks are embarrassingly parallel (process N files/URLs/records independently).

  • You need faster turnaround by using all cores or multiple machines.

  • You want fault isolation and targeted retries without restarting everything.

Example 1: Single-agent log summarizer (simple, reliable)

When your data fits comfortably on one machine and you value simplicity, a single-agent script with solid error handling wins.

Create a tiny JSON-lines log file and a summarizer:

mkdir -p demo && cd demo

cat > logs.jsonl <<'EOF'
{"level": "info", "msg": "start", "ts": 1710000001}
{"level": "warn", "msg": "low-disk", "ts": 1710000002}
{"level": "error", "msg": "failed", "ts": 1710000003}
{"level": "info", "msg": "finish", "ts": 1710000004}
EOF

cat > summarize_logs.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
trap 'echo "Error on line $LINENO" >&2' ERR

input="${1:-logs.jsonl}"

# Validate input quickly
if [[ ! -s "$input" ]]; then
  echo "No input: $input" >&2
  exit 1
fi

# Count entries by level with jq
jq -r '.level' "$input" | awk '
{count[$1]++}
END {for (k in count) printf "%s %d\n", k, count[k] | "sort"}'
EOF

chmod +x summarize_logs.sh
./summarize_logs.sh logs.jsonl

Why it works:

  • Self-contained, reproducible, minimal moving parts.

  • Great for daily jobs, CI hooks, and early prototypes.

Example 2: Multi-agent fan-out with GNU Parallel (fast and local)

When you want to saturate CPU cores on one box with almost-zero coordination overhead, use GNU Parallel.

Task: Compute checksums for many files in parallel.

mkdir -p data checksums && cd data
# Make some sample files
for i in $(seq -w 1 1000); do
  printf "file %s\n" "$i" > "file_${i}.txt"
done
cd ..

Run with GNU Parallel (uses all available cores by default, or set -j):

find data -type f -print0 \
  | parallel -0 -j "$(nproc)" 'sha256sum {} > checksums/{/}.sha256'

Notes:

  • {/} expands to the basename; each output file is independent, so no write contention.

  • For safety, fail fast if any job fails: add --halt soon,fail=1.

  • You can throttle IO-bound tasks with -j N to reduce disk pressure.

Example 3: Multi-agent queue with Redis (coordinated, retry-friendly)

When you need at-least-once processing, flexible scaling, or remote workers (over the network), a tiny Redis-backed queue is a great upgrade.

Make some demo data:

mkdir -p rqdata && for i in $(seq -w 1 100); do echo "payload $i" > "rqdata/item_${i}.txt"; done

Producer: push jobs to a Redis list:

cat > producer.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

QUEUE="${QUEUE:-jobs}"
for f in rqdata/*.txt; do
  redis-cli LPUSH "$QUEUE" "$f" >/dev/null
done

echo "Enqueued: $(redis-cli LLEN "$QUEUE") jobs"
EOF
chmod +x producer.sh
./producer.sh

Worker: blocks for work, processes, and acknowledges. Run multiple copies on the same or different machines.

cat > worker.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

QUEUE="${QUEUE:-jobs}"
OUT="${OUT:-checksums-redis.txt}"

# Ensure output exists; writes append per task
: > "$OUT" || true

while true; do
  # --raw makes parsing simple; BRPOP returns "key" then "value"
  job=$(redis-cli --raw BRPOP "$QUEUE" 0 | tail -n1) || continue

  if [[ -n "$job" && -f "$job" ]]; then
    # Do idempotent work; append is safe across workers
    sha256sum "$job" >> "$OUT"
    echo "Processed $job"
  else
    echo "Skipped invalid job: $job" >&2
  fi
done
EOF
chmod +x worker.sh

Run 4 workers in separate shells or background:

for i in 1 2 3 4; do ./worker.sh & done
wait

Why this works:

  • BRPOP blocks until work appears; simple backpressure.

  • Multiple workers compete fairly on the same queue.

  • Append-only outputs avoid lock contention; for shared files, consider flock.

Hardening ideas:

  • Use RPOPLPUSH for “in-flight” visibility and retries on crash.

  • Store job metadata (attempts, timestamps) in Redis hashes.

  • Add timeouts and dead-letter queues for poison messages.

Monitoring, logging, and safety tips

  • Add set -euo pipefail and trap ERR to every script.

  • Emit structured logs and summarize with jq:

    some_script | jq -c '{ts: now, stream: "worker", msg: .}'

  • For shared resources, use flock:

    exec 9>"/tmp/my.lock"
flock 9
# critical section

  • Keep tasks idempotent: re-running a job shouldn’t corrupt state.

  • Bound concurrency (e.g., -j N in parallel) to match CPU/IO capacity.

Putting it all together

  • Start with a single-agent script for clarity.

  • When runtime or throughput becomes a pain, try fan-out with GNU Parallel.

  • If you need coordination, retries, or cross-host scaling, introduce a tiny queue (Redis) and multiple workers.

Call to action:

  • Clone your heaviest Bash task and try both patterns above with a subset of your data. Measure wall-clock time and resource usage (time, vmstat, iostat).

  • Add one safety improvement (idempotency or flock) this week.

  • Share what you learned with your team and standardize a small “Bash agent” template for future jobs.

You don’t need a cluster to think like a distributed system designer—Linux, Bash, and a few small packages get you surprisingly far.

Related posts