Posted on
• Artificial Intelligence

Building Reliable AI Agents

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Building Reliable AI Agents on Linux (with Bash)

If your AI agent works on your laptop but flakes out in production, you don’t have a smart assistant—you have a liability. Reliability isn’t a luxury for agents that call APIs, juggle files, and run long jobs—it’s the difference between “neat demo” and “trusted system.”

In this post, we’ll build a reliability-first skeleton for AI agents on Linux using Bash and a few standard tools. You’ll learn why agents fail, and how to fix that with defensive patterns: timeouts, retries with backoff, idempotency, structured logging, and self-healing via systemd. You’ll leave with a ready-to-run template and clear next steps.

Why reliability is hard (and why this is worth your time)

  • AI calls are probabilistic. Outputs vary; errors and rate limits happen.

  • Networks aren’t reliable. Transient 5xx or 429 responses are normal under load.

  • Long-running tasks crash. Memory leaks and process bugs are inevitable.

  • Operators need visibility. Without structured logs and state, debugging is guesswork.

We can mitigate all of this using boring, proven Linux primitives: timeout, flock, sha256sum, systemd, jq, and careful Bash.

Prerequisites

We’ll use curl for HTTP and jq for JSON. Install them with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update && sudo apt install -y curl jq
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl jq
  • openSUSE/SLES (zypper):
sudo zypper refresh && sudo zypper install -y curl jq

No Python, containers, or extra services required. Systemd is assumed (standard on most modern distros).

5 reliability patterns you can copy-paste today

1) Harden your Bash entrypoint

  • Fail fast on errors, unset vars, and failed pipes.

  • Trap failures with line numbers for quick diagnosis.

  • Load configuration from a .env you can version and rotate.

#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

# Project layout
ROOT="${ROOT:-$PWD}"
LOG_DIR="$ROOT/logs"
RUN_DIR="$ROOT/run"
STATE_DIR="$ROOT/state"
INBOX_DIR="$ROOT/inbox"
OUTBOX_DIR="$ROOT/outbox"
mkdir -p "$LOG_DIR" "$RUN_DIR" "$STATE_DIR" "$INBOX_DIR" "$OUTBOX_DIR"

# Structured logger (prints JSON)
log() {
  local level="$1"; shift
  local msg="$*"
  local ts
  ts="$(date -Iseconds)"
  local req="${REQ_ID:-}"
  jq -nc \
    --arg ts "$ts" \
    --arg lvl "$level" \
    --arg msg "$msg" \
    --arg req "$req" \
    --arg pid "$$" \
    '{ts:$ts, level:$lvl, msg:$msg, req_id:$req, pid:($pid|tonumber)}' \
    | tee -a "$LOG_DIR/agent.log"
}

trap 'log error "exit trap status=$? line=$LINENO"' ERR
trap 'log info "shutdown"' EXIT

# Load env (AI_API_URL, AI_API_KEY, MODEL, etc.)
if [[ -f "$ROOT/.env" ]]; then
  # Only source trusted files; restrict permissions in production
  # e.g., chmod 600 .env; chown service user
  set -a
  # shellcheck source=/dev/null
  source "$ROOT/.env"
  set +a
fi

# Validate required config
require() { for v in "$@"; do [[ -n "${!v:-}" ]] || { log error "Missing env: $v"; exit 64; }; done; done; }
require AI_API_URL AI_API_KEY MODEL

Key takeaways

  • set -Eeuo pipefail will save you hours.

  • Keep .env under tight permissions and avoid untrusted input.

2) Make API calls resilient (timeouts, retries, backoff, idempotency)

  • Always bound time with timeout.

  • Retry only on transient errors (429, 5xx), with exponential backoff.

  • Use an idempotency key so providers can safely dedupe requests.

# Single-instance lock to prevent duplicate concurrent runs
exec 9>"$RUN_DIR/agent.lock"
if ! flock -n 9; then
  log warn "Another instance is running; exiting"
  exit 0
fi

# Retry wrapper for POSTing JSON to your provider
request_with_retry() {
  local body="$1"
  local url="${AI_API_URL%/}/v1/chat/completions"  # Adjust path for your provider
  local attempt=1
  local max="${MAX_RETRIES:-5}"
  local backoff=1
  local tmp
  tmp="$(mktemp)"

  while :; do
    local idem="${TASK_ID:-$(cat /proc/sys/kernel/random/uuid)}"
    REQ_ID="${REQ_ID:-$(cat /proc/sys/kernel/random/uuid)}"

    log info "API request attempt=$attempt idempotency_key=$idem"
    set +e
    local http
    http="$(timeout "${TIMEOUT_S:-30}s" curl -sS -o "$tmp" -w "%{http_code}" \
      -X POST "$url" \
      -H "Authorization: Bearer ${AI_API_KEY}" \
      -H "Content-Type: application/json" \
      -H "Idempotency-Key: ${idem}" \
      --data "$body")"
    local rc=$?
    set -e

    if [[ $rc -ne 0 ]]; then
      log warn "curl_exit=$rc (timeout/network?) backoff=${backoff}s"
    else
      case "$http" in
        200)
          cat "$tmp"
          rm -f "$tmp"
          return 0
          ;;
        429|500|502|503|504)
          log warn "retryable_http=$http backoff=${backoff}s"
          ;;
        *)
          log error "non_retryable_http=$http body_preview=$(tr -d '\n' <"$tmp" | cut -c1-400)"
          rm -f "$tmp"
          return 1
          ;;
      esac
    fi

    if (( attempt >= max )); then
      log error "exhausted_retries=$max"
      rm -f "$tmp"
      return 1
    fi

    sleep "$backoff"
    backoff=$(( backoff * 2 ))     # 1,2,4,8,16...
    attempt=$(( attempt + 1 ))
  done
}

Tips

  • Set TIMEOUT_S=30 and MAX_RETRIES=5 in .env to tune.

  • Backoff prevents thundering herds during provider incidents.

3) Idempotency and state: don’t do the same work twice

Use stable task IDs (e.g., hash the input) and keep a tiny state index. If the agent crashes and restarts, it can skip completed work.

HASH_INDEX="$STATE_DIR/done.index"
touch "$HASH_INDEX"

task_id_for_file() {
  local f="$1"
  sha256sum "$f" | awk '{print $1}'
}

already_done() {
  local id="$1"
  grep -q "^$id " "$HASH_INDEX"
}

mark_done() {
  local id="$1"; local src="$2"
  echo "$id $src $(date -Iseconds)" >> "$HASH_INDEX"
}

4) Structured logging and traceability

Emit JSON logs so you can grep, filter, and ship them. Keep a request/task ID across the pipeline.

  • View logs live via systemd:
journalctl -u ai-agent -f
  • Pretty-print logs:
journalctl -u ai-agent -o cat | jq .

With the log function above, every line has ts, level, msg, req_id, and pid.

5) Sandbox and self-heal with systemd

Run your agent as a managed service. Restart on failure, set resource limits, and restrict filesystem access.

Service unit (replace /home/USER with your path and USER with your username):

# /etc/systemd/system/ai-agent.service
[Unit]
Description=AI Agent (reliable Bash)
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=USER
Group=USER
WorkingDirectory=/home/USER/ai-agent
ExecStart=/home/USER/ai-agent/bin/agent.sh
Restart=always
RestartSec=5s
TimeoutStartSec=30s
RuntimeMaxSec=6h
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/USER/ai-agent
MemoryMax=1G
CPUQuota=200%
TasksMax=512
# Environment is loaded by the script from .env

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-agent.service

A minimal, reliable agent you can run today

This example scans an inbox/ directory for .txt files, sends each through your AI model to produce a short summary, and writes results to outbox/. It uses everything above: single-instance lock, idempotent processing, resilient API calls, and structured logs.

Project layout:

mkdir -p ~/ai-agent/{bin,inbox,outbox,logs,run,state}
cd ~/ai-agent

Create .env (chmod restrict it if it contains secrets):

cat > .env << 'EOF'
AI_API_URL=https://api.openai.com
AI_API_KEY=sk-your-key-here
MODEL=gpt-4o-mini
TIMEOUT_S=30
MAX_RETRIES=5
EOF
chmod 600 .env

Agent script:

cat > bin/agent.sh << 'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

ROOT="${ROOT:-$PWD}"
LOG_DIR="$ROOT/logs"; RUN_DIR="$ROOT/run"; STATE_DIR="$ROOT/state"; INBOX_DIR="$ROOT/inbox"; OUTBOX_DIR="$ROOT/outbox"
mkdir -p "$LOG_DIR" "$RUN_DIR" "$STATE_DIR" "$INBOX_DIR" "$OUTBOX_DIR"

log() {
  local level="$1"; shift
  local msg="$*"
  jq -nc --arg ts "$(date -Iseconds)" --arg lvl "$level" --arg msg "$msg" --arg req "${REQ_ID:-}" --arg pid "$$" \
    '{ts:$ts, level:$lvl, msg:$msg, req_id:$req, pid:($pid|tonumber)}' \
  | tee -a "$LOG_DIR/agent.log"
}

trap 'log error "exit trap status=$? line=$LINENO"' ERR
trap 'log info "shutdown"' EXIT

if [[ -f "$ROOT/.env" ]]; then set -a; source "$ROOT/.env"; set +a; fi
for v in AI_API_URL AI_API_KEY MODEL; do [[ -n "${!v:-}" ]] || { log error "Missing env: $v"; exit 64; }; done

# Single instance
exec 9>"$RUN_DIR/agent.lock"
if ! flock -n 9; then log warn "Another instance running"; exit 0; fi

HASH_INDEX="$STATE_DIR/done.index"; touch "$HASH_INDEX"

task_id_for_file() { sha256sum "$1" | awk '{print $1}'; }
already_done() { grep -q "^$1 " "$HASH_INDEX"; }
mark_done() { echo "$1 $2 $(date -Iseconds)" >> "$HASH_INDEX"; }

request_with_retry() {
  local body="$1"
  local url="${AI_API_URL%/}/v1/chat/completions"
  local attempt=1 max="${MAX_RETRIES:-5}" backoff=1
  local tmp; tmp="$(mktemp)"
  while :; do
    local idem="${TASK_ID:-$(cat /proc/sys/kernel/random/uuid)}"
    REQ_ID="${REQ_ID:-$idem}"
    log info "POST $url attempt=$attempt"
    set +e
    local http
    http="$(timeout "${TIMEOUT_S:-30}s" curl -sS -o "$tmp" -w "%{http_code}" \
      -X POST "$url" \
      -H "Authorization: Bearer ${AI_API_KEY}" \
      -H "Content-Type: application/json" \
      -H "Idempotency-Key: ${idem}" \
      --data "$body")"
    local rc=$?
    set -e
    if [[ $rc -ne 0 ]]; then
      log warn "curl_exit=$rc backoff=${backoff}s"
    else
      case "$http" in
        200) cat "$tmp"; rm -f "$tmp"; return 0 ;;
        429|500|502|503|504) log warn "retryable_http=$http backoff=${backoff}s" ;;
        *) log error "non_retryable_http=$http body_preview=$(tr -d '\n' <"$tmp" | cut -c1-400)"; rm -f "$tmp"; return 1 ;;
      esac
    fi
    if (( attempt >= max )); then log error "exhausted_retries=$max"; rm -f "$tmp"; return 1; fi
    sleep "$backoff"; backoff=$(( backoff*2 )); attempt=$(( attempt+1 ))
  done
}

process_file() {
  local f="$1"
  local id; id="$(task_id_for_file "$f")"
  TASK_ID="$id"

  if already_done "$id"; then
    log info "skip idempotent id=$id file=$f"
    return 0
  fi

  # Read up to 4000 bytes to keep prompts small; adjust as needed
  local content; content="$(head -c 4000 -- "$f")"
  local prompt="Summarize the following text in 3 bullet points with one-sentence bullets and no preamble:\n\n$content"

  local body
  body="$(jq -nc --arg m "${MODEL}" --arg p "$prompt" \
    '{model:$m, messages:[{role:"user", content:$p}], max_tokens:500, temperature:0.2}')"

  local resp
  if ! resp="$(request_with_retry "$body")"; then
    log error "failed_request id=$id file=$f"
    return 1
  fi

  local summary
  summary="$(jq -r '.choices[0].message.content // empty' <<< "$resp")"
  if [[ -z "$summary" ]]; then
    log error "empty_response id=$id file=$f"
    return 1
  fi

  local out="$OUTBOX_DIR/$(basename "$f").summary.txt"
  printf "%s\n" "$summary" > "$out"
  mark_done "$id" "$f"
  log info "wrote_summary out=$out id=$id"
}

main() {
  shopt -s nullglob
  local files=("$INBOX_DIR"/*.txt)
  if (( ${#files[@]} == 0 )); then
    log info "no_input_files inbox=$INBOX_DIR"
    return 0
  fi
  for f in "${files[@]}"; do
    process_file "$f"
  done
}

main "$@"
EOF
chmod +x bin/agent.sh

Try it:

echo "Linux makes reliable agents boring—and that's perfect." > inbox/example.txt
./bin/agent.sh

Watch logs:

tail -f logs/agent.log | jq .

You should see a .summary.txt file appear in outbox/.

To run it as a service, add the systemd unit from the previous section, update the paths, then:

sudo systemctl daemon-reload
sudo systemctl enable --now ai-agent.service
journalctl -u ai-agent -f

Real-world tips

  • Pin model and prompt. Record them with each output (include in state/logs) so results are reproducible.

  • Backpressure. If you process many files, add a simple queue and concurrency limit. Even flock plus sub-processes can get you far.

  • Budget and safety caps. Use max_tokens, temperature, timeout, and RuntimeMaxSec to bound cost/latency.

  • Secrets. Use least privilege. Consider storing AI_API_KEY in a dedicated service user’s env or secrets manager; restrict .env with chmod 600.

Conclusion and next steps

Reliable agents aren’t about fancy frameworks—they’re about disciplined engineering. With Bash, curl, and jq, you can build agents that survive bad networks, rate limits, and crashes, while giving you clear logs and safe restarts.

Your next steps: 1. Copy the template into your repo and wire it to your provider. 2. Add your domain logic in process_file (tools/functions, validation, guardrails). 3. Turn it into a systemd service for hands-free ops. 4. Iterate: add metrics, alerts, and better state (e.g., sqlite or a message queue) as needs grow.

Got stuck or want a deeper dive (queues, tracing, metrics)? Reach out or drop a comment—let’s make your agents boringly reliable.

Related posts