Posted on
• Artificial Intelligence

Testing AI Agents

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Testing AI Agents from the Linux Command Line: A Bash‑First Guide

Your AI agent works great—until it doesn’t. A single parameter change or dependency upgrade can turn yesterday’s solid answers into today’s costly failures. How do you test an inherently probabilistic system with the tools you already trust on Linux? With Bash, curl, and jq, you can build a practical, repeatable test harness that keeps your agents honest.

This article shows you how to:

  • Make AI agent tests reproducible enough to catch regressions

  • Build a tiny Bash harness that runs tests and produces clear pass/fail output

  • Use golden files that let you intentionally approve changes

  • Run tests in parallel and prep for CI

Everything runs from your terminal. No heavy frameworks required.

Why testing AI agents is different (and necessary)

  • Stochastic outputs: Language models are non-deterministic by design. Tests need to constrain randomness or assert on properties rather than exact strings.

  • Interface drift: Model versions and tool integrations change. Without tests, “good enough last week” silently becomes “broken today.”

  • Cost and latency: Fail fast locally before burning tokens or waiting on slow pipelines.

  • Reproducibility: A small, transparent Bash harness is easy to audit, vendor-free, and ideal for Linux environments.

What we’ll build

  • A JSONL file of test cases (prompt + expected property)

  • A portable Bash test runner that:

    • Calls your agent HTTP endpoint (or uses a deterministic stub if no endpoint is set)
    • Extracts the response text with jq
    • Asserts substring or regex expectations
    • Supports golden files for end-to-end regression checking
    • Runs tests in parallel for speed

Prerequisites and installation

We’ll use bash, curl, jq, and optionally GNU parallel.

Install with your package manager:

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl jq parallel
  • Fedora/RHEL (dnf):
sudo dnf install -y curl jq parallel
  • openSUSE (zypper):
sudo zypper refresh
sudo zypper install -y curl jq parallel

Verify:

bash --version
curl --version
jq --version
parallel --version

Step 1: Define test cases in JSONL

Create tests/cases.jsonl with one JSON object per line. We’ll support both “contains” and “regex” assertions so you can be strict where it matters and flexible where it doesn’t.

mkdir -p tests tests/golden

cat > tests/cases.jsonl << 'JSONL'
{"id":"hello", "prompt":"Say hello", "expect_contains":"hello", "temperature":0}
{"id":"sum", "prompt":"What is 2 + 2?", "expect_regex":"(^|[^0-9])4([^0-9]|$)", "temperature":0}
{"id":"facts", "prompt":"Name a planet in our solar system", "expect_regex":"(mercury|venus|earth|mars|jupiter|saturn|uranus|neptune)", "temperature":0}
JSONL

Notes:

  • temperature: Keep at 0 when possible to reduce variance.

  • Regex uses Extended Regular Expressions (ERE) compatible with grep -E.

Step 2: A minimal Bash harness

This script:

  • Calls your agent via HTTP if AGENT_URL is set

  • Otherwise uses a deterministic local stub (great for CI or offline work)

  • Extracts the response text from common JSON shapes

  • Checks contains/regex expectations

  • Supports golden files

Create agent-test.sh:

#!/usr/bin/env bash
set -euo pipefail

# Colors (fallback to no color if not a TTY)
if [[ -t 1 ]]; then
  green='\033[0;32m'; red='\033[0;31m'; yellow='\033[1;33m'; reset='\033[0m'
else
  green=''; red=''; yellow=''; reset=''
fi

require() {
  command -v "$1" >/dev/null 2>&1 || { echo "Missing dependency: $1" >&2; exit 127; }
}
require jq
require curl

AGENT_URL="${AGENT_URL:-}"     # e.g., http://localhost:5000/complete
SEED_DEFAULT="${SEED_DEFAULT:-42}"
MAX_TOKENS_DEFAULT="${MAX_TOKENS_DEFAULT:-256}"

# Call the agent or return a stubbed JSON response.
# Feel free to customize payload and headers for your real endpoint.
call_agent() {
  local prompt="$1" temperature="${2:-0}" max_tokens="${3:-$MAX_TOKENS_DEFAULT}" seed="${4:-$SEED_DEFAULT}"

  if [[ -n "$AGENT_URL" ]]; then
    curl -sS -X POST "$AGENT_URL" \
      -H 'Content-Type: application/json' \
      -d "$(jq -n \
            --arg prompt "$prompt" \
            --argjson temperature "$temperature" \
            --argjson max_tokens "$max_tokens" \
            --argjson seed "$seed" \
            '{prompt:$prompt, temperature:$temperature, max_tokens:$max_tokens, seed:$seed}')"
  else
    # Deterministic stub: lowercases and prefixes the prompt so tests can run offline.
    printf '{ "text": "%s" }\n' "$(printf '%s' "$prompt" | tr '[:upper:]' '[:lower:]' | sed '1s/^/stub: /')"
  fi
}

# Extract text from common response shapes.
extract_text() {
  jq -r '(
      .text
      // .choices[0].message.content
      // .choices[0].text
      // .output
      // .reply
      // .completion
      // empty
    )'
}

# Compare against golden file if present; update with GOLDEN_UPDATE=1
check_golden() {
  local id="$1" text="$2"
  local gf="tests/golden/${id}.txt"

  if [[ "${GOLDEN_UPDATE:-0}" == "1" ]]; then
    printf '%s\n' "$text" > "$gf"
    echo -e "${yellow}updated golden:${reset} $gf"
    return 0
  fi

  if [[ -f "$gf" ]]; then
    if diff -u --label "golden/$id" --label "actual/$id" "$gf" <(printf '%s\n' "$text") >/dev/null; then
      return 0
    else
      echo -e "${red}GOLDEN MISMATCH:${reset} $gf"
      diff -u --label "golden/$id" --label "actual/$id" "$gf" <(printf '%s\n' "$text") || true
      return 1
    fi
  else
    # No golden yet; skip with notice
    echo -e "${yellow}no golden:${reset} $gf (set GOLDEN_UPDATE=1 to create)"
    return 0
  fi
}

pass=0
fail=0

run_case_line() {
  local line="$1"

  # Parse fields
  local id prompt temperature max_tokens seed expect_contains expect_regex
  id="$(printf '%s' "$line" | jq -r '.id')"
  prompt="$(printf '%s' "$line" | jq -r '.prompt')"
  temperature="$(printf '%s' "$line" | jq -r '.temperature // 0')"
  max_tokens="$(printf '%s' "$line" | jq -r '.max_tokens // env.MAX_TOKENS_DEFAULT | tonumber? // env.MAX_TOKENS_DEFAULT')"
  seed="$(printf '%s' "$line" | jq -r '.seed // env.SEED_DEFAULT | tonumber? // env.SEED_DEFAULT')"
  expect_contains="$(printf '%s' "$line" | jq -r '.expect_contains // empty')"
  expect_regex="$(printf '%s' "$line" | jq -r '.expect_regex // empty')"

  # Call agent and extract text
  local response text
  response="$(call_agent "$prompt" "$temperature" "$max_tokens" "$seed" || true)"
  text="$(printf '%s' "$response" | extract_text || true)"

  local ok=0
  if [[ -n "$expect_contains" ]]; then
    if printf '%s' "$text" | grep -Fqi -- "$expect_contains"; then ok=1; fi
  fi
  if [[ -n "$expect_regex" ]]; then
    if printf '%s' "$text" | grep -Eqi -- "$expect_regex"; then ok=1; else ok=0; fi
  fi

  # Golden check (non-fatal unless you decide it should be)
  if ! check_golden "$id" "$text"; then
    # You can choose to fail the test on golden mismatch by uncommenting:
    # ok=0
    :
  fi

  if [[ "$ok" -eq 1 ]]; then
    echo -e "${green}PASS${reset} [$id]"
    pass=$((pass+1))
  else
    echo -e "${red}FAIL${reset} [$id]"
    echo "  prompt: $prompt"
    echo "  text:   $(printf '%q' "$text")"
    fail=$((fail+1))
  fi
}

run_all() {
  while IFS= read -r line; do
    [[ -z "$line" ]] && continue
    run_case_line "$line"
  done < tests/cases.jsonl

  echo
  echo "Summary: $pass passed, $fail failed"
  [[ "$fail" -eq 0 ]]
}

# Allow per-line execution for parallel runs
run_json() {
  local json="$1"
  run_case_line "$json"
}

mode="${1:-run}"
case "$mode" in
  run) run_all ;;
  run-json) run_json "${2:?json-line-required}" ;;
  *) echo "Usage: $0 [run|run-json <json>]" >&2; exit 2 ;;
esac

Make it executable:

chmod +x agent-test.sh

Run it (offline stub mode):

./agent-test.sh

You should see PASS/FAIL along with a summary.

To save initial goldens:

GOLDEN_UPDATE=1 ./agent-test.sh

To point the harness at a real agent:

export AGENT_URL="http://localhost:5000/complete"
./agent-test.sh

Tip: Adjust the JSON payload in call_agent() to match your API (headers, schema, etc.). Keep temperature near 0 and set a seed if your provider supports it.

Step 3: Property assertions and golden files

  • Property assertions (contains/regex) are resilient to small wording changes. Use them for core correctness.

  • Golden files are perfect for end-to-end UX snapshots. When a deliberate change happens, approve it with:

GOLDEN_UPDATE=1 ./agent-test.sh
  • If a change is accidental, the diff highlights exactly what regressed.

Step 4: Make tests as deterministic as possible

  • Temperature 0: Minimizes randomness.

  • Seed: If your provider supports a seed parameter, wire it in. We included seed in the payload.

  • Constrain output: Ask for specific formats (e.g., JSON) and assert structurally with jq, e.g.:

printf '%s' "$text" | jq -e '.answer == 4' >/dev/null
  • Stable context: Keep tools, prompts, and retrieval data versioned; reference by commit or tag in your agent.

Step 5: Run tests in parallel and prep for CI

Speed up runs with GNU parallel:

# One process per test line
cat tests/cases.jsonl | parallel --jobs 4 --line-buffer \
  './agent-test.sh run-json {=uq=}'

# Or with xargs (less robust for JSON)
jq -c . tests/cases.jsonl | xargs -I{} -P4 ./agent-test.sh run-json '{}'

CI tips:

  • Cache or pin models/containers so tests don’t drift across runs.

  • Fail the build if any test fails (the script already exits non-zero on failure).

  • Store goldens in your repo; protect updates with code review.

Bonus: Mock external tools and time

Agents often call tools or depend on timestamps. Keep your tests hermetic:

  • Time: Set SOURCE_DATE_EPOCH to freeze timestamps in your prompts/tools.
export SOURCE_DATE_EPOCH=1700000000
  • Tool shims: Put a fake tool earlier in PATH for tests:
mkdir -p .test-bin
cat > .test-bin/mytool <<'SH'
#!/usr/bin/env bash
echo '{"result":"ok","value":42}'
SH
chmod +x .test-bin/mytool
export PATH="$(pwd)/.test-bin:$PATH"

Now your agent “tool call” gets a deterministic reply.

Real‑world patterns that work

  • Normalize responses: Ask the agent for JSON, then validate fields with jq instead of brittle string matches.

  • Minimize context: Keep prompts small and focused in tests to reduce variance.

  • Golden at the edges, properties in the core: Use goldens for UX text and properties for factual or structural correctness.

  • Keep a “smoke” suite: 5–10 fast tests you can run before every commit.

Conclusion and next steps

You don’t need heavyweight frameworks to keep AI agents in line. With Bash, curl, and jq you can:

  • Write clear, property-based tests

  • Capture end-to-end behavior with goldens

  • Run everything quickly and repeatably on Linux

Your next step: 1) Copy the script and cases into your repo. 2) Point AGENT_URL at your agent. 3) Add 3–5 critical tests today. 4) Wire it into CI and protect GOLDEN_UPDATE with review.

If you want a deeper dive next, extend the harness to:

  • Validate JSON schemas with jq

  • Measure latency and token usage per test

  • Emit JUnit XML for CI dashboards

Testing AI agents doesn’t have to be mysterious. Keep it simple, keep it bashy, and ship with confidence.

Related posts