AI Agent Observability for Linux Bash Users: From Zero to Insightful Traces and Metrics

Artificial Intelligence (AI) agents are no longer just fancy demos—they route tools, reason over multiple steps, call external APIs, and make decisions that affect cost, latency, and correctness. That power comes with complexity: when something goes wrong (or just goes slow), how do you know what the agent did, why it did it, and how much it cost?

Observability is your flashlight. With a few simple Bash-friendly steps, you can add traces, metrics, and log correlation to your agents and get real-time insight into how they behave in the wild.

In this guide you’ll:

• Understand why observability is essential for AI agents.

• Spin up a zero-fuss local observability stack with containers.

• Add traces and metrics from plain Bash without changing languages or frameworks.

• Correlate logs with traces for powerful debugging.

• Leave with production-ready tips.

Why AI Agent Observability Matters

• Multi-step reasoning is opaque by default. Chains, tools, and callbacks create black boxes without explicit signals.

• Token costs and latency drift over time. Without metrics, you can’t control spend or catch slowdowns.

• LLM failures are nuanced. Timeouts, 429s, tool hallucinations—each needs different remediation.

• Teams move faster with shared truth. Dashboards and traces speed up triage, root-cause analysis, and rollback decisions.

Observability pays for itself when the first incident hits—or when a performance win cuts latency by 20%.

What We’ll Build

• Traces: Use Jaeger to visualize agent steps via Zipkin-compatible spans (easy to post from Bash with curl).

• Metrics: Use Prometheus and node_exporter; record custom agent counters via the textfile collector.

• Dashboards: Use Grafana to visualize metrics and link to traces.

• Correlated logs: Include trace_id/span_id in logs for powerful forensics.

All services run in containers so you don’t fight distro package quirks.

Prerequisites: CLI Tools

We’ll use Podman, curl, and jq. Install with your package manager:

• Ubuntu/Debian (apt):

sudo apt update
sudo apt install -y podman curl jq

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y podman curl jq

• openSUSE (zypper):

sudo zypper refresh
sudo zypper install -y podman curl jq

Tip: If you prefer Docker, you can substitute docker for podman in all commands below.

Step 1: Run a Local Observability Stack in Containers

We’ll start Jaeger (traces), Prometheus (metrics), Grafana (dashboards), and node_exporter (host + custom metrics).

1) Create a directory for Prometheus config and textfile metrics:

mkdir -p ~/obs-demo
mkdir -p ~/obs-demo/textfiles

2) Write a minimal Prometheus config:

cat > ~/obs-demo/prometheus.yml <<'YAML'
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['host.docker.internal:9100', '127.0.0.1:9100']
YAML

Note: host.docker.internal works on some setups; 127.0.0.1:9100 covers the port publish we’ll do.

3) Start Jaeger (traces UI on http://127.0.0.1:16686):

podman run -d --name jaeger \
  -p 16686:16686 -p 14268:14268 -p 9411:9411 \
  jaegertracing/all-in-one:1.54

• Port 9411 is the Zipkin endpoint we’ll POST spans to from Bash.

4) Start node_exporter with textfile collector (metrics on 9100):

podman run -d --name node_exporter \
  -p 9100:9100 \
  -v ~/obs-demo/textfiles:/textfiles:Z \
  quay.io/prometheus/node-exporter:latest \
  --collector.textfile \
  --collector.textfile.directory=/textfiles

5) Start Prometheus (UI on 9090):

podman run -d --name prometheus \
  -p 9090:9090 \
  -v ~/obs-demo/prometheus.yml:/etc/prometheus/prometheus.yml:Z \
  docker.io/prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml

6) Start Grafana (UI on 3000; default login admin/admin):

podman run -d --name grafana \
  -p 3000:3000 \
  grafana/grafana-oss:latest

Verify:

• Jaeger UI: http://127.0.0.1:16686

• Prometheus: http://127.0.0.1:9090/targets (node_exporter should be “UP”)

• Grafana: http://127.0.0.1:3000

To stop and clean up later:

podman stop jaeger prometheus grafana node_exporter
podman rm jaeger prometheus grafana node_exporter

Step 2: Add Traces from Bash via Zipkin/Jaeger

Traces model your agent’s behavior as spans: “plan → tool_call → summarize,” with durations, errors, and attributes.

The Zipkin v2 HTTP API accepts JSON, so you can instrument any Bash script with curl. Below is a small utility you can source in your agent scripts.

1) Create a tracing helper file:

cat > ~/obs-demo/trace.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail

ZIPKIN_ENDPOINT="${ZIPKIN_ENDPOINT:-http://127.0.0.1:9411/api/v2/spans}"
SERVICE_NAME="${SERVICE_NAME:-bash-agent}"

# Generate 16-byte (32 hex chars) IDs
hex_16b() {
  dd if=/dev/urandom bs=16 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n'
}

now_micros() {
  # microseconds since epoch
  echo $(( $(date +%s%N) / 1000 ))
}

# Start a span: span_start "name" [parent_span_id]
span_start() {
  local name="$1"
  local parent="${2:-}"
  TRACE_ID="${TRACE_ID:-$(hex_16b)}"
  SPAN_ID="$(hex_16b)"
  SPAN_NAME="$name"
  SPAN_START_TS="$(now_micros)"
  PARENT_ID="$parent"
  export TRACE_ID SPAN_ID SPAN_NAME SPAN_START_TS PARENT_ID
  echo "[$(date -Is)] level=info msg=\"span start\" service=$SERVICE_NAME name=\"$SPAN_NAME\" trace_id=$TRACE_ID span_id=$SPAN_ID parent_id=${PARENT_ID:-none}"
}

# Finish a span: span_end [status] [key=value key=value...]
span_end() {
  local status="${1:-ok}"; shift || true
  local end_ts="$(now_micros)"
  local duration="$(( end_ts - SPAN_START_TS ))"

  # Build tags object from key=value pairs
  local tags="\"status\":\"$status\""
  for kv in "$@"; do
    key="${kv%%=*}"
    val="${kv#*=}"
    # basic JSON escape for quotes/backslashes
    val="${val//\\/\\\\}"; val="${val//\"/\\\"}"
    tags+=",\"$key\":\"$val\""
  done

  # Conditionally include parentId
  local parent_json=""
  if [[ -n "${PARENT_ID:-}" ]]; then
    parent_json="\"parentId\": \"${PARENT_ID}\","
  fi

  # Post the span
  payload=$(cat <<JSON
[
  {
    "traceId": "${TRACE_ID}",
    "id": "${SPAN_ID}",
    ${parent_json}
    "name": "${SPAN_NAME}",
    "timestamp": ${SPAN_START_TS},
    "duration": ${duration},
    "localEndpoint": {"serviceName": "${SERVICE_NAME}"},
    "tags": { ${tags} }
  }
]
JSON
)
  curl -sS -X POST -H 'Content-Type: application/json' \
    --data "${payload}" "${ZIPKIN_ENDPOINT}" >/dev/null

  echo "[$(date -Is)] level=info msg=\"span end\" service=$SERVICE_NAME name=\"$SPAN_NAME\" trace_id=$TRACE_ID span_id=$SPAN_ID duration_us=$duration status=$status"
}

# Helper to run a command as a child span
run_with_span() {
  local step="$1"; shift
  local parent="${SPAN_ID:-}"
  span_start "$step" "$parent"
  if "$@"; then
    span_end ok "agent.step=$step" "cmd=$*"
  else
    span_end error "agent.step=$step" "cmd=$*" "error=$?"
    return 1
  fi
}
BASH
chmod +x ~/obs-demo/trace.sh

2) Example: instrument a simple AI-ish agent flow:

cat > ~/obs-demo/agent.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/trace.sh"

SERVICE_NAME="bash-agent"
export SERVICE_NAME

# Simulated tools
tool_search() { sleep 0.3; echo "result:42"; }
tool_llm() { sleep 0.6; echo "tokens_out=128"; }

main() {
  span_start "agent.run"
  run_with_span "plan" bash -c 'sleep 0.2; echo "plan: use search then llm" >/dev/null'
  run_with_span "tool.search" tool_search
  run_with_span "tool.llm" tool_llm
  span_end ok "user_id=u123" "session=abc"
}
main "$@"
BASH
chmod +x ~/obs-demo/agent.sh

3) Run it:

~/obs-demo/agent.sh

4) Open Jaeger UI at http://127.0.0.1:16686, find “bash-agent,” and view the trace timeline with child spans.

This pattern scales: wrap each agent step (plan, retrieve, call-API, parse, summarize) with spans, stamp errors, and add useful tags like user_id, session, model, temperature, or tool name.

Step 3: Export Prometheus Metrics from Bash (Textfile Collector)

Node exporter’s textfile collector lets you write metrics from any script as a simple .prom file. Prometheus scrapes node_exporter, which reads these files.

1) Create a helper to record counters and gauges:

cat > ~/obs-demo/metrics.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
TEXTDIR="${TEXTDIR:-$HOME/obs-demo/textfiles}"
METFILE="${METFILE:-$TEXTDIR/agent.prom}"
AGENT_NAME="${AGENT_NAME:-bash-agent}"

mkdir -p "$TEXTDIR"

# Increment a counter: inc_counter metric_name [labels as key=value ...]
inc_counter() {
  local name="$1"; shift
  local labels="agent=\"$AGENT_NAME\""
  for kv in "$@"; do
    k="${kv%%=*}"; v="${kv#*=}"
    labels+=",${k}=\"${v}\""
  done
  # naive in-place increment per labelset
  local line="${name}{${labels}}"
  local cur=0
  if grep -q "^${line} " "$METFILE" 2>/dev/null; then
    cur=$(grep "^${line} " "$METFILE" | awk '{print $2}')
  fi
  local next=$((cur+1))
  # remove old line and append new
  grep -v "^${line} " "$METFILE" 2>/dev/null > "${METFILE}.tmp" || true
  printf "%s %d\n" "$line" "$next" >> "${METFILE}.tmp"
  mv "${METFILE}.tmp" "$METFILE"
}

# Set a gauge: set_gauge metric_name value [labels ...]
set_gauge() {
  local name="$1"; local value="$2"; shift 2
  local labels="agent=\"$AGENT_NAME\""
  for kv in "$@"; do
    k="${kv%%=*}"; v="${kv#*=}"
    labels+=",${k}=\"${v}\""
  done
  local line="${name}{${labels}}"
  grep -v "^${line} " "$METFILE" 2>/dev/null > "${METFILE}.tmp" || true
  printf "%s %s\n" "$line" "$value" >> "${METFILE}.tmp"
  mv "${METFILE}.tmp" "$METFILE"
}
BASH
chmod +x ~/obs-demo/metrics.sh

2) Use it from your agent:

cat > ~/obs-demo/agent_with_metrics.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/trace.sh"
source "$(dirname "$0")/metrics.sh"

SERVICE_NAME="bash-agent"
export SERVICE_NAME
AGENT_NAME="bash-agent"
export AGENT_NAME

tool_llm() { sleep 0.4; echo "tokens_out=256"; }

main() {
  span_start "agent.run"
  run_with_span "tool.llm" tool_llm | {
    read -r line
    tokens=$(echo "$line" | awk -F= '{print $2}')
    inc_counter agent_requests_total model=gpt-4o
    inc_counter agent_tokens_total model=gpt-4o
    # For demo, write tokens as gauge
    set_gauge agent_last_tokens tokens="$tokens" model=gpt-4o
    echo "$line" >/dev/null
  }
  span_end ok "model=gpt-4o"
}
main "$@"
BASH
chmod +x ~/obs-demo/agent_with_metrics.sh

3) Run it a few times:

~/obs-demo/agent_with_metrics.sh
~/obs-demo/agent_with_metrics.sh

4) In Prometheus (http://127.0.0.1:9090), query:

• agent_requests_total

• agent_tokens_total

• agent_last_tokens

You should see counters increasing and gauges updating. In Grafana, add Prometheus as a data source (Settings → Data sources → Prometheus at http://prometheus:9090 if running inside the same network, or http://127.0.0.1:9090 from your browser) and build a dashboard:

• Panel 1: agent_requests_total by model

• Panel 2: rate(agent_requests_total[5m]) for QPS

• Panel 3: avg(agent_last_tokens) by model

• Panel 4: Link to Jaeger trace UI (dashboards → add “link” with URL http://127.0.0.1:16686)

Step 4: Correlate Logs with Traces

Every log line becomes more valuable when it carries trace_id and span_id. We already echo them in trace.sh. For your own logs:

• Include keys in every echo/printf:

echo "[$(date -Is)] level=info msg=\"calling tool\" trace_id=$TRACE_ID span_id=${SPAN_ID:-none} tool=search q=\"kubernetes\""

• In Grafana, you can add a “Jaeger trace” link template using ${__field.values} to jump directly from a log to a trace. If you use a log store (e.g., Loki), add parsers that extract trace_id/span_id.

Real-World Example: A 2x Latency Win

A team saw agent latency creeping from 1.2s to 2.6s p95. Traces showed “tool.search” ballooning after certain queries; metrics exposed a spike in token output too. Root cause: a misconfigured retriever hitting a slow index. Fixing the index and adding a small cache dropped “tool.search” by 60%, cutting end-to-end p95 back to 1.3s and saving ~18% in token cost.

Observability made the invisible obvious—and made the fix fast.

Production Tips

• Sample smartly: Keep end-to-end trace sampling low (e.g., 5–10%) but always trace errors. Record metrics for every request.

• Redact PII: Don’t put secrets or user content in tags. Prefer hashes or anonymized IDs.

• Budget-aware metrics: Track prompt_tokens_total, completion_tokens_total, and cost_usd_total to catch spend regressions early.

• SLOs: Define SLOs like “p95 agent.run < 2s” and alert on error rate and latency across key models.

• Backpressure: If your collector or backend is down, make tracing non-blocking so agents don’t fail.

Troubleshooting

• I don’t see traces:

  • Ensure Jaeger is running and port 9411 is open.
  • Check curl return codes in trace.sh (remove >/dev/null temporarily).

• I don’t see metrics:

  • Confirm node_exporter is UP in Prometheus targets.
  • Ensure your .prom file is in the mounted directory and node_exporter has --collector.textfile.

• SELinux denials on Fedora:

  • Keep the :Z suffix on volume mounts so Podman relabels contexts.

Conclusion and Next Steps

Agent observability isn’t a luxury—it’s table stakes for reliable AI systems. In under an hour you:

• Brought up Jaeger, Prometheus, and Grafana locally with containers.

• Emitted spans from plain Bash via Zipkin.

• Published custom metrics with node_exporter’s textfile collector.

• Correlated logs with traces for faster debugging.

Your next step:

• Instrument your real agent’s critical steps (planning, tools, LLM calls).

• Track token counts, latency, and error rates.

• Build a Grafana dashboard tailored to your SLOs and link it to Jaeger.

When your agent misbehaves—and it will—you’ll be ready with data, not guesses.

Happy hacking!