Posted on
• Artificial Intelligence

Artificial Intelligence OpenTelemetry Workflows

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Artificial Intelligence OpenTelemetry Workflows: Trace Your LLM Pipelines from Bash

If your AI workflow feels like a black box, you’re not alone. You ship a prompt, wait, and hope the output is accurate and fast. But what about latency spikes? Prompt regressions? Model drift? Hidden cost explosions? OpenTelemetry (OTel) turns guesswork into data, letting you observe AI pipelines end-to-end—from shell scripts to model servers—without locking into a vendor.

This guide shows how to build AI observability with OpenTelemetry from a Linux Bash workflow. You’ll install a minimal toolchain, run an OpenTelemetry Collector locally, and instrument a simple AI step (with privacy-friendly redaction) so you can see actionable traces and attributes right away.

Why OpenTelemetry for AI Pipelines

  • Vendor-neutral, open standard. Decouple your code from specific APMs. Emit OTLP once, ship anywhere (Tempo, Jaeger, DataDog, Elastic, etc.).

  • It fits multi-hop pipelines. Traces can connect data prep, model inference, and post-processing across processes and languages.

  • Attribute-rich spans. Add model, prompt version, token usage, and cost—then sample or redact safely at the collector.

  • Fewer surprises in prod. Catch regressions: “This model/version on this route increased p95 latency and cost for prompt group X.”

What we’ll build

  • A local OpenTelemetry Collector that receives traces via OTLP and logs them (no external backend required).

  • Bash-friendly environment variables to standardize metadata.

  • A minimal Python example that represents a model call and emits spans to the collector.

  • Practical patterns for safe prompt tracking, error handling, and sampling.

1) Install prerequisites

We’ll use curl, tar, and Python to send traces from a simple AI step. Choose commands for your distro.

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y curl tar jq python3 python3-venv python3-pip
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y curl tar jq python3 python3-pip
  • openSUSE/SLE (zypper):
sudo zypper install -y curl tar jq python3 python3-pip

Create and activate a virtual environment (recommended):

python3 -m venv ~/.venvs/ai-otel
source ~/.venvs/ai-otel/bin/activate

Install Python packages for OpenTelemetry and HTTP:

pip install --upgrade pip
pip install "opentelemetry-sdk>=1.24.0" "opentelemetry-exporter-otlp>=1.24.0" \
            "opentelemetry-instrumentation-requests>=0.45b0" requests

Tip: If you prefer system Python without venvs, run pip with --user.

2) Install and run the OpenTelemetry Collector

The collector is a single binary that receives telemetry and exports it elsewhere. We’ll export to a simple logging sink so you can see spans in your terminal.

Option A: Download the official binary (works across distros):

# Replace version/arch as needed; example for linux-amd64
export OTELCOL_VERSION=0.104.0
curl -L -o otelcol.tar.gz \
  https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol_${OTELCOL_VERSION}_linux_amd64.tar.gz
mkdir -p ~/otelcol && tar -xzf otelcol.tar.gz -C ~/otelcol
~/otelcol/otelcol --version

Create a minimal config file:

cat > ~/otelcol/config.yaml << 'EOF'
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  attributes:
    actions:
      - key: ai.prompt
        action: hash  # redact prompt text via hashing at the collector
      - key: ai.user_id
        action: delete # drop direct identifiers

exporters:
  logging:
    loglevel: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes, batch]
      exporters: [logging]
EOF

Run the collector in the foreground:

~/otelcol/otelcol --config ~/otelcol/config.yaml

Keep it running in this terminal, or background it with tmux/screen/systemd.

Option B: Try your distro’s package manager (package names vary; if not found, use Option A).

  • Debian/Ubuntu (apt):
sudo apt update
sudo apt install -y opentelemetry-collector || sudo apt install -y otelcol-contrib || true
  • Fedora/RHEL/CentOS (dnf):
sudo dnf install -y opentelemetry-collector || true
  • openSUSE/SLE (zypper):
sudo zypper install -y opentelemetry-collector || true

Note: If the package isn’t available, fall back to the binary method above.

3) Export baseline environment variables in Bash

These standardize service metadata and route all spans to your local collector.

export OTEL_SERVICE_NAME=ai-workflow
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=dev,team=mlops"
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"  # collector gRPC

You can put these in your shell rc file to persist:

echo 'export OTEL_SERVICE_NAME=ai-workflow' >> ~/.bashrc
echo 'export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=dev,team=mlops"' >> ~/.bashrc
echo 'export OTEL_EXPORTER_OTLP_PROTOCOL=grpc' >> ~/.bashrc
echo 'export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"' >> ~/.bashrc

4) Instrument a simple AI step (Python) and run it from Bash

We’ll create a minimal Python script that simulates a model call. You’ll see:

  • Span timing for “generate”

  • Attributes for model, prompt hash, temperature

  • Error handling surfaced in the span

Create ai_generate.py:

cat > ai_generate.py << 'PY'
import hashlib
import json
import os
import time
import requests

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Configure the tracer provider using env vars like OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES
resource = Resource.create({})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# Inputs you might pass from Bash
prompt = os.environ.get("AI_PROMPT", "Explain zero-shot prompting in one sentence.")
model = os.environ.get("AI_MODEL", "llama3:8b")
temperature = float(os.environ.get("AI_TEMPERATURE", "0.2"))

def hash_text(s: str) -> str:
    return hashlib.sha256(s.encode("utf-8")).hexdigest()[:16]

# Example: Hit a local Ollama endpoint if available; else simulate work
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434/api/generate")
payload = {"model": model, "prompt": prompt, "options": {"temperature": temperature}}

with tracer.start_as_current_span("ai.generate") as span:
    # Add rich attributes. Note ai.prompt is redacted (hashed) at the collector.
    span.set_attribute("ai.model", model)
    span.set_attribute("ai.temperature", temperature)
    span.set_attribute("ai.prompt", prompt)  # collector will hash this key
    span.set_attribute("ai.prompt_hash", hash_text(prompt))
    span.set_attribute("ai.provider", "local-ollama")
    span.set_attribute("ai.route", "cli-demo")

    start = time.time()
    try:
        # Try a real call; if it fails, simulate latency and an error message
        resp = requests.post(OLLAMA_URL, json=payload, timeout=30)
        if resp.status_code == 200:
            data = resp.json()
            # Ollama streams tokens; a simple non-stream example may differ
            span.set_attribute("ai.output_size_bytes", len(json.dumps(data)))
            span.set_attribute("http.status_code", 200)
        else:
            span.set_status(Status(StatusCode.ERROR, f"HTTP {resp.status_code}"))
            span.set_attribute("http.status_code", resp.status_code)
            span.set_attribute("error.msg", resp.text[:256])
    except Exception as e:
        span.set_status(Status(StatusCode.ERROR, str(e)))
        # simulate some work so we see non-zero duration
        time.sleep(0.3)
    finally:
        duration_ms = (time.time() - start) * 1000
        span.set_attribute("ai.latency_ms", round(duration_ms, 2))
PY

Run it from Bash with inputs:

export AI_PROMPT="Summarize the pros and cons of RAG for customer support."
export AI_MODEL="llama3:8b"
export AI_TEMPERATURE="0.1"

python3 ai_generate.py

Back in the collector terminal, you’ll see logged spans with your attributes. Notice the collector hashed ai.prompt, protecting raw text while keeping it joinable by hash.

If you run Ollama locally, you’ll see real HTTP 200s and output sizes. If not, the script will still emit a span and surface an error in a controlled way—useful in CI or smoke tests.

5) Actionable patterns you can copy today

1) Standardize AI attributes

  • ai.model, ai.provider, ai.route (which code path or feature)

  • ai.prompt_hash (not raw prompt), ai.temperature

  • ai.latency_ms, ai.output_size_bytes, http.status_code, exit.code

  • Add resource attributes like deployment.environment, team, cost.center

2) Capture inputs/outputs safely

  • Hash or drop sensitive fields in the collector using the attributes processor.

  • Example: redact PII-like keys and keep a joinable hash for debugging:

processors:
  attributes:
    actions:
      - key: ai.prompt
        action: hash
      - key: user.email
        action: delete

3) Trace the whole pipeline, not just inference

  • Wrap data prep, reranker, post-processing, and vector store calls in spans.

  • Propagate context with environment variables when chaining scripts:

# create a root span in Python, then call a child Bash step that also emits spans
# Use W3C context; OpenTelemetry SDKs propagate automatically within-process.

4) Fail loudly but informatively

  • Record errors on spans with StatusCode.ERROR and short messages.

  • Add exit.code attributes in shell scripts:

#!/usr/bin/env bash
set -euo pipefail
start_ts=$(date +%s%3N)
your_cmd || exit_code=$?
: "${exit_code:=0}"
latency_ms=$(( $(date +%s%3N) - start_ts ))
echo "ai.latency_ms=${latency_ms},exit.code=${exit_code}"
# If wrapping with a language SDK or CLI, set span status on non-zero exit

5) Sample where it saves you money and noise

  • In dev, keep 100% sampling. In prod, sample by ai.route or error status.

  • Add tail_sampling to the collector (emit errors, sample 5–10% of OKs).

Real-world extension ideas

  • Send traces to Jaeger/Tempo: swap the logging exporter for OTLP to your APM.

  • Record token counts and estimated cost: add ai.tokens.prompt/ai.tokens.completion using your model’s metadata.

  • Split routes by feature flags or prompt versions so you can compare p95 latency and cost per version.

  • Add a metrics pipeline (OTel metrics) for request rate, errors, and cost per minute.

Troubleshooting

  • No spans in logs? Ensure the collector is running and OTEL_EXPORTER_OTLP_ENDPOINT matches 4317 for gRPC.

  • Python exporter errors? Upgrade packages and verify your virtualenv is active.

  • Attribute redaction didn’t apply? Confirm the key names in your attributes processor match exactly.

Conclusion and next step

Observability makes AI pipelines trustworthy. With a local OTel Collector, a few Bash env vars, and lightweight instrumentation, you can see latency, errors, and key AI attributes immediately—and do it safely with redaction and sampling.

Your next step:

  • Keep the collector running, instrument one more step in your pipeline (e.g., vector search or reranker), and add ai.route to differentiate it.

  • Point the collector to your preferred backend to visualize traces over time.

If this helped, turn it into a template in your repo and standardize ai.* attributes across services—future you (and your pager) will thank you.

Related posts