Artificial Intelligence Observability Case Studies (For Linux Bash Users)

If your ML system works fine in staging but suddenly “goes weird” in production—hallucinating, slowing down, or drifting—you’re not alone. AI systems fail in ways that traditional services don’t, and the mean time to innocence is often hours too long. Observability turns these mysteries into measurable, actionable signals you can debug from a terminal.

This post explains why AI observability matters, then walks through four real-world case studies you can reproduce on Linux. You’ll get a minimal, Linux-native stack (Prometheus + Grafana + Jaeger), copy-paste commands for apt/dnf/zypper, and a few small code snippets you can adapt today.

Why AI observability is different (and valid)

• Data is half your system: Poor data quality or drift can cause silent failure even when your service is healthy.

• Latency distribution matters: P95 inference latency is driven by model loading, cold starts, or GPU scheduling—not just CPU or network.

• Black-box behavior needs white-box signals: You won’t debug hallucinations with CPU metrics. You need semantic/quality signals attached to spans, logs, and metrics.

• Cost and quality go together: Tokens, GPU seconds, and retrieval depth must be measured alongside success rates and user feedback.

• Regulations and trust: You may need to trace which sources contributed to an answer.

Observability for AI adds model-/data-aware signals to your existing reliability tools.

---

Minimal Linux-native observability stack

We’ll use:

• Prometheus for metrics

• Grafana for dashboards

• Jaeger for traces (works well with OpenTelemetry)

Below are installation instructions for apt, dnf, and zypper. Run as a sudo-capable user.

Install Prometheus

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y prometheus
sudo systemctl enable --now prometheus

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y prometheus
sudo systemctl enable --now prometheus

• openSUSE/SLES (zypper):

sudo zypper refresh
sudo zypper install -y prometheus
sudo systemctl enable --now prometheus

Verify: Prometheus UI at http://localhost:9090

Install Grafana

• Debian/Ubuntu (apt) via official Grafana repo:

sudo apt install -y software-properties-common apt-transport-https wget gpg
sudo mkdir -p /etc/apt/keyrings
wget -q -O- https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server

• Fedora/RHEL/CentOS (dnf):

sudo tee /etc/yum.repos.d/grafana.repo >/dev/null <<'EOF'
[grafana]
name=Grafana OSS
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
EOF
sudo dnf install -y grafana
sudo systemctl enable --now grafana-server

• openSUSE/SLES (zypper):

sudo rpm --import https://rpm.grafana.com/gpg.key
sudo zypper addrepo https://rpm.grafana.com grafana
sudo zypper refresh
sudo zypper install -y grafana
sudo systemctl enable --now grafana-server

Verify: Grafana at http://localhost:3000 (default login admin/admin; change it!)

Install Jaeger (all-in-one)

• Debian/Ubuntu (apt):

sudo apt update
sudo apt install -y jaeger-all-in-one
sudo systemctl enable --now jaeger-all-in-one

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y jaeger
sudo systemctl enable --now jaeger-all-in-one

• openSUSE/SLES (zypper):

sudo zypper install -y jaeger
sudo systemctl enable --now jaeger-all-in-one

Verify: Jaeger UI at http://localhost:16686

---

Quick wiring: scrape your app and send a trace

Configure Prometheus to scrape a demo app metrics endpoint on port 8000:

sudo tee /etc/prometheus/prometheus.yml >/dev/null <<'EOF'
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'ai-app'
    static_configs:
      - targets: ['localhost:8000']
EOF
sudo systemctl restart prometheus

Install Python tools (pip) if needed:

• Debian/Ubuntu (apt):

sudo apt install -y python3-pip

• Fedora/RHEL/CentOS (dnf):

sudo dnf install -y python3-pip

• openSUSE/SLES (zypper):

sudo zypper install -y python3-pip

Expose a couple of model-aware metrics:

python3 -m pip install --user prometheus-client
python3 - <<'PY'
from prometheus_client import start_http_server, Gauge, Histogram
import random, time

latency = Histogram('inference_latency_seconds', 'Model inference latency', buckets=[0.01,0.05,0.1,0.2,0.5,1,2,5])
drift_score = Gauge('drift_psi', 'Population Stability Index for a key feature')

start_http_server(8000)
while True:
    with latency.time():
        time.sleep(random.uniform(0.02,0.15))
    drift_score.set(random.uniform(0.0, 0.2))
    time.sleep(1)
PY

Send a sample trace to Jaeger using OpenTelemetry:

python3 -m pip install --user opentelemetry-sdk opentelemetry-exporter-otlp
python3 - <<'PY'
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import time, random

provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")  # Jaeger all-in-one with OTLP HTTP
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ai-demo")

with tracer.start_as_current_span("inference") as span:
    lat = random.uniform(35, 80)
    span.set_attribute("model.name", "resnet50")
    span.set_attribute("rag.doc_count", 4)
    span.set_attribute("quality.answer_score", 0.78)
    time.sleep(lat/1000.0)
PY

You now have:

• Metrics in Prometheus (query: inference_latency_seconds_bucket or drift_psi)

• A trace in Jaeger (service: ai-demo)

---

Case Study 1: Catching Feature Drift in a Support Bot

Problem: A customer-support NLP bot started giving off-topic suggestions after a dataset change. API was healthy; answers weren’t.

What we instrumented:

• Drift score (PSI) on a key categorical feature

• Volume of queries and fallback rate

How to reproduce on Linux:

• Install Evidently and export a PSI metric to Prometheus.

python3 -m pip install --user evidently prometheus-client pandas
python3 - <<'PY'
import pandas as pd
import numpy as np
from evidently.calculations.stattests import psi_stat_test
from prometheus_client import Gauge, start_http_server

# Demo: reference vs current category distributions
ref = pd.Series(np.random.choice(list("ABCDE"), size=5000, p=[.25,.25,.2,.2,.1]))
cur = pd.Series(np.random.choice(list("ABCDE"), size=5000, p=[.10,.10,.35,.35,.10]))

# Compute PSI
def psi(a, b):
    bins = sorted(set(a.unique()) | set(b.unique()))
    p = a.value_counts(normalize=True).reindex(bins, fill_value=0)
    q = b.value_counts(normalize=True).reindex(bins, fill_value=0)
    return float(((p - q) * (np.log((p + 1e-12)/(q + 1e-12)))).sum())

psi_val = psi(ref, cur)

g = Gauge('drift_psi', 'Population Stability Index for key feature')
start_http_server(8000)
g.set(psi_val)
print(f"PSI={psi_val:.3f}")
input("Press Enter to exit...")
PY

• Optional alert rule (drift > 0.2 for 10m):

sudo tee /etc/prometheus/rules-ai.yml >/dev/null <<'EOF'
groups:

- name: ai
  rules:
  - alert: FeatureDriftHigh
    expr: avg_over_time(drift_psi[10m]) > 0.2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Feature drift high (PSI > 0.2 for 10m)"
EOF

• Reference it in /etc/prometheus/prometheus.yml under rule_files: and restart Prometheus.

Impact:

• Before: Random support failures, hard to reproduce.

• After: 30-minute detection of upstream data shift; rollback to previous dataset fixed it.

---

Case Study 2: P95 Latency Spikes on Vision Inference

Problem: Sporadic 2–3s latency spikes in a CV service despite average latency ~80ms.

What we instrumented:

• inference_latency_seconds histogram with tight buckets

• GPU utilization (via exporter) and concurrent inference count

• Alert on P95 > 500ms for 5m

Action:

• Use Prometheus to compute P95 and correlate with GPU saturation.

curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,sum(rate(inference_latency_seconds_bucket[5m]))%20by%20(le))"

• Once identified, we reduced batch size during peak and pre-warmed model weights.

Impact:

• P95 down 62%, far fewer user-visible spikes.

Tip: If you run NVIDIA GPUs, run the nvidia/dcgm-exporter container and scrape its metrics; overlay P95 vs GPU mem/SM usage in Grafana.

---

Case Study 3: Hallucinations in a RAG Pipeline

Problem: Users reported fabricated facts in a Q&A system. Logs showed nothing unusual.

What we instrumented:

• Traces with attributes: rag.doc_count, rag.topk, tokens.input/output, retrieval.mrr, quality.answer_score (from human or heuristics)

• Span events containing snippet/source IDs

How to reproduce:

• The OpenTelemetry snippet you ran earlier already sends rag.doc_count and quality.answer_score. In a real app, set these on the inference span.

• In Jaeger UI, filter by quality.answer_score<0.6 and inspect spans with low doc_count or high top_k.

Action:

• We enforced a min doc_count, added a re-ranker, and penalized out-of-domain sources.

Impact:

• Hallucination reports dropped 40%; we also got a causal path for future regressions.

---

Case Study 4: Silent Data Pipeline Breakage

Problem: A schema change zeroed out a critical feature. The model served 200s with terrible decisions.

What we instrumented:

• Data validation at feature boundaries (non-null, within range) using Great Expectations

• Prometheus gauge feature_null_ratio and counter validation_failures_total

How to reproduce quickly:

python3 -m pip install --user great_expectations prometheus-client pandas
python3 - <<'PY'
import pandas as pd
import numpy as np
from prometheus_client import Gauge, Counter, start_http_server

df = pd.DataFrame({"credit_score": np.append(np.random.normal(680,50,990), [np.nan]*10)})

null_ratio = df['credit_score'].isna().mean()

g = Gauge('feature_null_ratio', 'Null ratio for credit_score')
c = Counter('validation_failures_total', 'Count of validation failures')
start_http_server(8000)
g.set(null_ratio)
if null_ratio > 0.01:
    c.inc()
print(f"null_ratio={null_ratio:.3f}")
input("Press Enter to exit...")
PY

Action:

• Fail the canary if null_ratio > 1%, block release, alert data engineers.

Impact:

• Prevented bad-serving windows; rollouts halted automatically with clear diagnostics.

---

Wire Grafana quickly

• Add Prometheus as a data source (http://localhost:9090).

• Create panels for:

  • histogram_quantile(0.95, sum(rate(inference_latency_seconds_bucket[5m])) by (le))
  • drift_psi
  • feature_null_ratio

• Add Jaeger panel link to jump from metrics to traces.

Grafana CLI example (optional) to install a common dashboard:

sudo grafana-cli plugins install grafana-polystat-panel
sudo systemctl restart grafana-server

---

Pro tips and gotchas

• Metric names: Use explicit units (seconds, bytes, tokens) and labels sparingly.

• Cardinality: Don’t label metrics with user IDs or request IDs. Put those in traces/logs instead.

• Sampling: Keep 100% sampling for error spans and a small % for normal traffic.

• Alerts: Alert on SLO violations (P95 latency, drift for a window) not single spikes.

• Privacy: Hash or redact PII before exporting signals.

---

Conclusion and Call To Action

AI systems don’t just need “is the service up?”—they need “is the data good, the model aligned, and the answers trustworthy?” With Prometheus, Grafana, and Jaeger, you can ship practical, Linux-native observability in an afternoon.

Next steps: 1. Install Prometheus, Grafana, and Jaeger using the commands above. 2. Expose two metrics today: inference latency histogram and one quality/drift metric. 3. Add a simple OpenTelemetry span around inference with semantic attributes. 4. Build one Grafana dashboard that correlates P95, drift, and error rate. 5. Write one alert that would have saved you from a past incident.

If you want a starter repo with the snippets above wired together, tell me your distro and I’ll generate a tailored bootstrap script.