AI Observability Architecture: LLM Tracing, Evaluation & Monitoring (2026)

Traditional APM tools were designed for request/response microservices: trace a call, measure latency, count errors. They were not designed for systems where a single user request might spawn three agents, make twelve tool calls, retrieve forty documents from a vector database, and consume 50,000 tokens — all before returning an answer that may or may not be accurate. AI observability architecture is the discipline of instrumenting, collecting, and analyzing the signals that matter for LLM-based systems: spans, token usage, prompt content, retrieval quality, generation faithfulness, latency per model call, and cost per response.

In 2026, this space standardized rapidly: OpenTelemetry published its GenAI semantic conventions, and LangChain, CrewAI, AutoGen, and major APM vendors (Datadog, Honeycomb, New Relic) now natively emit OTel spans for AI operations. This guide covers the architecture of a complete AI observability stack — from instrumentation to evaluation — with prompt templates for diagramming your observability layer in ArchitectureDiagram.ai.

Why AI observability is different from application observability

Standard observability (logs, metrics, traces) captures what happened at the infrastructure level. AI observability must also capture what the model was asked, what it returned, whether the return was correct, and why it behaved differently between two semantically identical requests. The key signals that are absent from standard APM:

• Token counts: Input tokens, output tokens, and cached tokens per call — the primary cost driver and a proxy for prompt efficiency.
• Model version: Which exact model version responded to this request. claude-sonnet-4-6 and a fine-tuned variant may have identical latency but very different output quality.
• Finish reason: Did the model stop because it finished, hit the max token limit, triggered a safety filter, or encounter a tool-call loop? Each finish reason indicates a different failure mode.
• Retrieval quality signals: For RAG systems: which documents were retrieved, what similarity scores did they have, and did the final answer actually use them (faithfulness)?
• Tool call graph: In agentic systems, the sequence and duration of tool calls is the primary performance and correctness signal.
• Prompt content: The actual prompt and completion text, needed for debugging and fine-tuning. Sensitive — requires access controls and retention policies separate from standard logs.

OpenTelemetry GenAI semantic conventions

The OpenTelemetry GenAI semantic conventions define a standard schema for spans produced by LLM calls, agent operations, and retrieval operations. Published in 2026, they are the equivalent of the HTTP semantic conventions for AI workloads. Key span attributes:

These attributes are emitted as OpenTelemetry spans. In your architecture diagram, show the OTel instrumentation layer as a cross-cutting concern — each service in your AI stack emits spans to an OTel collector, which routes them to your observability backend (Datadog, Honeycomb, Jaeger, Grafana Tempo). The span hierarchy mirrors your agent topology: a root span for the user request, child spans for each agent, grandchild spans for each LLM call and tool invocation within that agent.

The AI observability stack layers

Instrumentation layer

Auto-instrumentation libraries (opentelemetry-instrumentation-anthropic, opentelemetry-instrumentation-openai, LangChain callbacks) add span emission to LLM calls without code changes. For custom agents or tools, add manual spans around tool call execution and retrieval operations. The instrumentation layer should emit spans to an OTel collector sidecar (not directly to the backend) so you can change observability backends without redeploying your AI services.

Trace collection and routing layer

The OTel collector receives spans from all services, applies sampling, enriches with resource attributes (environment, service version, team), and routes traces to one or more backends. For AI workloads, configure a higher sampling rate for failed requests, slow requests (p99 latency outliers), and requests with abnormal token counts — since these are the ones worth investigating. Route full prompt content to a separate secure trace store (not the general APM backend) with appropriate access controls.

Evaluation pipeline

Tracing tells you what happened. Evaluation tells you whether it was good. An automated evaluation pipeline runs alongside (or slightly behind) production traffic, scoring a sample of responses on dimensions like:

• Faithfulness: Does the response only contain claims that can be grounded in the retrieved documents? (Critical for RAG systems.)
• Answer relevance: Does the response actually answer the user's question?
• Context relevance: Were the retrieved documents relevant to the question? (A low score here means your retrieval is the problem, not your generation.)
• Task completion: For agentic tasks: did the agent actually complete the assigned task?
• Safety: Did the response contain harmful, biased, or off-policy content?

Evaluation scores are metric time series — emit them as OTel metrics or custom events to your observability backend so you can alert on faithfulness drops (which indicate a retrieval regression) or safety spikes (which indicate prompt injection attempts).

Cost attribution layer

Token counts from gen_ai.usage.* spans are the raw material for cost attribution. A cost attribution pipeline multiplies token counts by the current $/token rate for each model, tags costs by team, feature, and user segment, and writes cost metrics to a FinOps dashboard. This pipeline typically runs as a stream processor on the OTel span stream (using Flink or a cloud streaming service), not as a separate API call to the billing endpoint, to avoid latency and rate limits.

Prompt templates for AI observability architecture diagrams

Full-stack LLM observability with OTel and Langfuse

Agentic system observability with tool call tracing

AI observability tools comparison

Frequently asked questions

What is AI observability?

AI observability is the practice of monitoring, tracing, and evaluating LLM-based systems in production. It extends traditional application observability (logs, metrics, distributed traces) with AI-specific signals: token usage, model versions, prompt content, retrieval quality, generation faithfulness, and task completion rates. The goal is to understand not just whether a request succeeded (infrastructure concern) but whether the AI system produced a correct, safe, and useful response (application concern) — a distinction that standard APM tools were not built to make.

What are OpenTelemetry GenAI semantic conventions?

The OpenTelemetry GenAI semantic conventions are a standardized set of span attribute names and values for AI operations, published by the OpenTelemetry project in 2026. They define how to name and tag spans for LLM calls (gen_ai.operation.name = chat), token counts (gen_ai.usage.input_tokens), model identifiers, finish reasons, and more. Standardizing on these conventions means that any OTel-compatible observability backend can display AI traces without custom parsers — and that switching backends doesn't require re-instrumenting your codebase.

What is the difference between AI observability and RAG evaluation?

AI observability encompasses the full instrumentation and monitoring stack for AI systems — traces, metrics, logs, alerting, cost attribution. RAG evaluation is one component of observability, focused specifically on measuring the quality of retrieval-augmented generation outputs: context relevance (did we retrieve the right documents?), faithfulness (does the answer reflect the retrieved documents?), and answer relevance (does the answer address the question?). A complete AI observability architecture includes RAG evaluation as one of its evaluation pipeline components alongside agent task completion scoring and safety checks.

Related guides: OpenTelemetry architecture diagrams, RAG architecture diagrams, Claude Agent SDK architecture, agentic AI security architecture, and platform engineering diagrams.