Observability (OpenTelemetry)

Padwan AI ships opt-in OpenTelemetry instrumentation following the GenAI semantic conventions. It only depends on opentelemetry-api, behind the otel extra:

pip install "padwan-ai[otel]"

Quick start

from padwan_ai import otel

otel.instrument()  # uses the global tracer/meter/logger providers

instrument() wraps every provider client (OpenAI, Gemini, Mistral, Grok, Anthropic): chat completions and streams, batch operations, embeddings, realtime sessions, agent turns and tool execution, and MCP tool calls. It is idempotent; call otel.uninstrument() to restore the original methods.

OpenAIClient.complete() and OpenAIClient.stream() also create chat spans when called directly. Calls through complete_chat() or stream_chat() share the existing chat span. With content capture enabled, raw responses preserve each choice separately, including function and custom tool calls.

Content capture is disabled by default. Enabling it records prompts, responses, tool definitions, and tool arguments or results, which may contain sensitive data.

otel.instrument(
    tracer_provider=...,  # spans
    meter_provider=...,  # metrics
    logger_provider=...,  # exception events
    capture_content=True,  # sensitive content capture
)

Langfuse

The optional Langfuse adapter configures Padwan instrumentation and the Langfuse trace exporter together:

pip install "padwan-ai[langfuse]"

export LANGFUSE_PUBLIC_KEY="<PUBLIC_KEY>"
export LANGFUSE_SECRET_KEY="<SECRET_KEY>"
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"

import asyncio

from padwan_ai import LLMClient
from padwan_ai.langfuse import instrument


async def main() -> None:
    async with LLMClient(model="gpt-4o-mini") as client:
        await client.complete_chat([{"role": "user", "content": "Hello!"}])


with instrument() as telemetry:
    asyncio.run(main())

The returned LangfuseIntegration exposes the configured Langfuse client as telemetry.client. Its context manager flushes and shuts down the exporter, then restores the original Padwan methods. Call flush() only to send pending traces without shutting down; shutdown() is idempotent.

The adapter enriches the copy of each span sent to Langfuse while leaving its standard OpenTelemetry attributes unchanged for other exporters:

Langfuse reads the standard GenAI model, usage, and cost attributes directly. Inputs and outputs remain absent unless capture_content=True.

Enabling content capture may send prompts, responses, tool definitions, tool arguments, and tool results to Langfuse. These values can contain personal data, secrets, or proprietary context. Use Langfuse's mask_otel_spans hook to redact them; the adapter applies that hook before deriving Langfuse input and output attributes, so deleted source content is not recreated under an alias.

from langfuse.types import MaskOtelSpansParams, MaskOtelSpansResult, OtelSpanPatch

from padwan_ai.langfuse import instrument


def redact_inputs(*, params: MaskOtelSpansParams) -> MaskOtelSpansResult:
    return MaskOtelSpansResult(
        span_patches={
            span_id: OtelSpanPatch(delete_attributes=("gen_ai.input.messages",))
            for span_id in params.spans
        }
    )


telemetry = instrument(capture_content=True, mask_otel_spans=redact_inputs)

instrument() accepts Langfuse credentials and routing (public_key, secret_key, base_url), trace metadata (environment, release), delivery controls (sample_rate, timeout, flush_at, flush_interval), debug, an existing tracer_provider, and should_export_span. For tests, span_exporter (e.g. an InMemorySpanExporter) and httpx_client (e.g. an httpx.MockTransport) are handed to the Langfuse client as-is, so a test can assert on every attribute the integration would send without opening a socket. The credential arguments fall back to the standard Langfuse environment variables. A custom span filter is applied after the adapter includes Padwan spans.

The integration exports traces only. Padwan metrics and exception log events still require separately configured OpenTelemetry meter and logger providers. Start the Langfuse integration before using Padwan; if padwan_ai.otel.instrument() is already active, the adapter raises instead of silently attaching to a different provider. See the Langfuse OpenTelemetry integration for backend configuration and troubleshooting.

Chat spans

Each chat call emits one CLIENT span named chat <model> (or chat when no model is set). For streams, the span starts on first iteration and ends when the stream completes, errors, is cancelled, or is abandoned.

¹ Token accounting follows each provider's usage report: OpenAI-style APIs count reasoning tokens inside output_tokens; Gemini reports thought tokens outside candidatesTokenCount, so they are not part of gen_ai.usage.output_tokens. Anthropic does not report a separate count. Per the Anthropic-specific conventions, gen_ai.usage.input_tokens includes cache read/write tokens (which Anthropic's raw input_tokens excludes).

Content capture (opt-in)

instrument(capture_content=True) additionally records gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions when the provider API separates them, and gen_ai.tool.definitions on chat spans as semconv-shaped JSON strings. It also emits the structured gen_ai.client.inference.operation.details log event and records gen_ai.tool.call.arguments and gen_ai.tool.call.result on tool spans. Image and audio parts are captured as their type only, never their payload.

Agent spans

Each AgentSession.send() / .stream() turn emits an invoke_agent span covering all rounds; the chat and tool spans of the turn nest under it:

Each dispatched tool emits an execute_tool <name> child span:

Agent invocations also record dedicated duration, inference-call count, and tool-call count metrics. Tool executions record their own duration metric.

Embeddings, batch, realtime, and MCP

• Embeddings: MistralClient.fetch_embeddings emits an embeddings <model> span (gen_ai.operation.name=embeddings).
• Batch: batch operations (create_batch, get_batch, list_batches, cancel_batch, and the OpenAI/Grok file helpers) emit a span named after the operation. No model attribute is set — batch requests carry their own per-request models.
• Realtime: a realtime <model> span covers the whole RealtimeClient session, from connect to close, with connect failures recorded as errors.
• MCP: initialization, tool-list refresh, ping, and direct tool calls emit CLIENT spans, named after mcp.method.name (tool calls use tools/call <name>). An MCP call dispatched through an agent enriches the existing execute_tool span instead of creating a duplicate span.

Metrics

Every instrument advises the bucket boundaries the GenAI semantic conventions recommend — the SDK default ladder starts at 5s and would collapse every GenAI latency into a single bucket, making quantiles meaningless.

Token metrics are recorded only for model calls. invoke_agent reports aggregate usage as span attributes to avoid double counting.

Exception events

Failed GenAI client operations additionally emit a gen_ai.client.operation.exception log event (severity WARN) through the logs API, correlated to the failing span and carrying exception.type, exception.message, exception.stacktrace, and the request attributes.

Semconv coverage

Detailed status per section of the GenAI semantic conventions:

Local dev stack

just e2e-otel runs the e2e suite instrumented and exports traces and metrics to a local otel-lgtm container (Grafana on :3000), provisioned with a GenAI dashboard from bin/observability/dashboards/. just e2e-langfuse does the same through the Langfuse adapter against a local Langfuse on :3001, headless-initialised with a dev project and seeded model prices. Both stacks live in one compose file (bin/observability/docker-compose.yml) behind the otel and langfuse profiles.