> ## Documentation Index
> Fetch the complete documentation index at: https://flow.engineer/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace your agent framework

> Attach model and tool spans to your conversation's turn trace.

Every framework below works the same way with Flow Observe:

1. **Set up the exporter** as on [Send traces from any agent](https://flow.engineer/docs/unified-messaging-api/observe/traces.md#1-set-up-the-exporter).
2. **Wrap each turn in your own `agent.turn` span** with the `flow.*` attributes, as in [step 2](https://flow.engineer/docs/unified-messaging-api/observe/traces.md#2-wrap-each-turn-in-a-span). Set `flow.agent_reply` from the text you actually send, not from the framework's output.
3. **Get the framework's spans into that trace**, so Observe shows the model calls, tools and errors under the turn. This is the only part that differs per framework.

Step 3 is optional: your `agent.turn` span alone is enough for conversations and grading. If the framework's spans end up in traces of their own, they are kept but not shown under any conversation. Do not copy the `flow.*` attributes onto those traces: Observe reads every trace with a `flow.conversation_id` as a turn of its own.

Frameworks record prompts and outputs in their spans by default. Observe stores them encrypted and does not show them today. If you would rather not send them, each section says how to turn recording off.

## OpenAI Agents SDK

The SDK has its own tracing, which by default sends traces to OpenAI's traces dashboard. It does not export OpenTelemetry itself. The [OpenInference](https://github.com/Arize-ai/openinference) instrumentation turns its traces into OpenTelemetry spans, nested under your current span, and by default **replaces** the export to OpenAI.

```python Python
# pip install openai-agents openinference-instrumentation-openai-agents opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
from agents import Agent, Runner
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor
from tracing import provider, tracer  # the setup from step 1

OpenAIAgentsInstrumentor().instrument(tracer_provider=provider)  # replaces the export to OpenAI
agent = Agent(name="Support", instructions="You help customers of Asha's Bakery. Keep answers short.")

async def handle_message(thread_id: str, message_id: str, text: str) -> None:
    with tracer.start_as_current_span("agent.turn") as turn:
        turn.set_attribute("flow.conversation_id", thread_id)
        turn.set_attribute("flow.message_id", message_id)
        turn.set_attribute("flow.user_message", text)
        result = await Runner.run(agent, text)
        reply = str(result.final_output)
        send(thread_id, reply)                      # your channel's send
        turn.set_attribute("flow.agent_reply", reply)
```

```ts TypeScript
// npm install @openai/agents @arizeai/openinference-instrumentation-openai-agents
import * as agents from "@openai/agents";
import { OpenAIAgentsInstrumentation } from "@arizeai/openinference-instrumentation-openai-agents";
import { provider, tracer } from "./tracing"; // the setup from step 1

new OpenAIAgentsInstrumentation({ tracerProvider: provider }).manuallyInstrument(agents); // replaces the export to OpenAI
const agent = new agents.Agent({ name: "Support", instructions: "You help customers of Asha's Bakery. Keep answers short." });

export async function handleMessage(threadId: string, messageId: string, text: string) {
  await tracer.startActiveSpan("agent.turn", async (turn) => {
    try {
      turn.setAttributes({ "flow.conversation_id": threadId, "flow.message_id": messageId, "flow.user_message": text });
      const result = await agents.run(agent, text);
      const reply = String(result.finalOutput);
      await send(threadId, reply); // your channel's send
      turn.setAttribute("flow.agent_reply", reply);
    } finally {
      turn.end();
    }
  });
}
```

* OpenInference names spans its own way, so Observe shows these spans with their names, timings and errors, but leaves the model and token columns empty for them today.
* Do not turn the SDK's tracing off (`OPENAI_AGENTS_DISABLE_TRACING`, `set_tracing_disabled`): that stops the instrumentation too. To keep sending to OpenAI as well, pass `exclusive_processor=False` (Python) or `manuallyInstrument(agents, { exclusiveProcessor: false })` (TypeScript).
* To leave prompts and outputs out of the spans: `RunConfig(trace_include_sensitive_data=False)` (Python) or `traceIncludeSensitiveData: false` (TypeScript).

## LangChain and LangGraph

Use the [OpenInference](https://github.com/Arize-ai/openinference) instrumentation. It sends LangChain's and LangGraph's runs as OpenTelemetry spans through your provider, nested under your current span, and never contacts LangSmith as long as `LANGSMITH_TRACING` (and the older `LANGCHAIN_TRACING_V2`) stay unset.

```python Python
# pip install openinference-instrumentation-langchain opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
from openinference.instrumentation.langchain import LangChainInstrumentor
from tracing import provider, tracer  # the setup from step 1

LangChainInstrumentor().instrument(tracer_provider=provider)

def handle_message(thread_id: str, message_id: str, text: str) -> None:
    with tracer.start_as_current_span("agent.turn") as turn:
        turn.set_attribute("flow.conversation_id", thread_id)
        turn.set_attribute("flow.message_id", message_id)
        turn.set_attribute("flow.user_message", text)
        state = graph.invoke(
            {"messages": [("user", text)]},
            config={"configurable": {"thread_id": thread_id}},
        )  # your chain or graph
        reply = state["messages"][-1].content
        send(thread_id, reply)                      # your channel's send
        turn.set_attribute("flow.agent_reply", reply)
```

```ts TypeScript
// npm install @arizeai/openinference-instrumentation-langchain
// (version 4 is for LangChain 1.0 and later; use @arizeai/openinference-instrumentation-langchain-v0 for older versions)
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { LangChainInstrumentation } from "@arizeai/openinference-instrumentation-langchain";
import { provider } from "./tracing"; // the setup from step 1

new LangChainInstrumentation({ tracerProvider: provider }).manuallyInstrument(CallbackManagerModule);
// then call your chain or graph inside tracer.startActiveSpan("agent.turn", ...), as in step 2
```

As with the OpenAI Agents SDK, Observe shows OpenInference spans with their names, timings and errors, with the model and token columns empty today.

Already on LangSmith? Its OpenTelemetry mode can send to Observe instead: `pip install "langsmith[otel]"`, then set `LANGSMITH_TRACING=true`, `LANGSMITH_OTEL_ENABLED=true` and **`LANGSMITH_OTEL_ONLY=true`** (without the last one, traces go to LangSmith too), with the provider from step 1 set before LangChain starts.

## Vercel AI SDK

The AI SDK's spans use the global OpenTelemetry provider and nest under your current span. Telemetry must be turned on, and the option's name depends on your version:

| AI SDK  | Turn on telemetry                                                                                                                                                                                                                           |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 5 and 6 | `experimental_telemetry: { isEnabled: true, functionId: "reply" }` on each call                                                                                                                                                             |
| 7       | Once at startup: `registerTelemetry(new OpenTelemetry())` (`registerTelemetry` from `ai`, `OpenTelemetry` from `@ai-sdk/otel`). Without it, AI SDK 7 makes no spans, whatever the call says. Per call: `telemetry: { functionId: "reply" }` |

```ts TypeScript
import { generateText } from "ai";
import { tracer } from "./tracing"; // the setup from step 1 (or registerOTel in Next.js, below)

export async function handleMessage(threadId: string, messageId: string, text: string) {
  await tracer.startActiveSpan("agent.turn", async (turn) => {
    try {
      turn.setAttributes({ "flow.conversation_id": threadId, "flow.message_id": messageId, "flow.user_message": text });
      const { text: reply } = await generateText({
        model, // your model
        messages: [{ role: "user", content: text }],
        experimental_telemetry: { isEnabled: true, functionId: "reply" }, // AI SDK 5 and 6; on 7: telemetry: { functionId: "reply" }
      });
      await send(threadId, reply); // your channel's send
      turn.setAttribute("flow.agent_reply", reply);
    } finally {
      turn.end();
    }
  });
}
```

In Next.js, register the exporter in `instrumentation.ts` instead of step 1:

```ts instrumentation.ts
import { registerOTel, OTLPHttpProtoTraceExporter } from "@vercel/otel";

export function register() {
  registerOTel({
    serviceName: "support-agent",
    traceExporter: new OTLPHttpProtoTraceExporter({
      url: "https://flow.engineer/observe/v1/traces",
      headers: { Authorization: `Bearer ${process.env.FLOW_OBSERVE_TRACE_KEY}` },
    }),
  });
  // AI SDK 7 only: registerTelemetry(new OpenTelemetry());
}
```

* With `streamText`, end the `agent.turn` span only after the stream has been read to the end and sent.
* Do not also set `OTEL_EXPORTER_OTLP_ENDPOINT` when you pass a `traceExporter`: `@vercel/otel` would export through both and send every span twice.
* The AI SDK records prompts and outputs by default. To leave them out: `recordInputs: false, recordOutputs: false` in the telemetry options.
* On AI SDK 5 and 6 you can also pass the ids as telemetry `metadata` (`metadata: { "flow.conversation_id": threadId }`); Observe reads them there. AI SDK 7 removed `metadata`, so the `agent.turn` span is the way that works on every version.

## Claude Agent SDK

The SDK runs Claude Code, which exports its own spans over OpenTelemetry when you turn it on through the SDK's `env` option (a beta feature). Run `query()` inside your `agent.turn` span and the SDK passes your trace context on, so its spans join your trace. Needs `claude-agent-sdk` 0.1.60 or later installed as `claude-agent-sdk[otel]` (Python), or `@anthropic-ai/claude-agent-sdk` 0.2.113 or later (TypeScript).

```python Python
# pip install "claude-agent-sdk[otel]" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
import os
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage
from tracing import tracer  # the setup from step 1

OBSERVE_ENV = {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
    "OTEL_TRACES_EXPORTER": "otlp",
    "OTEL_METRICS_EXPORTER": "none",
    "OTEL_LOGS_EXPORTER": "none",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "https://flow.engineer/observe/v1/traces",
    "OTEL_EXPORTER_OTLP_TRACES_HEADERS": f"Authorization=Bearer%20{os.environ['FLOW_OBSERVE_TRACE_KEY']}",
}

async def handle_message(thread_id: str, message_id: str, text: str) -> None:
    with tracer.start_as_current_span("agent.turn") as turn:
        turn.set_attribute("flow.conversation_id", thread_id)
        turn.set_attribute("flow.message_id", message_id)
        turn.set_attribute("flow.user_message", text)
        reply = None
        async for message in query(prompt=text, options=ClaudeAgentOptions(env=OBSERVE_ENV)):
            if isinstance(message, ResultMessage) and message.subtype == "success" and not message.is_error:
                reply = message.result
        if reply:
            send(thread_id, reply)                  # your channel's send
            turn.set_attribute("flow.agent_reply", reply)
```

```ts TypeScript
import { query } from "@anthropic-ai/claude-agent-sdk";
import { tracer } from "./tracing"; // the setup from step 1

const observeEnv = {
  ...process.env, // options.env replaces the environment, so keep yours
  CLAUDE_CODE_ENABLE_TELEMETRY: "1",
  CLAUDE_CODE_ENHANCED_TELEMETRY_BETA: "1",
  OTEL_TRACES_EXPORTER: "otlp",
  OTEL_METRICS_EXPORTER: "none",
  OTEL_LOGS_EXPORTER: "none",
  OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://flow.engineer/observe/v1/traces",
  OTEL_EXPORTER_OTLP_TRACES_HEADERS: `Authorization=Bearer%20${process.env.FLOW_OBSERVE_TRACE_KEY}`,
};

export async function handleMessage(threadId: string, messageId: string, text: string) {
  await tracer.startActiveSpan("agent.turn", async (turn) => {
    try {
      turn.setAttributes({ "flow.conversation_id": threadId, "flow.message_id": messageId, "flow.user_message": text });
      let reply: string | undefined;
      for await (const m of query({ prompt: text, options: { env: observeEnv } })) {
        if (m.type === "result" && m.subtype === "success") reply = m.result;
      }
      if (reply) {
        await send(threadId, reply); // your channel's send
        turn.setAttribute("flow.agent_reply", reply);
      }
    } finally {
      turn.end();
    }
  });
}
```

* Its spans are named `claude_code.interaction`, `claude_code.llm_request` and `claude_code.tool`. Observe shows them with their timings and errors; they use Claude Code's own attribute names, so model and token columns stay empty for them today.
* Prompts and responses are left out of these spans unless you set `OTEL_LOG_USER_PROMPTS=1` or `OTEL_LOG_ASSISTANT_RESPONSES=1`. You do not need them: `flow.user_message` and `flow.agent_reply` carry the text.
* Claude Code batches spans and fails quietly. Set `CLAUDE_CODE_OTEL_DIAG_STDERR=1` to see export errors. Never use the `console` exporter: standard output is the SDK's message channel.

## Mastra

Mastra's tracing has its own OpenTelemetry exporter, `@mastra/otel-exporter`. It does not join your active span by itself: pass your turn span's trace id and span id in `tracingOptions`, and Mastra's spans nest under it. Your `agent.turn` span is still exported by your own setup from step 1.

```ts TypeScript
// npm install @mastra/core @mastra/observability @mastra/otel-exporter @opentelemetry/exporter-trace-otlp-proto
import { Mastra } from "@mastra/core/mastra";
import { Observability } from "@mastra/observability";
import { OtelExporter } from "@mastra/otel-exporter";
import { tracer } from "./tracing"; // the setup from step 1
import { supportAgent } from "./agents"; // your agent

export const mastra = new Mastra({
  agents: { supportAgent },
  observability: new Observability({
    configs: {
      observe: {
        serviceName: "support-agent",
        exporters: [
          new OtelExporter({
            provider: {
              custom: {
                endpoint: "https://flow.engineer/observe/v1/traces",
                protocol: "http/protobuf",
                headers: { Authorization: `Bearer ${process.env.FLOW_OBSERVE_TRACE_KEY}` },
              },
            },
            signals: { logs: false }, // traces only
          }),
        ],
      },
    },
  }),
});

export async function handleMessage(threadId: string, messageId: string, text: string) {
  await tracer.startActiveSpan("agent.turn", async (turn) => {
    try {
      turn.setAttributes({ "flow.conversation_id": threadId, "flow.message_id": messageId, "flow.user_message": text });
      const { traceId, spanId } = turn.spanContext();
      const result = await mastra.getAgent("supportAgent").generate(text, {
        tracingOptions: { traceId, parentSpanId: spanId }, // Mastra's spans join this turn's trace
      });
      await send(threadId, result.text); // your channel's send
      turn.setAttribute("flow.agent_reply", result.text);
    } finally {
      turn.end();
    }
  });
}
```

* Put the `flow.*` attributes on your `agent.turn` span. Observe also reads them from Mastra's `metadata` (exported as `mastra.metadata.flow.*`), but Mastra exports its spans apart from yours, so keep the ids on the turn span with the texts.
* `signals: { logs: false }` keeps the exporter from also posting logs.
* In a serverless function, `await mastra.observability.flush()` before returning, besides flushing your own provider.
* To leave prompts and outputs out of Mastra's spans: `tracingOptions: { hideInput: true, hideOutput: true }`.

## No framework: the model SDK called directly

> **Direct model SDK examples**
>
> If your agent calls a model SDK itself, with no framework around it, the model calls still belong inside your `agent.turn` span.
>
> ### Python, OpenAI SDK
>
> Install `opentelemetry-instrumentation-openai-v2` and call `OpenAIInstrumentor().instrument()` once at startup. Each model call then makes a span with the model and token counts, and its spans land inside your `agent.turn` span.
>
> ```python Python
> # pip install openai opentelemetry-instrumentation-openai-v2 opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
> from openai import OpenAI
> from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
> from tracing import provider, tracer  # the setup from step 1
>
> OpenAIInstrumentor().instrument(tracer_provider=provider)  # once, at startup
> client = OpenAI()
>
> def handle_message(thread_id: str, message_id: str, text: str) -> None:
>     with tracer.start_as_current_span("agent.turn") as turn:
>         turn.set_attribute("flow.conversation_id", thread_id)
>         turn.set_attribute("flow.message_id", message_id)
>         turn.set_attribute("flow.user_message", text)
>         response = client.chat.completions.create(
>             model=MODEL,  # your model
>             messages=[{"role": "user", "content": text}],
>         )
>         reply = response.choices[0].message.content
>         send(thread_id, reply)                      # your channel's send
>         turn.set_attribute("flow.agent_reply", reply)
> ```
>
> * It traces Chat Completions (`client.chat.completions.create`, sync and async) and embeddings. Calls to the Responses API (`client.responses.create`) make no spans with it: add those spans yourself, as below.
> * It leaves prompts and completions out unless you set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`. You do not need them: `flow.user_message` and `flow.agent_reply` carry the text.
>
> ### Any other SDK or language
>
> Add one child span per model call yourself, named `chat <model>`, with these attributes:
>
> | Attribute                    | Value                                    |
> | ---------------------------- | ---------------------------------------- |
> | `gen_ai.operation.name`      | `chat`                                   |
> | `gen_ai.request.model`       | the model you called                     |
> | `gen_ai.usage.input_tokens`  | input tokens, from the response's usage  |
> | `gen_ai.usage.output_tokens` | output tokens, from the response's usage |
>
> Then add one child span per tool call, named after the tool and never after message text, with `gen_ai.operation.name` = `execute_tool` and `gen_ai.tool.name` = the tool's name.
>
> ```python Python
> from tracing import tracer  # the setup from step 1
>
> # inside your agent.turn span:
> with tracer.start_as_current_span(f"chat {MODEL}") as span:
>     span.set_attribute("gen_ai.operation.name", "chat")
>     span.set_attribute("gen_ai.request.model", MODEL)
>     response = call_model(MODEL, messages)  # your model SDK
>     span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
>     span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
>
> with tracer.start_as_current_span(tool_name) as span:  # e.g. "lookup_order"
>     span.set_attribute("gen_ai.operation.name", "execute_tool")
>     span.set_attribute("gen_ai.tool.name", tool_name)
>     result = run_tool(tool_name, args)  # your tool
> ```
>
> ```ts TypeScript
> import { tracer } from "./tracing"; // the setup from step 1
>
> // inside your agent.turn span:
> const response = await tracer.startActiveSpan(`chat ${MODEL}`, async (span) => {
>   try {
>     span.setAttributes({ "gen_ai.operation.name": "chat", "gen_ai.request.model": MODEL });
>     const r = await callModel(MODEL, messages); // your model SDK
>     span.setAttributes({
>       "gen_ai.usage.input_tokens": r.usage.inputTokens,
>       "gen_ai.usage.output_tokens": r.usage.outputTokens,
>     });
>     return r;
>   } finally {
>     span.end();
>   }
> });
>
> const result = await tracer.startActiveSpan(toolName, async (span) => { // e.g. "lookup_order"
>   try {
>     span.setAttributes({ "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": toolName });
>     return await runTool(toolName, args); // your tool
>   } finally {
>     span.end();
>   }
> });
> ```
>
> ### What Observe reads
>
> * The model column: `gen_ai.response.model`, or `gen_ai.request.model` when the response's model is not set.
> * The token columns: `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. The older `gen_ai.usage.prompt_tokens` and `gen_ai.usage.completion_tokens` work too.
> * Only on model-call spans: `gen_ai.operation.name` must be `chat`, `text_completion`, `generate_content` or `embeddings`. A span without it is shown with its name and timing, and its model and token columns stay empty.

## Another framework

Any framework that emits OpenTelemetry spans through the global provider works like the Vercel AI SDK above: run it inside your `agent.turn` span. One that has its own tracing system needs a bridge to OpenTelemetry, as the OpenAI Agents SDK and LangChain do. If you cannot get its spans into your trace, your `agent.turn` span is still enough for conversations and grading.

## Related

* [Send traces from any agent](https://flow.engineer/docs/unified-messaging-api/observe/traces.md)
* [On Flow Messaging](https://flow.engineer/docs/unified-messaging-api/observe/flow-messaging.md)
* [Agent frameworks on Flow Messaging](https://flow.engineer/docs/unified-messaging-api/frameworks/vercel-ai-sdk.md)