Inspect traces

Follow conversations and tool calls across messaging, MCP, plugins and the CLI.

See which tools ran, how long they took and where a request failed. Flow can send runtime events to your own system or export traces to Flow Observe.

Trace a conversation

Use an EventSink to send events to your logs or observability system. Events include messages, turns, model calls, tool calls and schedules.

import type { EventSink } from "@flow-engineer/flow";

const log: EventSink = (event) => {
  console.log(event.type, event.conversationId, event.at);
};
// Pass events: [log] when creating the agent.

An event sink must not throw. jsonlSink(path) is available for writing events to a local file. A tool’s output on tool.end and task.done is whole up to 64,000 characters; a longer one is cut there and ends with the line [cut: N more characters not shown] (from Framework 0.14.5; before, it was cut at 4,000 with …). Trace text attributes keep their own cap of 32,768 characters.

To export OTLP/JSON traces, set the following on the agent’s host:

FLOW_FRAMEWORK_TRACES_KEY=your_trace_key
FLOW_FRAMEWORK_TRACES_URL=https://flow.engineer/observe

Metadata is exported by default. Set FLOW_FRAMEWORK_TRACES_CONTENT=1 to include message content. From Framework 0.8.1, flow setup writes that line, under a comment, when it adds a test traces key (otk_test_, the Staging dashboard) and .env does not set it already, so test chats can be read and graded there. Set it to 0 for metadata only. Setup never turns content on for a live key.

Traces go in batches of up to 64 turns. From Framework 0.16.1, a batch over 64 KB goes gzipped (Content-Encoding: gzip, as OTLP/HTTP allows), so a self-hosted endpoint must accept gzip. When a batch cannot be sent (an endpoint that refuses it, or no answer in 10 seconds), the agent logs one warning per kind of failure that says how many turns’ traces were not sent; the agent’s replies never wait on traces, and under flow test the results do not depend on them (only those conversations are missing from flow conversations and the dashboard). A turn on a channel the agent does not define, such as the test cloud’s Telegram run of an agent with no Telegram channel, is traced as the platform it came with (telegram), not other.

Scenario-test traces carry a test run marker so the dashboard can distinguish them from production. From Framework 0.13.0, flow test sends message content by default, since --synthetic already says the test files hold no real data; set FLOW_FRAMEWORK_TRACES_CONTENT=0 in the test’s environment to keep its traces metadata only. Production stays opt-in.

Channel and playground turns

From Framework 0.8.1:

The chinwag. prefix is the framework’s old name, kept on the wire for compatibility.

chinwag.channel is the platform the turn was on: whatsapp, sms, telegram, imessage, web (a custom UI), cli (the terminal), voice or other. It is the same on the root span and on every send <name> span; the channel’s own name is only in the span name. Flow Messaging’s fm.telegram() and fm.imessage() trace as telegram and imessage (before, web). A custom channel of your own names its platform with traceChannel; any other value traces as other.
A playground turn (npm start’s page, flow chat, agent.simulate(), agent.simulateVisitor()) has chinwag.via set to "playground" on its root span. So does a message to the playground person that intercept or a keyword answered, and one sent to them outside a turn. Real turns have no chinwag.via. Playground turns share the process, and so chinwag.run_kind, with real conversations; the dashboard keeps them apart from real traffic by this attribute.

Check it worked: send a message that calls a tool, then find the turn and its tool span in your trace backend. Raw event sinks may contain message and tool content regardless of the trace content flag; apply your own access and retention controls.

What started a turn

From Framework 0.13.0, every trace’s root span has flow.turn.trigger:

`flow.turn.trigger`	What started it
`person`	The person’s message, also one `intercept` or a keyword answered with no model call.
`nudge`	A prompt nudge: `agent.nudge()` or `recordNudge()` with `prompt`, also one with `at` when it comes due.
`event`	A business event (`flowEvents`): the agent phrasing it, or its template or text sent as written.
`wake_up`	A time that was set came: the agent’s wake-up (`wake_me_up`, `context.wakeUps`, `agent.wakeUp()`), or a reminder the person asked for.
`task`	A background tool’s result, or word that one was cut off by a restart.
`queued_message`	A message your code queued as written: a nudge with `text` or `template`, also one with `at`.
`direct`	Flow’s own message, sent as written: a rule’s or keyword’s answer, the fallback template, or word that a message was not delivered.

With content on, flow.turn.trigger_text is the instruction or result the turn began with: the nudge’s prompt, the wake-up’s note or the reminder’s instruction, the background result. It is capped like other text attributes. A turn cut off by a restart that resumes with nothing new has the trigger only when it was the person’s.

A message sent as written, with no model call (a nudge’s text or template, a fixed scheduled text, a rule’s answer), is always a trace of its own, even while a turn is open in that conversation. Its send <name> span has flow.message.as_written set to true, so a grader reads it as your code’s words, not the model’s reply.

Messages your code queued

When your code records a nudge while it runs in a conversation, the nudge keeps where it came from. For example, a tool in the seller’s conversation tells the buyer:

const answerBuyer = tool({
  name: "answer_buyer",
  description: "Passes the seller's answer on to the buyer.",
  input: z.object({ buyer: z.string(), text: z.string() }),
  run: async ({ buyer, text }) =>
    (await agent.recordNudge({ to: { channel: "whatsapp", address: buyer }, text: `The seller answered your question: ${text}` })).status,
});

The buyer’s trace for that message links to the tool call that queued it:

a span link from its root span to the tool call’s span (else the turn’s root span), with the link attribute flow.link.kind = queued_by;
the same as plain attributes on the root span, for backends that drop links: flow.origin.trace_id, flow.origin.span_id, flow.origin.conversation_id, flow.origin.tool_call_id and flow.origin.tool_name. Any that are not known are left out: without traces on, only the conversation and tool call are kept; a nudge from your server’s own code, outside any conversation, has none.

The origin is stored with the nudge, so it holds when the nudge waits for quiet hours, is sent later with at, or goes after a restart. It works the same for a prompt nudge: the turn it starts links to where it was queued. Nudges queued before the upgrade have no origin.

Check it worked: in a seller’s conversation, call a tool that nudges the buyer. In your trace backend, the buyer’s trace has flow.turn.trigger = queued_message, a send span with flow.message.as_written, and flow.origin.tool_name naming your tool; its link opens the seller’s trace at that tool call.

Messages the channel fails later

Flow Messaging can accept a send (202) that the channel then fails seconds later (message.failed): for example a WhatsApp template refused with error type permission and channel_code 131042 because the business account has no payment method. From Framework 0.15.6 (flow#199), such a failure is never dropped, even after the send resolved or for a person this process has not heard from:

every send span (send <channel>) carries flow.send.message_id, Flow Messaging’s msg_..., with content on or off;
the failure is a span delivery <channel> with error status, under its send span (same trace) when this process sent the message, else the root span of a trace of its own. It carries flow.delivery.message_id, flow.delivery.status = failed, flow.delivery.error_type, flow.delivery.channel_code (when the channel gave one), chinwag.channel and chinwag.conversation_id or chinwag.contact_id. Never content. Flow’s dashboard reads it to show the send as failed;
the agent logs it once at error level, with the message and conversation ids, the error type, the channel code and Flow’s explanation, never the message or the person’s address:

@flow-engineer/flow/flow-messaging: message msg_... on whatsapp (conversation conv_...) was not delivered (type permission, channel_code 131042): <Flow's message> <hint> (<doc_url>)

The agent is also told it was not delivered (see Delivery guarantees).

Check it worked: after a send that ended in message.failed, the log line appears once, and your trace backend has a delivery <channel> span with error status whose flow.delivery.message_id matches the send span’s flow.send.message_id.

Agent version

From Framework 0.10.3, every trace carries the agent’s version as the resource’s service.version, with no setting, so the dashboard can tell which version wrote a reply and list the versions it has seen. It is the first of:

traces: { version } in code;
FLOW_AGENT_VERSION;
AGENT_VERSION, which Flow’s hosting sets to the deploy’s version;
the CI’s commit: GITHUB_SHA, CI_COMMIT_SHA, BUILDKITE_COMMIT, CIRCLE_SHA1 or GIT_COMMIT;
the folder’s full git commit, <sha>+dirty when it has uncommitted changes;
package.json’s version.

flow test keeps the same label with its session and flow deploy with the deploy. A value that is not 1 to 100 letters, digits and . _ : / + - is never sent; a warning names where it was set. A folder without git (a container image without .git) falls through to the next source.

Check it worked: the agent logs flow: agent version <version> (from <source>) once when it starts, and agent.status().version returns it; the same value is the trace’s service.version.

Framework log lines

From Framework 0.16.0 (flow#207), the lines the framework writes itself (the runtime, the flow-messaging adapter and the playground) carry a severity your logs can filter on. What your own code prints is unchanged.

On Cloud Run (Flow’s hosting included; K_SERVICE or CLOUD_RUN_JOB is set), each line is one JSON entry Cloud Logging reads: severity (ERROR, WARNING, INFO or DEBUG), message, and the line’s own fields. An error’s stack is inside message, so a stack is one entry. A Cloud Logging query reads the words from jsonPayload.message.
Everywhere else (a terminal, flow dev, tests), the lines are plain text, written with console.error, console.warn, console.info or console.debug.
A failed tool call is logged in every environment, with no event sink: one line with the tool’s name, the kind of failure, the error’s code and status when it has them, the conversation and call ids, and how long it ran. Never the call’s input or output, or the error’s message. It is a WARNING, or an ERROR when the tool threw JavaScript’s own error for broken code (TypeError, ReferenceError, …). A call refused before it ran has the kind unknown_tool, invalid_json or invalid_input.
The flow-messaging stream’s first reconnect after a healthy connection (Flow closes connections about hourly, code 1006) is INFO. A refusal, or a close right after another, is a WARNING.

{"severity":"WARNING","message":"flow: tool refund failed after 212 ms (Error, not_refundable, 409) in conversation p_3f2a9c1d0b4e","tool":"refund","kind":"Error","code":"not_refundable","status":409,"conversation":"p_3f2a9c1d0b4e","callId":"call_1","ms":212}

Check it worked: make a hosted agent’s tool throw, then run npx @flow-engineer/flow logs --since 15m --json and look for a line with "severity":"WARNING" whose text starts flow: tool <name> failed.

Tool-call traces

MCP, plugin and CLI calls emit tool.start and tool.end events with their surface, client name and version, session, person, duration, outcome and error.

mcpServer uses the same FLOW_FRAMEWORK_TRACES_* environment settings. Explicit traces options override them; traces: false stops trace export without disabling event sinks.

What an MCP or CLI trace contains

Each call is a root trace without a conversation. Arguments and results are included only with FLOW_FRAMEWORK_TRACES_CONTENT=1. await handle.flushTraces() flushes the bounded background queue.

CLI calls send the command and an idempotency key in _meta; the caller can reuse its key on a retry. defineCli sends its version (default "1"). CLI arguments and trace credentials are not placed in _meta.

Legacy client initialization metadata is retained for at most 24 hours and 4,096 sessions. Bearer tokens are never cached.

Test a failure

Reproduce a conversation with a scenario test.

Run your own infrastructure

Configure persistence and restart behavior.

Was this page helpful?