Skip to content
Flow Flow

Ask the docs

Answers come only from these docs and are written by an AI model (Gemini), so check the linked pages. Your question is not kept. Do not include personal details or keys.

Observe conversations

Overview · AI agent observability · chat agent monitoring · LLM conversation grading · OpenTelemetry AI agent · SMS bot monitoring · WhatsApp bot analytics
▾

Inspect replies, grades and failures across your agent's conversations.

Flow Observe shows you what your agent says to people, and which replies went wrong. It works on any agent, built with any framework or none.

  • Every conversation, message by message, with how long each reply took.
  • Every reply graded. Exact rules run first, then a grading model. Each reply gets a verdict: good, wrong, unhelpful or unsafe.
  • Failures grouped into issues by a fixed rule (the check that failed, the verdict and the topic), ranked by how many conversations they touch. Each issue links to the replies that show it and comes with a suggested fix you can copy into your prompt or code.
  • The inside of each turn, from your agent’s traces: model calls (model, tokens), tool calls, retrieval, errors and retries, and how long each step took.
  • Stats and usage: conversations, replies, reply time (median and 95th percentile), and the conversation count you are billed on.

Two ways in

Section titled “Two ways in”

Pick the one that matches how your agent talks to people.

On Flow MessagingOn any other channel
Your agent talks throughFlow Messaging (Telegram, WhatsApp, iMessage, SMS)Your own SMS provider, web chat, WhatsApp integration or anything else
What you addNothing. Optionally one tag on your tracesOne OpenTelemetry trace per turn
Where conversations come fromFlow Messaging’s event logYour traces
The text gradedWhat Flow Messaging delivered and sentWhat you report in flow.user_message and flow.agent_reply
The inside of each turnWhen you also send tracesYes, from the same traces
Test chatsSimulated contacts todayflow test (early access, see Test before you ship)
On Flow Messaging

No code. Observe reads your app’s conversations.

 On any other channel

Send one trace per turn. Python and TypeScript, with or without a framework.

Works alongside your other tools

Section titled “Works alongside your other tools”

Observe takes standard OpenTelemetry over OTLP/HTTP. If you already send traces somewhere, add Observe as a second exporter: the same spans go to both, and nothing in your code is specific to Flow except the flow.* attributes. You can try Observe next to what you use today, and stop at any time by removing the exporter.

provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(  # next to your existing processor
    endpoint="https://flow.engineer/observe/v1/traces",
    headers={"Authorization": f"Bearer {os.environ['FLOW_OBSERVE_TRACE_KEY']}"})))

The TypeScript lines go in your provider’s constructor (OpenTelemetry JS 2.x). The imports and the rest of the setup are on Send traces from any agent.

Get access

Section titled “Get access”

Sign in with your Flow login

Open flow.engineer/observe and sign in with the same Flow login you use on flow.engineer. Observe is in early access: if it isn’t on for you yet, the page says so; ask the Flow team (samyak@flow.engineer or prabal@flow.engineer) to turn it on.

Add your agent

In the dashboard, add your agent. On Flow Messaging, choose Flow Messaging and paste a test key of your app (fk_test_...; a key made just for Observe is best). On any other channel, choose traces.

Make your keys

For traces, make a trace key (otk_test_...); for flow test in CI, a CI key (oci_...). Each is shown once. Keep a trace key in FLOW_OBSERVE_TRACE_KEY on your server, never in a browser or mobile app, and never in a chat with an AI assistant. The dashboard shows your test conversations, labelled as test data.

Test data and production

Section titled “Test data and production”
  • Keys come in two modes, as in Flow Messaging: otk_test_... and otk_live_... for traces, fk_test_... and fk_live_... for Flow Messaging apps. Data never crosses between modes.
  • Until the Flow team enables your account for production, Observe accepts test keys and test data only. A live key is refused with 403.
  • Test conversations are never billed.

How your data is kept

Section titled “How your data is kept”
  • Message text and the content of your traces (prompts, tool inputs and results) are encrypted at rest with a key per account.
  • Observe deletes message text, including the text inside traces, 30 days after the message was sent (or after the period set for your account), and it is gone from Observe’s database backups within 7 more days. Conversations, traces, verdicts, issues and counts keep no message text, only ids, times, labels and numbers, so they stay. Flow Messaging keeps its own copy of your messages under its own retention.
  • Every time a person opens a conversation’s text in the dashboard, it is logged.
  • On other channels, the ids you send (conversation, contact, message) are stored only as keyed hashes, so a phone number used as an id is never stored as a phone number. On Flow Messaging, Observe keeps Flow’s own ids (conv_..., msg_..., ct_...) and never the person’s phone number or handle.
  • The dashboard shows span details (names, models, tokens, timings, errors), not prompts or tool inputs.
  • Dashboard pages load no analytics and no third-party scripts. Sign-in is with your Flow login, limited to the people on your account’s list.
Section titled “Related”