> ## Documentation Index > Fetch the complete documentation index at: https://flow.engineer/docs/llms.txt > Use this file to discover all available pages before exploring further. # Test your integration > Exercise conversations before you send a new agent version to real people. A change to a prompt, a model or a tool can fix one reply and break ten others. People rarely tell you; they stop writing. The goal of this step is that a version that makes your agent worse never reaches them. ## A check in your CI: `flow test` > [!NOTE] > > Early access. Make your agent's **CI key** in the Observe dashboard ([Get access](https://flow.engineer/docs/unified-messaging-api/observe.md#get-access)); if Observe isn't on for you yet, write to the Flow team at [samyak@flow.engineer](mailto:samyak@flow.engineer) or [prabal@flow.engineer](mailto:prabal@flow.engineer). The `flow` binary is a public download (below). `flow test` runs your test chats against **your own agent**, sends the transcripts to Observe, and fails the build when a reply breaks a rule or misses what the test expects. Observe grades every reply and decides pass or fail by a fixed rule, so the same transcript always gets the same answer. Your agent can use any framework, or none: `flow test` only needs an HTTP endpoint or a command that answers one message. ``` your CI ──> flow test ──(each test message)──> your agent │ └──(transcripts + expectations)──> Observe ──> pass / fail ``` ### 1. Write test chats Put them in `.flow/tests.yaml` in your repo. Each case is one conversation: the person's turns, and for each reply what it must and must not do. ```yaml version: 1 channel: sms cases: - id: no-delivery-promise name: Doesn't promise a date it can't confirm turns: - user: "Can you get my order to me by Sunday?" expect: ["Doesn't promise a delivery date it can't confirm, and offers to check or to connect a person."] must_not: ["guaranteed"] - user: "ok thanks" - id: cancel turns: - user: "I want to cancel my order" includes: ["cancel"] ``` * `expect`: plain sentences a grader checks. * `includes` and `must_not`: exact text, case-insensitive. * `rubrics`: rule ids from your check set that must pass. Every list is optional. A turn with none is still graded against your checks. Test chats are test data you write. Never put a real person's conversation in a test file. ### 2. Let `flow test` reach your agent Start your agent in the CI job, and give `flow test` one of two ways to send it a message: * **`--http URL`**: `flow test` posts each turn as JSON: ```json {"conversation_id": "ft_k3m9q2x7ab_cancel", "contact_id": "ft_k3m9q2x7ab_cancel", "channel": "sms", "message": {"text": "I want to cancel my order"}, "test": {"case": "cancel", "turn": 0}} ``` Your agent answers with `{"replies": [{"text": "..."}]}`, `{"reply": "..."}`, or a plain-text body. Keep the conversation's history by `conversation_id`, as you would for a real person. * **`--cmd "..."`**: `flow test` runs your command once per turn, with the same JSON on its stdin, and reads the answer from its stdout. Optionally, send your agent's [traces](https://flow.engineer/docs/unified-messaging-api/observe/traces.md) during the test too: in CI, give your agent the **CI key** as its trace key, and set `flow.conversation_id` to the `conversation_id` it was given. Each test chat then shows the inside of every turn in the dashboard, so you can see why a reply failed. With a CI key, Observe takes traces only for that agent's test chats; they never reach its real conversations. ### 3. Run it in CI ```yaml GitHub Actions - name: Get flow run: | V=v0.1.0 base="https://github.com/flow-engineer/sdk/releases/download/flow-cli-$V" curl -fsSLO "$base/flow_${V}_linux_amd64" curl -fsSLO "$base/SHA256SUMS" grep " flow_${V}_linux_amd64\$" SHA256SUMS | sha256sum -c - install -m 755 "flow_${V}_linux_amd64" ./flow - name: Start the agent env: FLOW_OBSERVE_TRACE_KEY: ${{ secrets.FLOW_CI_KEY }} # optional: traces during the test run: | python3 my_agent_server.py & for i in $(seq 1 50); do curl -s -o /dev/null http://127.0.0.1:3000 && break; sleep 0.2; done - name: flow test env: FLOW_CI_KEY: ${{ secrets.FLOW_CI_KEY }} run: ./flow test --http http://127.0.0.1:3000 --version "$GITHUB_SHA" --junit flow-test.xml ``` `flow` is published for Linux and macOS, on x86-64 and arm64, on the [SDK's releases page](https://github.com/flow-engineer/sdk/releases), with a `SHA256SUMS` file to check it against. Any other CI works the same way, as shell steps. `flow test` prints a line per case, and for a failure, which expectation it missed and which rules failed: ``` FAIL no-delivery-promise Doesn't promise a date it can't confirm turn 1, reply 1 (verdict wrong): expect "Doesn't promise a delivery date it can't confirm, ..." failed PASS cancel 1 passed, 1 failed, 0 error: FAILED ``` It exits `0` when every case passes, `1` when one fails, and `2` when it couldn't run (your agent didn't answer, or Observe couldn't be reached). `--junit` writes a report your CI can show. * **The key.** Use your agent's **CI key** (`oci_...`), kept in your CI's secret store as `FLOW_CI_KEY`. It works whether your agent is in test or live mode, and it can only run tests and send traces for them: it can't read or add to your agent's real conversations. Trace keys are refused. `flow test` never prints the key. * **Limits.** * Up to 50 cases per run, 20 turns per case and 2,000 characters per message. Replies can be up to 8,000 characters. * A few runs per agent at once, and a daily budget of graded test replies. * When Observe is busy, `flow test` waits and retries. Test runs aren't billed. They show in the dashboard labelled "Test run", kept apart from your production stats. ## Also today **Every test conversation is graded.** Observe grades every reply your agent sends, test conversations included, with the same checks as production. * **On Flow Messaging:** [simulated contacts](https://flow.engineer/docs/unified-messaging-api/concepts/test-and-live-mode.md#simulated-contacts-test-without-a-phone) play chats through the real send gate. Each reply's verdict shows in the [Observe dashboard](https://flow.engineer/observe). ```ts TypeScript const conv = await flow.test.createConversation({ channel: "sms" }); await flow.test.inbound(conv.id, { kind: "basic", text: "Can I change my delivery to Sunday?" }); // your agent answers as usual; then: for await (const m of flow.test.transcript(conv.id)) console.log(m.direction, m.content); // exactly what the contact sees ``` * **Failures become issues.** When replies fail the same check, Observe groups them into one issue under **Needs attention**, with the replies as evidence and a suggested fix. For each issue it also proposes a test case: the conversation up to the failing reply, and what the reply must and must not do. Copy it into `.flow/tests.yaml`. ## Coming > [!NOTE] > > These are being built with the first early-access teams. Their shape may change before they open. * **Test cases in the dashboard**: write them there, or save one from a failing conversation in one click, and `flow test` runs them with your file. * **`flow test` on Flow Messaging**: cases played through simulated contacts instead of an HTTP endpoint. ## Related * [What Observe gives you](https://flow.engineer/docs/unified-messaging-api/observe/index.md) * [Send traces from any agent](https://flow.engineer/docs/unified-messaging-api/observe/traces.md) * [Simulated contacts](https://flow.engineer/docs/unified-messaging-api/concepts/test-and-live-mode.md#simulated-contacts-test-without-a-phone) * [What changes when real people text your bot](https://flow.engineer/docs/unified-messaging-api/guides/real-people.md)