Skip to content
Flow Flow

Ask the docs

Answers come only from these docs and are written by an AI model (Gemini), so check the linked pages. Your question is not kept. Do not include personal details or keys.

Test your integration

Testing · test AI agent before deploy · chatbot regression testing · LLM CI check · AI agent test suite · conversation testing · flow test
▾

Exercise conversations before you send a new agent version to real people.

A change to a prompt, a model or a tool can fix one reply and break ten others. People rarely tell you; they stop writing. The goal of this step is that a version that makes your agent worse never reaches them.

A check in your CI: flow test

Section titled “A check in your CI: flow test”

flow test runs your test chats against your own agent, sends the transcripts to Observe, and fails the build when a reply breaks a rule or misses what the test expects. Observe grades every reply and decides pass or fail by a fixed rule, so the same transcript always gets the same answer. Your agent can use any framework, or none: flow test only needs an HTTP endpoint or a command that answers one message.

your CI ──> flow test ──(each test message)──> your agent
                 │
                 └──(transcripts + expectations)──> Observe ──> pass / fail

1. Write test chats

Section titled “1. Write test chats”

Put them in .flow/tests.yaml in your repo. Each case is one conversation: the person’s turns, and for each reply what it must and must not do.

version: 1
channel: sms
cases:
  - id: no-delivery-promise
    name: Doesn't promise a date it can't confirm
    turns:
      - user: "Can you get my order to me by Sunday?"
        expect: ["Doesn't promise a delivery date it can't confirm, and offers to check or to connect a person."]
        must_not: ["guaranteed"]
      - user: "ok thanks"
  - id: cancel
    turns:
      - user: "I want to cancel my order"
        includes: ["cancel"]
  • expect: plain sentences a grader checks.
  • includes and must_not: exact text, case-insensitive.
  • rubrics: rule ids from your check set that must pass.

Every list is optional. A turn with none is still graded against your checks.

Test chats are test data you write. Never put a real person’s conversation in a test file.

2. Let flow test reach your agent

Section titled “2. Let flow test reach your agent”

Start your agent in the CI job, and give flow test one of two ways to send it a message:

  • --http URL: flow test posts each turn as JSON:

    {"conversation_id": "ft_k3m9q2x7ab_cancel", "contact_id": "ft_k3m9q2x7ab_cancel",
     "channel": "sms", "message": {"text": "I want to cancel my order"},
     "test": {"case": "cancel", "turn": 0}}

    Your agent answers with {"replies": [{"text": "..."}]}, {"reply": "..."}, or a plain-text body. Keep the conversation’s history by conversation_id, as you would for a real person.

  • --cmd "...": flow test runs your command once per turn, with the same JSON on its stdin, and reads the answer from its stdout.

Optionally, send your agent’s traces during the test too: in CI, give your agent the CI key as its trace key, and set flow.conversation_id to the conversation_id it was given. Each test chat then shows the inside of every turn in the dashboard, so you can see why a reply failed. With a CI key, Observe takes traces only for that agent’s test chats; they never reach its real conversations.

3. Run it in CI

Section titled “3. Run it in CI”
GitHub Actions
- name: Get flow
  run: |
    V=v0.1.0
    base="https://github.com/flow-engineer/sdk/releases/download/flow-cli-$V"
    curl -fsSLO "$base/flow_${V}_linux_amd64"
    curl -fsSLO "$base/SHA256SUMS"
    grep " flow_${V}_linux_amd64\$" SHA256SUMS | sha256sum -c -
    install -m 755 "flow_${V}_linux_amd64" ./flow
- name: Start the agent
  env:
    FLOW_OBSERVE_TRACE_KEY: ${{ secrets.FLOW_CI_KEY }}   # optional: traces during the test
  run: |
    python3 my_agent_server.py &
    for i in $(seq 1 50); do curl -s -o /dev/null http://127.0.0.1:3000 && break; sleep 0.2; done
- name: flow test
  env:
    FLOW_CI_KEY: ${{ secrets.FLOW_CI_KEY }}
  run: ./flow test --http http://127.0.0.1:3000 --version "$GITHUB_SHA" --junit flow-test.xml

flow is published for Linux and macOS, on x86-64 and arm64, on the SDK’s releases page, with a SHA256SUMS file to check it against. Any other CI works the same way, as shell steps.

flow test prints a line per case, and for a failure, which expectation it missed and which rules failed:

FAIL   no-delivery-promise  Doesn't promise a date it can't confirm
       turn 1, reply 1 (verdict wrong):
         expect "Doesn't promise a delivery date it can't confirm, ..." failed
PASS   cancel
1 passed, 1 failed, 0 error: FAILED

It exits 0 when every case passes, 1 when one fails, and 2 when it couldn’t run (your agent didn’t answer, or Observe couldn’t be reached). --junit writes a report your CI can show.

  • The key. Use your agent’s CI key (oci_...), kept in your CI’s secret store as FLOW_CI_KEY. It works whether your agent is in test or live mode, and it can only run tests and send traces for them: it can’t read or add to your agent’s real conversations. Trace keys are refused. flow test never prints the key.
  • Limits.
    • Up to 50 cases per run, 20 turns per case and 2,000 characters per message. Replies can be up to 8,000 characters.
    • A few runs per agent at once, and a daily budget of graded test replies.
    • When Observe is busy, flow test waits and retries.

Test runs aren’t billed. They show in the dashboard labelled “Test run”, kept apart from your production stats.

Also today

Section titled “Also today”

Every test conversation is graded. Observe grades every reply your agent sends, test conversations included, with the same checks as production.

  • On Flow Messaging: simulated contacts play chats through the real send gate. Each reply’s verdict shows in the Observe dashboard.

    TypeScript
    const conv = await flow.test.createConversation({ channel: "sms" });
    await flow.test.inbound(conv.id, { kind: "basic", text: "Can I change my delivery to Sunday?" });
    // your agent answers as usual; then:
    for await (const m of flow.test.transcript(conv.id)) console.log(m.direction, m.content); // exactly what the contact sees

  • Failures become issues. When replies fail the same check, Observe groups them into one issue under Needs attention, with the replies as evidence and a suggested fix. For each issue it also proposes a test case: the conversation up to the failing reply, and what the reply must and must not do. Copy it into .flow/tests.yaml.

Coming

Section titled “Coming”
  • Test cases in the dashboard: write them there, or save one from a failing conversation in one click, and flow test runs them with your file.
  • flow test on Flow Messaging: cases played through simulated contacts instead of an HTTP endpoint.
Section titled “Related”