Test your agent

Write a situation and the outcome you expect. Flow’s test cloud plays the person, simulates your tools and judges the conversation, while your agent runs on your machine.

A good reply isn’t enough if the agent ordered the wrong thing. Test the product outcome: what was changed, what was confirmed and what happened when someone corrected themselves. The scenario below checks that a second message changes the order, rather than creating another one.

Scenario tests use models and a test cloud. They are separate from flow check, which checks your tool, skill and CLI definitions locally without a model call.

Before you start

You need a Flow project, a model key for its agent, and FLOW_FRAMEWORK_CLOUD_KEY for a test cloud. Set up a test cloud if this is your first run.

Use synthetic scenarios. --synthetic confirms that your test files contain no real customer data.

Write a scenario

Create tests/scenarios/latte-correction.yaml:

tests/scenarios/latte-correction.yaml

id: latte-correction
channels: [whatsapp]
person:
  persona: Sam, in a hurry; types in short bursts.
  goal: Get a large oat latte for pickup, under the name Sam.
opening:
  - say: "can i get a large latte"
  - say: "oat milk actually"
    after_ms: 1200
world: The point of sale is slow this morning, but orders go through.
expect:
  outcome:
    - Exactly one order is placed, for one large latte with oat milk.
  judge:
    - The agent asks Sam to confirm before placing the order.

The opening deliberately sends a correction after the first message. The outcome checks that only one order is placed and that the agent confirms it before acting.

Run it

From the project folder:

Terminal

npx @flow-engineer/flow test --synthetic --only latte-correction

Check it worked: the command opens a session and prints a result for each run, including the scenario, channel, judge’s reasons, cost and a link to the conversation. A file error is reported before a session starts.

Your agent uses its own model and key. The cloud plays the person and tools and evaluates the run.

Inspect and fix a failure

Use show last to read the checks, conversation and simulated tool calls. Then rerun only the scenarios that did not pass:

Terminal

npx @flow-engineer/flow show last
# Make the change in your agent.
npx @flow-engineer/flow test --synthetic --failed
# Run the full suite before finishing.
npx @flow-engineer/flow test --synthetic

show <run-id> reads a particular run. --failed uses .flow/last-session.json and includes failed, flaky, errored and unplayed scenarios. It cannot be combined with --only.

Describe tools and time

tests/tools/*.yaml define how the cloud plays each tool, including slow responses and errors. expect.calls and expect.never check the recorded tool calls by code.

For date-sensitive cases, set now and timezone and write dates relative to that clock, such as {{now + 1d | date}}. Tests can also restart an agent with restart: crash or restart: deploy to check recovery on the same store.

To test late in a long conversation without playing it, start the run from an earlier one with a top-level history: what each side said, oldest first (1 to 200 items, each up to 4,000 characters, {{now ...}} allowed). It is written into the agent’s store before the run, as if it had been said, with no model call and nothing sent; the person model and the judge see it as the conversation so far. It needs @flow-engineer/flow 0.5.0 or newer; an older client’s run ends client_outdated.

history:
  - person: "Book a table for four on Friday"
  - agent: "Done: Friday at 8 pm, for four."
opening:
  - say: "can we make it 9?"

The scenario-writing skill covers the full file format.

Get a test cloud

Sign in

Terminal

npx @flow-engineer/flow login

The command opens the browser and waits for sign-in. A coding agent can continue waiting for the same code with login --wait.

Set up the agent

From the project folder:

Terminal

npx @flow-engineer/flow setup

This fills the cloud and trace keys in .env without overwriting your configured values. Keep the file private; setup sets its permissions to 0600.

Run a synthetic scenario

Terminal

npx @flow-engineer/flow test --synthetic --only latte-correction

The default cloud is https://flow.engineer/observe. Use --cloud or FLOW_FRAMEWORK_CLOUD to choose another one.

The cloud prints the workspace’s concurrency and budget limits when a session opens. Test records are retained for 30 days. Your agent’s own model calls are billed through its provider.

Make a CI key by hand

Sign in to Flow, create an agent in the workspace, and make a CI key on its Developers page. Save it in the project’s .env as FLOW_FRAMEWORK_CLOUD_KEY, never on the command line.

The coding-agent guide documents setup flags, exit codes and next steps.

Test an agent you already have

An agent on another framework can use the same scenarios through an HTTP endpoint. With the agent running locally and tests/scenarios/*.yaml in your current folder:

Terminal

npx @flow-engineer/flow test \
  --agent-url http://localhost:3000/flow \
  --synthetic

The CLI calls the endpoint from your machine, so it does not need to be publicly reachable. Your real tools run on the agent’s side; use judge checks for their effects.

HTTP request, reply and timeout contract

Before the session opens, the command sends one HEAD request to the URL (never a message), and stops with Could not reach your agent at <url> if nothing answers; any HTTP status counts as an answer.

flow test runs on your machine and POSTs each of the person’s messages to that URL from there, so your agent never has to be reachable from the internet (http://localhost is fine). The request is JSON, with authorization: Bearer <FLOW_FRAMEWORK_AGENT_TOKEN> when that variable is set:

{
  "conversationId": "whatsapp:ses_8f2a.opening-hours.whatsapp.1",
  "messageId": "m1",
  "channel": "whatsapp",
  "message": { "text": "hi, what time are you open today?", "attachments": [{ "kind": "image", "mime": "image/jpeg", "name": "receipt.jpg", "data": "<base64>" }] }
}

Keep the conversation’s history by conversationId, as you would per phone number. channel is the platform the run plays (whatsapp, sms, imessage, telegram, web), or the name you give with --channel. message.text is "" when the person only sent a file; a tapped button or list row also comes as message.choice ({ id, title }, the button’s or row’s id), a quoted message as message.replyTo. Answer 200 with the replies to send, in order, in the shape a Flow Framework agent sends ({ "type": "message", "text", "buttons"? (labels, or { id, label }), "list"?, "link"?, "image"?, "replyTo"? }, { "type": "reaction", "emoji", "messageId" }, { "type": "template", ... }, …); [] sends nothing:

{ "replies": [{ "type": "message", "text": "We're open 8am to 6pm today.", "buttons": ["Order ahead"] }] }

A minimal server, with no dependencies (the whole file is examples/http-agent/server.mjs, with a test scenario beside it; it also ships in the package, in node_modules/@flow-engineer/flow/examples/http-agent):

import { createServer } from "node:http";

createServer(async (req, res) => {
  let body = "";
  for await (const chunk of req) body += chunk;
  const { conversationId, channel, message } = JSON.parse(body);
  const text = await myAgent.reply(conversationId, message.text); // your framework here
  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({ replies: [{ type: "message", text }] }));
}).listen(3000);

Each reply is checked against what the run’s platform can show, as a Flow Framework agent’s is: one that does not fit (buttons on SMS, text over the limit) is not sent, and the run says why. A status other than 200, a body that is not { "replies": [...] } or no answer within 60 seconds (--agent-timeout <s>) fails that message’s turn, with the reason in the run; the session goes on. A conversation’s messages are sent one at a time: the next waits for the last one’s answer.

Your agent’s own tools run on its side, for real, so tests/tools/*.yaml are not used, and outcome, calls and never checks (which read simulated tool calls) see none: write what must be true as judge items. The agent’s model cost shows as $0, since Flow Framework does not see its model calls. Not yet supported: agents that answer later through a callback instead of in the response, and the person’s reactions to the agent’s messages.

Any language

The endpoint’s contract is written down in spec/: agent-endpoint.openapi.yaml (OpenAPI 3.1: the request, the answer, statuses, timeouts, the token, and what each platform shows) with JSON Schemas for the request and the answer. A coding agent can implement it in any language from those files alone.

Check an endpoint against the contract before a test run, with no cloud, no key and no model call on Flow Framework’s side:

npx @flow-engineer/flow check-endpoint http://localhost:3000/flow [--token-env NAME] [--json]

It sends a fixed set of synthetic messages (a typed message, a double text, a photo, a tapped button quoting the agent, a message that may need no answer) and checks each answer: status 200, { "replies": [...] }, every reply against the schema and what WhatsApp can show, and every id a reply names (replyTo, a reaction’s messageId) one of the person’s messages. One line per case; exit 0 when all pass, else 1. The token is the value of NAME (default FLOW_FRAMEWORK_AGENT_TOKEN).

The café’s agent as an HTTP endpoint, with its two tools and one model through any OpenAI-compatible API (the same key variables as openAICompatible, MODEL for the model; FAKE_MODEL=1 for a fixed reply and no model call): examples/http-agent-python (standard library only) and examples/http-agent-go (standard library only).

Command output and reliability

Exit codes, flaky runs, retries and partial sessions

Exit	Meaning
0	Every run passed, or was flaky (without --strict).
1	A run failed (with --strict, or was flaky), or a message took a second send path (below). Or nothing was sent: a wrong flag, a test file that does not parse, no FLOW_FRAMEWORK_CLOUD_KEY; the message is printed on its own.
2	The connection, the tunnel or the stream failed (before or after the session opened), the cloud answered 5xx or no JSON, the cloud ended the session for its own failure, or runs errored (the cloud could not play or judge them) or were not played (the session ended early), and none failed.
3	The cloud refused: the open (a bad or wrong key, 401/403; an invalid request, 400; busy, 429 rate_limited; the spend cap, 429 spend_cap; 503 capacity), or it ended the session for the spend cap or capacity.
130	Interrupted (Ctrl-C).

A busy cloud’s refused open (429 other than the spend cap, or 503 capacity) with a Retry-After of 60 s or less is sent once more after it, with one line saying so; the spend cap never is. On a cloud that resumes, a dropped stream is opened again for up to 2 minutes, picking up after the last command. On a cloud that can re-open, a session the cloud lost is opened once more for only the runs with no result (the log names them, since they are played and paid for again), and the report covers both sessions with both dashboard links. Whenever a session ends before every run played (the tunnel, the stream or the connection failed, the cloud stopped it, or its summary left runs out), the runs that had already finished are printed, then N of M runs finished before the session ended early (K not played): <reason>, with the cloud’s error as the reason when it sent one, then the dashboard link; it exits 2 (3 when the cloud stopped it for the spend cap or capacity). Ctrl-C prints that same partial report, says the cloud keeps playing until it is stopped from the dashboard, and exits 130.

Flaky runs. The cloud plays a run that failed once more. When the second play passes the run is flaky: its line reads FLAKY, with played again: passed <watch> under it, and the summary lists it in its own section (1 flaky (failed once, passed when played again): with both watch links), apart from the failures. A flaky run does not fail the session (exit 0 when it is all that did not pass); --strict counts it as failed (exit 1). When the second play fails too, the run failed. The cloud also replays the person’s, the tools’ and the judge’s stored answers for a scenario it has played unchanged, so runs vary less and --runs 3 with a “2 of 3” bar should rarely be needed; --fresh plays every simulator anew. An older cloud does neither, and the output is as before.

Runs, pass rate and credit. --runs 3 plays each scenario three times on each of its channels. A run comes to one of four outcomes: passed, failed, credit or infra. Credit (the workspace’s daily test budget, spend_cap, or the agent’s model refused for credit, agent_credit, such as Flow’s trial credit used up) and infra (the cloud could not play or judge it, such as judge_failed) say nothing about the agent: their lines read CREDIT and ERROR, the summary counts them apart from the failures (4 of 6 runs passed. 1 failed. 1 stopped for credit (not the agent's fault, not counted as failed).), and a session with only those and passes exits 2. When the cloud stops the session for the agent’s credit, it ends with agent_credit: ... and exits 2. With --runs above 1 (or a run the cloud played again), the summary gives each scenario’s pass rate on each channel over the runs that passed or failed, flaky when some passed and some failed ( latte-correction (whatsapp): 2/3 passed, flaky), naming any credit or infra runs as not counted. A failed run still fails the session (exit 1).

One send path. Every message an agent sends in a test must go through its send (its channels), which announces it with a message.out event. On a Flow Framework agent the CLI counts, per conversation, the messages that reached the cloud and the ones announced; when they differ (an app’s own wrapper or code sent one the agent never did, or the agent says it sent one that never arrived), it names the conversation and the session fails (exit 1).

Each run’s line carries its id (FAIL latte-correction #1 (whatsapp) trn_…: 6 turns, …). --dotenv <path> reads the environment from that file instead of the folder’s .env (variables already set still win; no value is printed), and --verbose also prints slow event uploads.

JSON output and saved session records

--json prints one JSON document on stdout when the command ends, on every exit once its flags are read; every other line goes to stderr:

{
  "version": 1, "outcome": "failed", "exitCode": 1,
  "sessionId": "tse_…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…",
  "error": null,
  "counts": { "runs": 2, "passed": 1, "failed": 1, "flaky": 0, "errored": 0, "notPlayed": 0, "credit": 0, "infra": 0 },
  "scenarios": [{ "scenario": "latte-correction", "path": "scenarios/latte-correction.yaml", "channel": "whatsapp", "runs": 2, "passedRuns": 1, "credit": 0, "infra": 0, "label": "flaky" }],
  "sendPath": [],
  "strict": false,
  "costUsd": { "agent": 0.0123, "test": 0.0456 },
  "runs": [
    { "runId": "trn_…", "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "verdict": "failed", "outcome": "failed",
      "error": null, "ended": "goal_met", "turns": 6, "watch": "https://…",
      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }],
      "invalid": null, "rerun": { "runId": "trn_…", "passed": false, "watch": "https://…" }, "label": "failing",
      "replayed": { "person": 3, "tools": 1, "judge": 0 } }
  ],
  "invalid": [],
  "sessionInvalid": null
}

outcome is passed (exit 0), failed or invalid (1), errored or transport (2), refused (3) or interrupted (130). A run’s verdict is errored whenever the cloud names an error for it (such as judge_failed, simulator_failed or spend_cap), whatever it ended as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads ERROR instead of PASS or FAIL, and the run’s error is that code (else null). A run’s verdict is flaky when it failed and passed when the cloud played it again; rerun is that second play ({ "runId", "passed", "watch" }, else null), label is flaky or failing (it failed again) on a run played again (else null), and replayed counts the stored simulator answers the run replayed (else null). The second play is never a run of its own in runs or counts. counts.failed leaves flaky runs out unless --strict (strict: true); counts.flaky counts them. A run’s outcome is passed, failed, credit or infra (null for one not played; a flaky run’s is failed, its first play), and counts.credit and counts.infra count those runs, which counts.errored also holds. scenarios is each scenario (by its id, with its file as path) on each channel: runs the runs that passed or failed, passedRuns how many passed, credit and infra the runs not counted, and label flaky when some but not all passed, or a run failed and passed when played again. sendPath lists each conversation whose messages did not all go through the agent’s send, as { "conversationId", "delivered", "announced" }. error is { "message", "status", "type", "retryAfterSeconds" } when the session did not finish (type is the cloud’s code, such as spend_cap; the others are null when it did not say).

A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are errored with error scenario_invalid and invalid { "path", "reason" } (the reason starts unsupported: when the cloud does not know a key), its line reads INVALID with the file and what is wrong, and the top-level invalid lists each such scenario once as { "scenario", "path", "reason" }. When the cloud ends the whole session scenario_invalid (a tools or checks file it cannot read, or an older cloud), sessionInvalid is { "reason", "olderCloud", "sent" }: what the cloud said after the code (else null), whether that suggests a cloud older than this client, and the scenarios sent as { "scenario", "path" }; the same lines are printed.

--only latte-correction,order-fails runs only those scenarios (file names without .yaml); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes .flow/last-session.json in the agent’s folder and prints its path; flow new git-ignores .flow/. It holds what each run ended as and the judge’s reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by --synthetic):

{
  "sessionId": "…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…", "endedAt": "2026-10-16T09:12:03.000Z",
  "ended": "finished",
  "only": null,
  "runs": [
    { "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "runId": "trn_…", "verdict": "failed",
      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }] }
  ]
}

A run of a scenario the cloud could not play also has invalid ({ "path", "reason" }), and a session the cloud ended scenario_invalid has sessionInvalid, as in --json.

ended is finished, stopped (the session broke off or the cloud stopped it) or interrupted. agent is the cloud’s id for the agent (a hosted cloud names it), which flow show reads. only is the scenarios the session was limited to, null for the whole suite. verdict is passed, failed, flaky (failed, then passed when the cloud played it again), errored (the cloud could not play or judge the run) or not_played (the session ended first; its runId is null), and outcome is as in --json.

Try a case by hand

Inspect the agent and tools before writing a scenario.

Deploy a checked version

Exercise staging, inspect logs and promote.