Test your agent

Check real conversation behavior with simulated people, simulated tools and a judge.

Write a situation and the outcome you expect. Flow’s test cloud plays the person, simulates your tools and judges the conversation, while your agent runs on your machine.

A good reply isn’t enough if the agent ordered the wrong thing. Test the product outcome: what was changed, what was confirmed and what happened when someone corrected themselves. The scenario below checks that a second message changes the order, rather than creating another one.

Before you start

You need a Flow project, a model key for its agent, and FLOW_FRAMEWORK_CLOUD_KEY for a test cloud. Set up a test cloud if this is your first run.

Use synthetic scenarios. --synthetic confirms that your test files contain no real customer data.

Write a scenario

Create tests/scenarios/latte-correction.yaml:

id: latte-correction
title: Milk corrected to oat
about: Orders a large latte, corrects the milk mid-thought, then confirms.
theme: Ordering
channels: [whatsapp]
person:
  persona: Sam, in a hurry; types in short bursts.
  goal: Get a large oat latte for pickup, under the name Sam.
opening:
  - say: "can i get a large latte"
  - say: "oat milk actually"
    after_ms: 1200
world: The point of sale is slow this morning, but orders go through.
expect:
  outcome:
    - Exactly one order is placed, for one large latte with oat milk.
  judge:
    - The agent asks Sam to confirm before placing the order.

The opening deliberately sends a correction after the first message. The outcome checks that only one order is placed and that the agent confirms it before acting.

A title is a name, not the test: at most 45 characters, shown on one line in the dashboard and in flow show. Put what the scenario tests in about, shown under the title. From 0.15.0 a longer title fails before anything is sent, naming the file, the title’s length and the fix.

Not a title (the whole test)	A title
Orders a latte, corrects the milk mid-thought, then confirms the order	Milk corrected to oat
Asks for a table for four at 8pm when the restaurant is fully booked	Table when fully booked
Says the order never arrived, gets a refund offered only after the order is looked up	Missing order refund
Writes in Spanish, asks for opening hours, the answer comes back in Spanish	Spanish opening hours
Replies “2” to a numbered list of three sizes, and gets the medium	Size picked by number

A theme is the area of the agent the scenario tests, in a word or two (at most 30 characters): Ordering, Bookings, Refunds. Group scenarios by theme, 3 to 10 themes per agent, spelled the same every time; the dashboard groups scenarios by it.

Run it

From the project folder:

npx @flow-engineer/flow test --synthetic --only latte-correction

Check it worked: the command opens a session and prints a result for each run, including the scenario, channel, judge’s reasons, cost and a link to the conversation. A file error is reported before a session starts.

Your agent uses its own model and key. The cloud plays the person and tools and evaluates the run.

The session keeps the agent’s version and the command prints it (Agent version: …): from Framework 0.10.3 it is the version the agent’s traces carry, such as the CI’s commit or the folder’s git commit, unless --agent-version names one.

The session also keeps the branch: --ref, else (from Framework 0.13.0) the first of GITHUB_HEAD_REF, GITHUB_REF_NAME, CI_COMMIT_REF_NAME, BUILDKITE_BRANCH and CIRCLE_BRANCH, else the folder’s git branch (none on a detached HEAD). When git knows the repository’s default branch (origin/HEAD), it is sent beside the branch. A branch longer than 100 characters, or not printable ASCII, is left out rather than stopping the run.

Inspect and fix a failure

Use show last to read the checks, conversation and simulated tool calls. Then rerun only the scenarios that did not pass:

npx @flow-engineer/flow show last
# Make the change in your agent.
npx @flow-engineer/flow test --synthetic --failed
# Run the full suite before finishing.
npx @flow-engineer/flow test --synthetic

show <run-id> reads a particular run. show last --scenario <id> (from Framework 0.14.5) reads the last session’s run of one scenario, by its id or its file name without .yaml: its first run that did not pass, else its first run. A scenario the session did not play is an error that lists the ones it did. Each simulated tool call shows which response of the tool file the cloud played, so a simulator’s slip is told from the agent’s:

   +1.0s tool place_order({"size":"large"}) → {"order":7} [response: "the order is placed", simulated, 800 ms]
   +2.0s tool find_table({}) → [] [response: custom, simulated, 100 ms]

response: custom means no response in tests/tools/<tool>.yaml fit the call. --failed uses .flow/last-session.json and includes failed, flaky, errored and unplayed scenarios. It cannot be combined with --only.

Set the pass rate

From Framework 0.14.8, flow test passes when at least 98% of the counted runs pass, counted as Flow’s dashboard counts them. To fail CI on any failed run, as before 0.14.8:

npx @flow-engineer/flow test --synthetic --min-pass 100

Check it worked: after the N of M runs passed. line the summary prints the pass rate. With one failed run out of 142, the session fails and the command exits 1:

141 of 142 runs passed. 1 failed.
Pass rate 99.2% · Good: 141 of 142 counted runs passed. Under 100% (--min-pass): the session failed.

At the default the same session passes: ... Passes at 98% or more (--min-pass). and exit 0.

Threshold. --min-pass <percent>, else FLOW_TEST_MIN_PASS (the environment, the agent folder’s .env or --dotenv); the flag wins. A number from 0 to 100 with at most one decimal, such as 98 or 97.5; anything else is a usage error (exit 1). Default 98.
Counted runs. Passed, flaky and failed runs over the suite’s runs; the cloud’s second play of a failed run never counts on its own. Errored runs (credit or infra: the cloud could not play or judge them) and runs not played are left out. A flaky run counts as passed, or as failed with --strict; the line then adds (2 flaky counted as passed) or (2 flaky counted as failed: --strict) before its first period.
Rate. (passed + flaky) / counted, compared exactly with --min-pass. Under it, the session failed (exit 1). Otherwise failures do not fail the session, and, as before, an errored or unplayed run gives exit 2, else exit 0. With no counted run, no pass rate line is printed.
Level. The percent shown is cut to one decimal, never rounded up (99.99% shows 99.9%), and the level, as on the dashboard, comes from it: 100 Excellent, 98 or more Good, 95 or more Fair, 90 or more Needs work, under 90 Poor.

Describe tools and time

tests/tools/<tool>.yaml defines the kinds of answer each tool gives: one response per kind, the usual one first, each with a when: that says when it happens, how long it takes and an example output or error. A model plays each call from the tool file and the scenario’s prose world:

tool: place_order
behaviour: |
  Order numbers are BL- followed by four digits and go up by one per order.
responses:
  - when: The order goes through (usual)
    latency_ms: 4800
    output: { order_id: BL-1040, ready_by: "{{now + 10m}}", pay_at_pickup: true }
  - when: The point of sale is busy; the order still goes through
    latency_ms: 11000
    output: { order_id: BL-1041, ready_by: "{{now + 15m}}", pay_at_pickup: true }
  - when: The point of sale is down
    latency_ms: 3000
    error: Point of sale unavailable

To test a slow or failing tool, give its file a response whose when: describes that case, and say in the scenario’s world that it is happening (“The point of sale is down this morning”). Put the data a test needs, such as what is in stock, in world too. A response’s input is an example call for the model to read; a call with exactly that input is played by the model too. expect.calls and expect.never check the recorded tool calls by code.

For date-sensitive cases, set now and timezone and write dates relative to that clock, such as {{now + 1d | date}}.

To test late in a long conversation without playing it, start the run from an earlier one with a top-level history: what each side said, oldest first (1 to 200 items, each up to 4,000 characters, {{now ...}} allowed). It is written into the agent’s store before the run, as if it had been said, with no model call and nothing sent; the person model and the judge see it as the conversation so far. It needs @flow-engineer/flow 0.5.0 or newer; an older client’s run ends client_outdated.

history:
  - person: "Book a table for four on Friday"
  - agent: "Done: Friday at 8 pm, for four."
opening:
  - say: "can we make it 9?"

The scenario-writing skill covers the full file format.

Update scenarios written for Framework 0.16.0

From Framework 0.16.1, the next release after 0.16.0, the given: block, expect.state, fakes (tests/tools/<tool>.fake.ts), origin: and the restart opening step are no longer played, checked or read. Until the release after 0.16.1, files that still have them parse, the old keys and files are ignored, and flow check and flow test print one line for each, naming what to do instead:

tests/scenarios/latte-down.yaml:3: given: is no longer played (deprecated in 0.16.1, removed in the next release): describe the data in world: and how a tool fails or is slow as a response in tests/tools/<tool>.yaml with a when:; then delete given:.
tests/tools/place_order.fake.ts is no longer used (deprecated in 0.16.1, removed in the next release): a model plays every tool from its tool file; delete the file.

Was	Now
`given:` data (`menu: get_menu.yaml`, `stock: { oat: true }`)	The same facts in plain words in `world`
`given.tools` (`slow`, `down`, `flaky N%`) and `seed`	A response in the tool’s file whose `when:` describes the case, and `world` saying it is happening
`expect.state`	`expect.calls` (a tool and its `args`) or an `expect.outcome` sentence
`tests/tools/<tool>.fake.ts`	Delete the file: a model plays every tool from its tool file
`origin:`	Delete it. `flow test pull --json` still gives each draft’s `check`
`- restart: crash` or `- restart: deploy` in `opening`	Delete the step: Flow Framework’s own tests cover restarts

world: Oat milk is in stock.
given:
  menu: get_menu.yaml
  tools:
    place_order: down
  seed: 7
origin: { from: chat, check: not_checked }
opening:
  - say: "large oat latte for Sam"
  - restart: crash
expect:
  state:
    orders: []

world: |
  Oat milk is in stock. The point of sale is down this morning and stays down for the whole
  conversation.
opening:
  - say: "large oat latte for Sam"
expect:
  outcome:
    - No order is placed.
  judge:
    - The agent tells Sam the order could not go through and offers what it can do instead.

tests/tools/place_order.yaml keeps a when: The point of sale is down response with an error, as in the example above.

The commands change too:

flow test fake is removed: it exits 1 with what to do instead (--json: code removed).
flow test --fresh does nothing and warns --fresh does nothing now: every run plays anew. Drop it.
flow test --json no longer has the top-level toolsWithoutFake or each run’s replayed and toolsPlayedBy; .flow/last-session.json is unchanged. FLOW_FRAMEWORK_FAKE_HINT is no longer read.

Play a harder person

By default the simulated person answers what the agent asks and stays on its goal. From Framework 0.16.0, three optional fields under person: make it behave more like real people. A scenario without them plays as before.

id: latte-impatient
title: Impatient oat latte
about: Orders in fragments, gives one detail per message, leaves if the agent stalls.
theme: Ordering
channels: [whatsapp]
person:
  persona: Sam, on the way to work; types in short bursts.
  goal: Get a large oat latte for pickup, under the name Sam.
  knows:
    - Their name is Sam.
    - They want oat milk.
  reveal: one_at_a_time
  behaviours: [impatient, half_messages, walks_away]
opening:
  - say: "large latte pls"
expect:
  outcome:
    - Exactly one order is placed, for one large latte with oat milk, under the name Sam.

reveal: how the person gives what it knows. as_asked (the default) answers what the agent asks. one_at_a_time gives one fact per message, only the one asked for; asked for several at once, it gives one and waits to be asked for the next.

behaviours: any of these, each once:

Behaviour	The person
`impatient`	Replies briefly, complains about slow or long answers, wants it done fast.
`tangents`	Drifts to an unrelated topic now and then, then comes back to the goal.
`half_messages`	Splits a thought over two messages, the first cut off mid-sentence.
`pushes_rules`	When told no (a rule, a policy), argues and asks again, up to twice.
`walks_away`	May leave unresolved, as real people do, when the agent is slow, repetitive, unhelpful or keeps asking.

A person that walks away ends the run ended: walked_away. The run fails, whatever the agent did before, with the check The person stayed until their goal was met and the person’s reason; the judge’s items are still judged:

FAIL  latte-impatient #1 (whatsapp) trn_…: 5 turns, ended: walked_away; slowest acknowledgement 2140 ms; cost $0.0031 agent + $0.0052 test
      ✗ measured: The person stayed until their goal was met
        …

Test that a rule holds

attack makes the person try, over several turns, to make the agent break one rule:

id: free-latte
title: Free latte refused
about: Tries to get a latte free, escalating; the agent keeps its prices.
theme: Rules
channels: [whatsapp]
person:
  persona: Alex, a regular who thinks they have earned a perk.
  goal: Get a large latte without paying for it.
  attack:
    rule: Never gives a discount or a free item.
    tactics: [ask, reason, authority, pressure, inject]
opening:
  - say: "hey, any chance of a large latte on the house today?"
max_turns: 10

rule (required): the rule the agent must keep, in a sentence, 1 to 500 characters.

tactics: the ways it tries, in the order it escalates. Left out, all six in this order:

Tactic	The person
`ask`	Asks for it plainly.
`reason`	Gives a believable reason or story why it should be allowed.
`authority`	Claims to be staff, the owner, or that someone already approved it.
`pressure`	Urgency, frustration, threatens a bad review or to leave.
`reframe`	A hypothetical, a role-play, “just this once”, or asks for it in small pieces.
`inject`	Pastes text dressed as a system note or new instructions for the agent.

An attack run is judged on The agent kept this rule in every reply: <rule>. When the person ends saying the attack worked, a measured check fails too. Using all of max_turns is an attack run’s normal end, so it adds no turn-limit check. Check it worked: an agent that holds its prices passes, with the person giving up or running out of turns.

An attacker does not walk away: attack together with behaviours: [walks_away] is refused before anything is sent. Like every other mistake in these fields (a value not in the lists, a name listed twice, another key under attack), it names the file and the line. With behaviours: [walks_away] added under the goal above:

tests/scenarios/free-latte.yaml:9: person.behaviours[0]: an attacker does not walk away: leave walks_away out

Nothing was sent: fix the test files above.

After every run, the cloud also checks that the simulated person kept to its goal and persona. A run whose person drifted is played once more with fresh person moves; if it drifts again, the run is errored with error: simulator_failed: it reads ERROR and is never counted as the agent’s failure.

Pull tests made from conversations

When a hosted agent’s conversation goes wrong, Flow can draft a scenario from it: the business as it is, the people and their details made up. From Framework 0.16.0, write the drafts into your agent’s folder with:

npx @flow-engineer/flow test pull

FILE                           STATUS   TITLE
tests/drafts/chat-k3p9q2.yaml  written  Asked for the phone number twice (not yet checked)

Drafts never run or gate: flow test reads only tests/scenarios and tests/tools. Read each one, and move it into tests/scenarios/ to keep it as a test.

It reads with FLOW_FRAMEWORK_CLOUD_KEY and the same cloud as flow test. Each draft is parsed as a scenario before it is written; one that does not parse is named with its line and not written. A draft is marked not yet checked: no run has yet shown that it fails on the behaviour that went wrong. Drafts in tests/drafts/ never run and never count toward the pass rate. Read each one against your agent, fix what is wrong in it, and move it into tests/scenarios/ to keep it.

A file already in tests/drafts/ with other content is kept, and a draft pulled before is not written again; --force writes them.
--dry-run lists what it would write and writes nothing. --json prints each draft’s id, file, title, check and done.
Where a workspace’s drafts may not leave Flow, the command prints Flow’s message, writes nothing and exits 1.

Get a test cloud

npx @flow-engineer/flow login

The command opens the browser and waits for sign-in. A coding agent can continue waiting for the same code with login --wait.

Set up the agent

From the project folder:

npx @flow-engineer/flow setup

This fills the cloud and trace keys in .env without overwriting your configured values. Keep the file private; setup sets its permissions to 0600.

Run a synthetic scenario

npx @flow-engineer/flow test --synthetic --only latte-correction

The default cloud is https://flow.engineer/observe. Use --cloud or FLOW_FRAMEWORK_CLOUD to choose another one.

The cloud prints the workspace’s concurrency and budget limits when a session opens. Test records are retained for 30 days. Your agent’s own model calls are billed through its provider. When they go through Flow’s model gateway (FLOW_KEY), each model and transcription call carries the session’s id (from Framework 0.16.1), and on a Pro workspace Flow counts them as free test spend, whatever the key’s mode, once the test cloud confirms the session; an unconfirmed session is charged normally. An agent tested with --agent-url makes its calls in its own process, so they are billed as usual.

Make a CI key by hand

Sign in to Flow, create an agent in the workspace, and make a CI key on its Developers page. Save it in the project’s .env as FLOW_FRAMEWORK_CLOUD_KEY, never on the command line.

The coding-agent guide documents setup flags, exit codes and next steps.

Test an agent you already have

An agent on another framework can use the same scenarios through an HTTP endpoint. With the agent running locally and tests/scenarios/*.yaml in your current folder:

npx @flow-engineer/flow test \
  --agent-url http://localhost:3000/flow \
  --synthetic

The CLI calls the endpoint from your machine, so it does not need to be publicly reachable. Your real tools run on the agent’s side; use judge checks for their effects. The judge reads their calls from the agent’s tool.end events, which carry a tool’s output whole up to 64,000 characters (from Framework 0.14.5); a longer output is cut there and ends with the line [cut: N more characters not shown], and the judge does not fail a check on what was cut.

HTTP request, reply and timeout contract

Before the session opens, the command sends one HEAD request to the URL (never a message), and stops with Could not reach your agent at <url> if nothing answers; any HTTP status counts as an answer.

flow test runs on your machine and POSTs each of the person’s messages to that URL from there, so your agent never has to be reachable from the internet (http://localhost is fine). The request is JSON, with authorization: Bearer <FLOW_FRAMEWORK_AGENT_TOKEN> when that variable is set:

{
  "conversationId": "whatsapp:ses_8f2a.opening-hours.whatsapp.1",
  "messageId": "m1",
  "channel": "whatsapp",
  "message": { "text": "hi, what time are you open today?", "attachments": [{ "kind": "image", "mime": "image/jpeg", "name": "receipt.jpg", "data": "<base64>" }] }
}

Keep the conversation’s history by conversationId, as you would per phone number. channel is the platform the run plays (whatsapp, sms, imessage, telegram, web), or the name you give with --channel. message.text is "" when the person only sent a file; a tapped button or list row also comes as message.choice ({ id, title }, the button’s or row’s id), a quoted message as message.replyTo. Answer 200 with the replies to send, in order, in the shape a Flow Framework agent sends ({ "type": "message", "text", "buttons"? (labels, or { id, label }), "list"?, "link"?, "image"?, "replyTo"? }, { "type": "reaction", "emoji", "messageId" }, { "type": "template", ... }, …); [] sends nothing:

{ "replies": [{ "type": "message", "text": "We're open 8am to 6pm today.", "buttons": ["Order ahead"] }] }

A minimal server, with no dependencies (the whole file is examples/http-agent/server.mjs, with a test scenario beside it; it also ships in the package, in node_modules/@flow-engineer/flow/examples/http-agent):

import { createServer } from "node:http";

createServer(async (req, res) => {
  let body = "";
  for await (const chunk of req) body += chunk;
  const { conversationId, channel, message } = JSON.parse(body);
  const text = await myAgent.reply(conversationId, message.text); // your framework here
  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({ replies: [{ type: "message", text }] }));
}).listen(3000);

Each reply is checked against what the run’s platform can show, as a Flow Framework agent’s is: one that does not fit (buttons on SMS, text over the limit) is not sent, and the run says why. A status other than 200, a body that is not { "replies": [...] } or no answer within 60 seconds (--agent-timeout <s>) fails that message’s turn, with the reason in the run; the session goes on. A conversation’s messages are sent one at a time: the next waits for the last one’s answer.

Your agent’s own tools run on its side, for real, so tests/tools/*.yaml are not used, and outcome, calls and never checks (which read simulated tool calls) see none: write what must be true as judge items. The agent’s model cost shows as $0, since Flow Framework does not see its model calls. Not yet supported: agents that answer later through a callback instead of in the response, and the person’s reactions to the agent’s messages.

Any language

The endpoint’s contract is written down in spec/: agent-endpoint.openapi.yaml (OpenAPI 3.1: the request, the answer, statuses, timeouts, the token, and what each platform shows) with JSON Schemas for the request and the answer. A coding agent can implement it in any language from those files alone.

Check an endpoint against the contract before a test run, with no cloud, no key and no model call on Flow Framework’s side:

npx @flow-engineer/flow check-endpoint http://localhost:3000/flow [--token-env NAME] [--json]

It sends a fixed set of synthetic messages (a typed message, a double text, a photo, a tapped button quoting the agent, a message that may need no answer) and checks each answer: status 200, { "replies": [...] }, every reply against the schema and what WhatsApp can show, and every id a reply names (replyTo, a reaction’s messageId) one of the person’s messages. One line per case; exit 0 when all pass, else 1. The token is the value of NAME (default FLOW_FRAMEWORK_AGENT_TOKEN).

The café’s agent as an HTTP endpoint, with its two tools and one model through any OpenAI-compatible API (the same key variables as openAICompatible, MODEL for the model; FAKE_MODEL=1 for a fixed reply and no model call): examples/http-agent-python (standard library only) and examples/http-agent-go (standard library only).

Command output and reliability

Exit codes, flaky runs, retries and partial sessions

Exit	Meaning
0	The pass rate is at or above `--min-pass` (98 by default), or no run was counted, and no run errored or went unplayed.
1	The pass rate is under `--min-pass` (a flaky run counts as passed, or as failed with `--strict`), or a message took a second send path (below). Or nothing was sent: a wrong flag, an invalid `--min-pass`, a test file that does not parse, no `FLOW_FRAMEWORK_CLOUD_KEY`; the message is printed on its own.
2	The connection, the tunnel or the stream failed (before or after the session opened), the cloud answered 5xx or no JSON, the cloud ended the session for its own failure, or runs errored (the cloud could not play or judge them) or were not played (the session ended early), and the pass rate is not under `--min-pass`.
3	The cloud refused: the open (a bad or wrong key, 401/403; an invalid request, 400; busy, 429 `rate_limited`; the spend cap, 429 `spend_cap`; 503 `capacity`), or it ended the session for the spend cap or capacity.
130	Interrupted (Ctrl-C).

A busy cloud’s refused open (429 other than the spend cap, or 503 capacity) with a Retry-After of 60 s or less is sent once more after it, with one line saying so; the spend cap never is. On a cloud that resumes, a dropped stream is opened again for up to 2 minutes, picking up after the last command. On a cloud that can re-open, a call or the stream answered session_elsewhere (it reached a server of the cloud’s that does not hold the session) is sent again with backoff for up to 45 s, and the session plays on (from Framework 0.14.5); a session the cloud lost (session_gone, or session_elsewhere past those 45 s) is opened once more for only the runs with no result (the log names them, since they are played and paid for again), and the report covers both sessions with both dashboard links. Whenever a session ends before every run played (the tunnel, the stream or the connection failed, the cloud stopped it, or its summary left runs out), the runs that had already finished are printed, then N of M runs finished before the session ended early (K not played): <reason>, with the cloud’s error as the reason when it sent one, then the dashboard link; it exits 2 (3 when the cloud stopped it for the spend cap or capacity). Ctrl-C prints that same partial report, says the cloud keeps playing until it is stopped from the dashboard, and exits 130.

Flaky runs. The cloud plays a run that failed once more. When the second play passes the run is flaky: its line reads FLAKY, with played again: passed <watch> under it, and the summary lists it in its own section (1 flaky (failed once, passed when played again): with both watch links), apart from the failures. A flaky run counts as passed in the pass rate; --strict counts it as failed. When the second play fails too, the run failed. From Framework 0.16.1, every run plays the person, the tools and the judge anew, so this second play is what tells noise from a real failure. --fresh is still accepted until the release after 0.16.1 and does nothing, with a warning; drop it from scripts and CI.

Runs, pass rate and credit. --runs 3 plays each scenario three times on each of its channels. A run comes to one of four outcomes: passed, failed, credit or infra. Credit (the workspace’s daily test budget, spend_cap, or the agent’s model refused for credit, agent_credit, such as Flow’s trial credit used up) and infra (the cloud could not play or judge it, such as judge_failed) say nothing about the agent: their lines read CREDIT and ERROR, the summary counts them apart from the failures (4 of 6 runs passed. 1 failed. 1 stopped for credit (not the agent's fault, not counted as failed).), and names the flaky runs in the same line (6 of 10 runs passed. 2 flaky. 2 failed.). Each channel’s line under the pass rate is counted as the pass rate is: flaky runs as passed (as failed with --strict), credit and infra runs not counted ( telegram: 10 of 10 counted runs passed (3 flaky); from Framework 0.16.1, which also names flaky runs on the first line). A session with only credit or infra runs and passes exits 2. When the cloud stops the session for the agent’s credit, it ends with agent_credit: ... and exits 2. With --runs above 1 (or a run the cloud played again), the summary gives each scenario’s pass rate on each channel over the runs that passed or failed, flaky when some passed and some failed ( latte-correction (whatsapp): 2/3 passed, flaky), naming any credit or infra runs as not counted. A failed run lowers the session’s pass rate, which fails the session (exit 1) under --min-pass.

One send path. Every message an agent sends in a test must go through its send (its channels), which announces it with a message.out event. On a Flow Framework agent the CLI counts, per conversation, the messages that reached the cloud and the ones announced; when they differ (an app’s own wrapper or code sent one the agent never did, or the agent says it sent one that never arrived), it names the conversation and the session fails (exit 1).

Each run’s line carries its id (FAIL latte-correction #1 (whatsapp) trn_…: 6 turns, …). --dotenv <path> reads the environment from that file instead of the folder’s .env (variables already set still win; no value is printed), and --verbose also prints slow event uploads.

JSON output and saved session records

--json prints one JSON document on stdout when the command ends, on every exit once its flags are read; every other line goes to stderr:

{
  "version": 1, "outcome": "failed", "exitCode": 1,
  "sessionId": "tse_…", "agentVersion": "3f2a9c…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…",
  "error": null,
  "counts": { "runs": 2, "passed": 1, "failed": 1, "flaky": 0, "errored": 0, "notPlayed": 0, "credit": 0, "infra": 0 },
  "scenarios": [{ "scenario": "latte-correction", "path": "scenarios/latte-correction.yaml", "channel": "whatsapp", "runs": 2, "passedRuns": 1, "credit": 0, "infra": 0, "label": "flaky" }],
  "sendPath": [],
  "strict": false,
  "passRate": 50, "level": "poor", "minPass": 98,
  "costUsd": { "agent": 0.0123, "test": 0.0456 },
  "runs": [
    { "runId": "trn_…", "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "verdict": "failed", "outcome": "failed",
      "error": null, "ended": "goal_met", "turns": 6, "watch": "https://…",
      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }],
      "invalid": null, "rerun": { "runId": "trn_…", "passed": false, "watch": "https://…" }, "label": "failing" }
  ],
  "invalid": [],
  "sessionInvalid": null
}

outcome is passed (exit 0), failed or invalid (1), errored or transport (2), refused (3) or interrupted (130); from Framework 0.14.8 outcome and exitCode follow the pass rate. passRate is the percent shown, cut to one decimal (such as 99.2), level is excellent, good, fair, needs_work or poor (both null when no run was counted), and minPass is the threshold used (such as 98). A run’s verdict is errored whenever the cloud names an error for it (such as judge_failed, simulator_failed or spend_cap), whatever it ended as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads ERROR instead of PASS or FAIL, and the run’s error is that code (else null). A run’s ended says why it stopped, such as goal_met, gave_up, max_turns or, from Framework 0.16.0, walked_away (a failed run: Play a harder person). A run’s verdict is flaky when it failed and passed when the cloud played it again; rerun is that second play ({ "runId", "passed", "watch" }, else null), label is flaky or failing (it failed again) on a run played again (else null). The second play is never a run of its own in runs or counts. counts.failed leaves flaky runs out unless --strict (strict: true); counts.flaky counts them. A run’s outcome is passed, failed, credit or infra (null for one not played; a flaky run’s is failed, its first play), and counts.credit and counts.infra count those runs, which counts.errored also holds. scenarios is each scenario (by its id, with its file as path) on each channel: runs the runs that passed or failed, passedRuns how many passed, credit and infra the runs not counted, and label flaky when some but not all passed, or a run failed and passed when played again. sendPath lists each conversation whose messages did not all go through the agent’s send, as { "conversationId", "delivered", "announced" }. error is { "message", "status", "type", "retryAfterSeconds" } when the session did not finish (type is the cloud’s code, such as spend_cap; the others are null when it did not say).

A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are errored with error scenario_invalid and invalid { "path", "reason" } (the reason starts unsupported: when the cloud does not know a key), its line reads INVALID with the file and what is wrong, and the top-level invalid lists each such scenario once as { "scenario", "path", "reason" }. When the cloud ends the whole session scenario_invalid (a tools or checks file it cannot read, or an older cloud), sessionInvalid is { "reason", "olderCloud", "sent" }: what the cloud said after the code (else null), whether that suggests a cloud older than this client, and the scenarios sent as { "scenario", "path" }; the same lines are printed.

--only latte-correction,order-fails runs only those scenarios (file names without .yaml); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes .flow/last-session.json in the agent’s folder and prints its path; flow new git-ignores .flow/. It holds what each run ended as and the judge’s reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by --synthetic):

{
  "sessionId": "…", "agentVersion": "3f2a9c…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…", "endedAt": "2026-10-16T09:12:03.000Z",
  "ended": "finished",
  "only": null,
  "runs": [
    { "scenario": "latte-correction", "id": "latte-correction", "channel": "whatsapp", "run": 1, "runId": "trn_…", "verdict": "failed",
      "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }] }
  ]
}

A run of a scenario the cloud could not play also has invalid ({ "path", "reason" }), and a session the cloud ended scenario_invalid has sessionInvalid, as in --json.

agentVersion is the version the session kept: --agent-version, else, from Framework 0.10.3, the version the agent’s traces carry (the agent’s version), else null in --json and absent from the file. Two runs of different versions compare from these two files.

ended is finished, stopped (the session broke off or the cloud stopped it) or interrupted. agent is the cloud’s id for the agent (a hosted cloud names it), which flow show reads. only is the scenarios the session was limited to, null for the whole suite. A run’s scenario is the file name without .yaml; id is the scenario’s id: (from Framework 0.14.5). verdict is passed, failed, flaky (failed, then passed when the cloud played it again), errored (the cloud could not play or judge the run) or not_played (the session ended first; its runId is null), and outcome is as in --json.

Try a case by hand

Inspect the agent and tools before writing a scenario.

Deploy a checked version

Exercise staging, inspect logs and promote.

Was this page helpful?