Testing agents

A test suite is a set of simulated phone conversations for one agent. In each case, a simulated caller talks to your agent in text, and your checks decide whether it passed. Run the suite after you change a prompt, a tool or a flow to see what broke before a customer does.

A case

{
  "name": "Asks for a refund after 35 days",
  "caller": {
    "mode": "persona",
    "persona": "Marta, 54, impatient. She bought a blender five weeks ago and it stopped working.",
    "goal": "Get a refund or a replacement",
    "max_turns": 6
  },
  "dynamic_variables": { "store": "Acme" },
  "tool_mocks": { "lookup_order": { "status": "delivered", "days_ago": 35 } },
  "checks": [
    { "type": "tool_called", "name": "lookup_order" },
    { "type": "not_contains", "text": "full refund" },
    { "type": "rubric", "criteria": "The agent explains the 30-day policy and offers a replacement or store credit." }
  ]
}

The caller

If the agent has a first message it speaks first, as on a call.

Dynamic variables and tools

dynamic_variables fill {{placeholders}} in the agent's prompt and first message, over the agent's own defaults. tool_mocks maps a tool name to what it answers (any JSON). A tool without an entry answers {"status": "ok"}. Every call the agent makes is recorded with its arguments, so you can check them.

A flow agent runs through the same flow engine a call uses, but without its function nodes: those take their failure edge. tool_mocks apply to prompt agents only.

Checks

A case needs at least one check, and passes when all of them do.

Everything except rubric is checked exactly, with no model. All of a case's rubrics are graded in one call. There is no regular-expression check on purpose; use contains, or a rubric.

Give a check a label to have it named in the results.

Running a suite

curl -X POST https://api.dialgem.com/v1/simulations/suites/SUITE_ID/runs \
  -H "X-API-Key: $DIALGEM_API_KEY" -H "Content-Type: application/json" \
  -d '{ "agent_version": 0 }'

agent_version is 0 for the draft you are editing (the default), a published version number, or "latest_published". The run starts in the background and answers 202; poll GET /v1/simulations/runs/{id} until status is completed. Each result has the full transcript, every tool call, and the outcome of every check.

A case is passed, failed (a check did not hold), errored (the model or the case itself broke, which says nothing about your agent) or skipped (the run was cancelled or ran out of budget).

What a run costs

A run is bounded in three ways:

• Model turns. An agent reply, a simulated caller line and a judge verdict each count one. A run's limit defaults to the most its cases can use; set max_model_turns to lower it. When it is used up, the remaining cases are skipped, not failed, and the run says budget_exhausted. The platform also caps it.
• Runs. An organization can have a couple of runs in flight and a limited number a day.
• Reuse. A case that already ran against exactly this agent content (prompt, settings, tools; the agent's name does not count) with the same definition is not run again: the earlier result is shown, marked as cached, and costs nothing. Pass "fresh": true to run everything again, which is worth doing for persona cases, since their results vary. Errors and skipped cases are never reused.

Editing a case, or changing the agent, naturally means it runs again. Renaming a case does not.

The publish gate

A suite can guard publishing. Set its publish_gate:

"Has passed" means: the latest finished run of the suite tested the draft exactly as it is now, covered the suite's current cases, and every case passed. Change the agent or a case and the earlier pass no longer counts. Because publishing copies the draft, a pass on the draft holds for the version published from it.

Run the suite against the draft (agent_version: 0), and when it passes, publish.