Agent Simulation: testing voice and chat agents before real users do.
Teams shipped voice and chat agents built on happy-path conversations, and they broke the moment real users hit an edge case. I designed a simulation environment where anyone can stress-test an agent against synthetic customers, then go from failure to fix without writing code.
- 3
Building blocks
Agent Definition, Scenarios, and Personas, composed freely at run time.
- 4
Scenario types
Graph, Dataset, Script, and SOP: manual, structured, and AI-generated test cases.
- 18
Pre-built personas
Configurable for voice and chat, plus fully custom personas.
Case snapshot
- Role
Sole product designer: the end-to-end creation flow, the concept architecture, and the UI for all three building blocks
- Team
1 designer (me) + 2 ML engineers + 1 PM
- Key constraint
Non-technical teams had to go from a failed call to a fix without writing a line of code.
AI agents were shipping untested.
Teams shipped voice and chat agents built on happy-path conversations. Agents that demoed well broke the moment real users hit an edge case, and there was no way to reproduce the exact conditions that caused it.
The question every team was asking: how will the agent behave with real users? Three gaps showed up everywhere.
Only the happy path
Tests covered the conversations teams expected, not the frustrated, confused, or off-script ones real users bring.
Only the final answer
Teams judged the last reply, not the turns, tool calls, and handoffs that led to it.
Failures hit production
Edge cases were found by customers first, with nothing to replay or reproduce.
Let anyone stress-test an agent against synthetic customers, then go from failure to fix without writing code.
Sessions with ML engineers, PMs and QA. Four things we learned.
I paired research on simulating human behaviour with sessions with the teams who would run the tests.
- Stanford: generative agent simulations
A short description of a person gets 70% accuracy; a real interview transcript gets 85%. Cutting 80% of the interview barely moved it. Relevance beats volume.
Stanford HAI policy brief - AgentSimulator (SAP Signavio)
Accuracy comes from modelling the individual: their schedule, skills, and habits. Explainability is what makes the output usable.
AgentSimulator paper (arXiv)
How you build the persona decides how good the simulation is.
Four patterns in how teams tested
Agents are not chatbots
Users kept pattern-matching agents to single-turn chatbots. The UI had to teach the multi-step, tool-using mental model.
Failure lives in the middle
Agents broke on a turn, a tool call, or a misheard word mid-conversation, long before the final answer.
Can't reproduce the bad call
A flaky failure in production couldn't be recreated, so it couldn't be fixed with confidence.
“Tested” meant a few chats
Quality was judged from a handful of samples. “Failed 12 of 200 runs” beats “here's one transcript that broke.”
Seven tools, all missing the same thing.
Each tool solved an adjacent problem well. None built a bridge from a failed simulation to a fix, and none designed for QA and PMs as primary users.
Voice & chat agent QA, built around call-center-style workflows rather than a composable test system. No reusable agent-definition versioning or persona library.
Simulation is one module inside a larger eval suite, with no equivalent to graph, dataset, script, or SOP scenarios for non-technical teams.
Observability-first, not simulation-first. It catches failures after they've hit users, with no synthetic persona system.
Synthetic customers, realistic scenarios, pre-production stress testing, and a failure → diagnosis → fix loop that non-technical users can run without code.
Three questions, three building blocks.
Who is the agent? What does it face? Who is the customer? Each question became its own block, with its own entry in the FutureAGI sidebar under Evaluate.
- Agent definition
What AI agent is being tested? How does the platform connect to it? Voice or chat? Which provider, which system prompt, which version?
- Scenarios
What will the simulated customer want or do? What paths, objections, and outcomes should the agent be tested against?
- Personas
Who is the simulated customer? Their age, location, communication style, speed, and emotional tone.
Four principles I held myself to.
Each one came with an anti-pattern I wanted to avoid, and together they framed a single cycle: define, simulate, evaluate, diagnose, iterate.
Both modalities, one mental model
A scenario runs as voice or chat. Same persona, same rubric, two outputs; not separate voice and chat tabs.
Designed for the whole pod
QA can build a persona without a YAML schema. PMs can read results without asking what a raw score means.
Every failure has a next step
A failing run leads to a fix, a regression test, or a dataset row; never a dead-end results page.
Trust comes from volume
Show the distribution, not the anecdote. Never judge quality from a single sample.
Blocks are independent and reusable. Create one agent definition, five scenarios, and eight personas, then compose them freely across any number of runs.
Block 01 · Agent definition
connecting the platform to any agent.
An agent definition is a configuration that specifies how an AI agent behaves during voice or chat conversations.
Agent type
Voice or chat, each with channel-specific settings.
Prompt
The agent's behaviour definition.
Versioning
Every save creates a snapshot, so a run can target a specific version.
Provider + assistant ID
For outbound voice agents, the platform calls on behalf of the customer.
Knowledge base
What the agent is allowed to know.
Observability provider
Optional. Auto-creates a project in Observe under the agent's name.
Prompts get tweaked and models swapped several times a day. A green run means nothing unless you know which version it tested, so every version carries a status: Active, Draft, Archived, or Deprecated.
Block 02 · Scenarios
four ways to define what the agent faces.
Scenarios define the test cases, customer profiles, and conversation flows an agent will encounter during simulations.
- Workflow
A visual conversation flow with branching nodes.
- Dataset
Upload a CSV or Excel file, or generate synthetic data.
- Script
Paste a call script with customer and agent lines.
- SOP
Numbered steps converted into a graph, for verifying documented procedures.
Block 03 · Personas
who the simulated customer actually is.
Personas show how the agent deals with people. Voice and chat personas needed genuinely different controls, not the same form with different labels.
- Voice persona
- Chat persona
Personas carry mood and escalation rules, not just a script: interrupt an up-sell, escalate after a second deflection, ask for a human after four turns.
Three rounds to get the scenario structure right.
Each round fixed the problem the previous one exposed.
Problem:
A flat list of test casesSolution: A branching scenario graph
Round 01 borrowed from unit testing: one input and one expected output per row. Teams couldn't model branching conversations.
Problem:
Build the graph, then pick personasSolution: Graph and personas reviewed together
Round 02 separated the two steps, and users lost the connection between who the persona was and how the graph read.
Flat list vs. graph
- Scenarios as a flat list
Simple and familiar, but it assumes every test starts from the same place. It treats an agent like a function.
- Scenarios as a graph
Each node is a conversation state, edges represent branches, and global nodes handle errors once, referenced everywhere.
A flat list of scenarios treats an agent like a function. A scenario graph treats it like what it actually is: a decision tree with branching paths and compounding failures.
Feature 04 · Three blocks, one run
running a test: pick the blocks, add evals, run.
The run
Add evaluations
Search built-in evals across chat, audio, and quality, or create a custom one.
Run the simulation
Name the run, attach evaluations, and start.
Read the results
Run distribution, system and evaluation metrics, and every call with its branch, persona, and outcome.
Fix the agent
AI compares the ideal outcome with what happened and suggests exactly what to change.
Simulation as the center of the loop.
Simulation doesn't end at a score. Every run flows into the rest of the platform.
Evaluation: automatic scoring
Every conversation is scored against attached evals: CSAT, hallucination rate, task completion, tool call accuracy.
Observability: replay & debug
Traces flow into Observe, so teams replay any conversation turn by turn without reconstructing it from logs.
Fix My Agent: diagnosis to action
One click from a failed run to the exact config field that caused it. The most-loved feature in beta.
Shipped to beta, working as intended.
Three teams were running simulations in beta without engineering support.
- 60–70%
Less testing effort
Through automated scenario generation.
- 50%
Faster validation
Cut in pre-production validation time.
- 2×
Earlier detection
Faster issue detection before launch.
“This is the first time I can see why my agent failed — not just that it failed.”
What designing a test environment for AI taught me.
The three-block separation was the most important design decision. Voice and chat personas needed to be genuinely different, and versioning on agent definitions was a product insight disguised as a feature request.
Defining the words (simulation, trace, evaluation) mattered as much as defining the screens. The vocabulary became the product.
Two things I'd do differently
Pull non-technical users into round one
Their vocabulary for what's broken is more direct than an engineer's.
Ship “compare two runs” at launch
Teams immediately wanted to diff a before-and-after run, and needed workarounds until it shipped.