Deploy agents your users can actually trust
Untested agents make up facts, break on edge cases and fail in front of users. Agent Testing runs your scenarios against a version of your agent and shows what failed and why, before you release it.
Agent Testing is available on request. Tell us what you are building and we will arrange access.
- Many scenarios in one run, common paths and edge cases
- A result for every conversation, with the reasoning
- Team review, so a person makes the call on what ships
Write the scenarios once. Run them on every version.
Testing an agent by chatting with it finds the failures you thought to look for. A suite of scenarios finds the same failures again after every change.
- 01
Write scenarios
Describe the conversations your agent has to handle: the common paths and the edge cases. For each one, say what a good outcome looks like. Write them in the product or import them from a CSV file.
- 02
Run them against an agent version
Pick a version of your agent and run the suite. Many scenarios run in one run, each as a full conversation with real tool calls, so you are not testing one chat at a time by hand.
- 03
Read the results
Each conversation comes back as Pass or Needs review, with the reasoning and the turn where it went wrong. Fix the agent, run the same suite again, and compare.
What goes in, and what the run hands back.
Three scenarios from one suite. On the left, the scenario and the outcome you asked for. On the right, the result for that conversation. The agent and the scenarios are invented.
A customer asks for a refund 45 days after purchase. The policy allows 30.
Declines the refund, explains the window, offers store credit, stays courteous.
Needs review
Verdict
- Turn 3 offered a full refund
- Tone stayed courteous throughout
Reasoning
- Rule: Refunds within 30 days of purchase
- Found: The agent did not check the purchase date
Suggested fix
- Look up the purchase date before answering a refund request
Illustrative · fictional agent
The agent offered a refund the policy does not allow
The scenario asks for a refund after the refund window has closed. The agent was polite and wrong. The result names the turn, quotes the rule it was measured against, and says what to change.
Every conversation, measured against the outcome you wrote.
The evaluation reads each conversation turn by turn and gives the same analysis on every run. Issues are labelled by severity, so you fix what blocks a release first.
Made-up and missing facts
Answers that state something your sources do not support, and answers that leave out what the customer needed.
Policy and scope
Replies that break a rule you set, or that give advice the agent should have declined or handed to a person.
Tone and brand voice
Whether the agent sounds the way your team would, measured against the expectations you wrote for the scenario.
Tool calls
Whether the agent called the right tool with the right parameters, and what it did when the tool failed or returned something unexpected.
The people who know the work read the conversations.
A result is a starting point. Reviewers outside engineering read the conversation as written and flag what they would not have said.
Customer service
“Would I say this to a customer?”
Reads the conversation as written and flags replies that do not meet the service standard. No technical tooling to learn.
Legal and compliance
“Could this create liability?”
Catches risky advice and rule breaches before release. Conversations are stored with their timestamps.
Product and marketing
“Are the product details correct?”
Flags outdated features, wrong prices and off-brand wording, so the agent’s facts and voice stay current.
Your teamDecides what ships. Agent Testing shows the evidence for each conversation.
Build, test, deploy. Then do it again.
Agent Testing is the middle step of Agent Builder. What you learn in production becomes the next scenario in the suite.
- 01
Build
Describe the agent, connect its tools and save a version in Agent Builder.
Agent Builder - 02
Test
Run the suite against that version. Fix what needs review and run it again until the results hold.
You are here
- 03
Deploy and watch
Release the version and read real conversations and traces in Agent Insights.
Agent Insights
Questions about Agent Testing.
How do I get Agent Testing?
Agent Testing is available on request. Use the Request access button, tell us what your agent does and what you want to test, and we will reply to arrange access.
What is a test run?
You write scenarios, each with a description of a good outcome. A run plays many scenarios against one version of your agent as full conversations. Each conversation comes back as Pass or Needs review, with the reasoning and the turn where it went wrong.
Can I test agents built outside Agentman?
Agent Testing works most closely with agents built in Agentman Studio. If your agent was built elsewhere, tell us how it is reached when you request access and we will say what is possible.
How much should I rely on the AI evaluation?
Treat it as a consistent first read. Every conversation is measured against the outcome you wrote and gets the same analysis on every run, with its reasoning shown. For the scenarios that matter most, add team review so a person makes the call.
Who can review the results?
Anyone on your team. Conversations are shown as written, so customer service, legal and compliance, and product can review them and flag a reply without learning a technical tool.
What about sensitive data?
HIPAA, ISO 27001 and SOC 2 ready. Compliance audits underway. The Security and Governance page describes how data is handled, and we can walk your security team through it before any data is shared.
HIPAA, ISO 27001 and SOC 2 ready. Compliance audits underway.
Review security and governanceFind the failures before your users do.
Agent Testing is available on request. Tell us what your agent does and what you want to test.