Introducing Experiment Hub: the latest experimentation platform on Decagon

Last year, we introduced experimentation and A/B testing on Decagon, and it has since become a core part of how teams improve their agents, with teams already running hundreds of experiments.

As they get more comfortable experimenting, they’re testing changes that are more specific than broad hypotheses, like the wording of one escalation step or a shorter greeting. Likewise, the results should be just as granular.

That raises two challenges. First, the Decagon agent is live and constantly evolving, often with several people updating it at once, but an effective test needs to isolate one change. Second, making sense of experiment outcomes and the takeaways usually takes specialized analysis. Teams need to know a lift is real, and why one version won, before they can confidently take the next step.

Today, we’re introducing Experiment Hub, the next generation of Decagon’s experimentation platform. It’s built so any team can test a change, have confidence in the result, and ship the winner on their own. With Experiment Hub, teams can:

• Keep improving their agent while experiments run: test one change while the rest of your agent keeps shipping, without choosing between learning and improving.
• Reach decisions faster with results analyzed automatically: instantly get a clear verdict and explanation of why a variant won.
• Pinpoint the impact of every change: know exactly which change moved resolution, CSAT and quality, so you can confidently roll it out everywhere.
  ‍

“Experimentation is how we make sure every change to our member support Al agent actually improves the experience for our members. With Decagon's Experiments, we could test a tailored welcome message on voice with a small slice of traffic and get a clear read on its impact without slowing down anything else we were shipping. Having results visible in one place gives our team the confidence to decide what to roll out” - Harsha Bhangdia, Senior Program Manager, Chime®

‍

Keep improving your agent while experiments run

Teams ship agent updates every week, even while experiments are running. Experiment Hub keeps each test focused on one change, so teams know exactly what drove the result and can keep publishing other updates in the meantime.

Say a team wants to know whether asking a clarifying question before connecting a customer to a representative leads to more resolved conversations than connecting them right away. With Experiment Hub, the team adds an experiment block to that step of Agent Operating Procedures (AOPs) and writes each version inside it. Teams can experiment the same way on other parts of the agent, like testing variations of a welcome message or testing a new voice profile. 

Each experiment block holds every variant’s content and traffic split, with up to 20 variants compared against a shared control. While the test runs, only the versions being compared are locked. Teams can keep editing and publishing everything else, including other steps in the same AOP.

Teams can also ask Duet to set up an experiment for them. Duet drafts the experiment, writes the experiment block with each variant, and checks what needs to be published before the test starts. Before any live traffic, teams can also check each variant in Simulations against their golden test set. Once the test is live, traffic ramps up gradually. Alerts notify the team if a key metric starts to slip, and the test stops automatically if a variant performs significantly worse. When the results are in, the team picks the winner with one click. The AOP updates to the winning version automatically and publishes like any other change.

Reach decisions faster with results analyzed automatically

Knowing whether a result is significant requires technical and statistical analysis, and understanding why it happened takes someone to pull the metrics and conversations together. Most CX teams don’t have an analyst free for every test.

Experiment Hub does the analysis and surfaces the outcome in an obvious way. Up front, test sizing shows how many conversations each variant needs and when the test will be conclusive at current volume. Once results come in, a verdict at the top of the page shows each primary metric’s effect, its confidence interval, and whether it’s significant, with built-in safeguards that keep the results trustworthy.

Each result comes with a plain-language summary of what differed between variants, drawn from the hypothesis, metrics, and sample conversations. From there, teams can ask Duet follow-up questions to dig into why a result happened.

Pinpoint the impact of every change

Experiment Hub measures each change only on the conversations it reached, using the metrics teams already report on.

Each conversation keeps the same variant from the moment it reaches the experiment block, even if a new agent version publishes mid-test. Results count only conversations that reached the part of the agent being tested, so ones the change never touched don’t dilute the result.

Teams can choose from over 20 conversation-specific metrics, grouped by what they measure. Examples include:

• Outcomes: resolution, CSAT, and NPS
• Quality and safety: instruction following, rule violations, and conversationality
• Speed: time to first token and end-to-end latency
• Effort and cost: turn count and cost per conversation

Book a demo to learn more about how Experiment Hub turns every agent update into a measurable improvement.