How do you A/B test an AI customer service deployment?

A/B testing an AI customer service deployment means routing a fraction of contacts to the AI and the rest to the prior workflow, while controlling for selection (the AI gets a different mix), measuring outcomes that survive AI's edge cases (resolution, re-contact across channels, downstream churn), and reading the results over weeks rather than days. The shape is different from a web A/B test, and using the web shape produces wrong answers.

A team launches AI for ten percent of incoming chats. After two weeks, CSAT on the AI side is two points higher. The vendor case study writes itself. Twelve weeks later, the team realises the AI side had been routed to the easier contacts by an early filter, and the actual resolution rate was the same. The A/B test was honest about the numbers and wrong about what they meant.

What people in the field are saying

Gradient Flow's "Agentic AI: challenges and opportunities" argues that AI systems behave well on benchmarks designed for them and badly on the edge cases that matter most. The same shape applies to AI A/B tests in customer service: the test usually measures the part the AI is good at.

How do you set up the test honestly?

Random assignment at the contact level, before any triage that the AI itself might do. If the AI does the routing, the AI gets to pick its own contacts. The control needs to be the actual prior workflow, not an idealised version of it. The traffic split has to be large enough to detect the effect size you care about, which is usually larger than the team would like.

What should you measure?

Not CSAT alone. Resolution from the customer's perspective (did they come back about the same issue within a window). Re-contact across all channels, not only the channel of the original contact. Average handle time on the human side of the comparison (the AI's residual cases on the human side often take longer; this is real cost). Downstream churn, if you can attribute it. Each of these is harder than CSAT and closer to the truth.

What confounders should you watch for?

Selection: any pre-AI filtering that gives one side an easier mix. Time: the AI side improves over the test as the team tunes the prompts, while the control is static. Channel mix: the AI is on chat, the control includes voice; you are not comparing like with like. Customer composition: returning customers cluster in one arm because of how IDs are assigned.

How long should the test run?

Long enough that re-contact, downstream churn, and the rare-but-expensive edge cases have time to show up. Two weeks is rarely enough; four to eight weeks is more honest. The temptation to call the test early is strong because the team wants the result; resist it. The CSAT signal on day three is the signal the AI is good at producing; the signal you care about lives further out.

What does "passing" the test mean?

Better customer outcomes on the metrics that matter, holding the team size constant or with credible savings on the human side. AI that performs the same on real outcomes but saves a third of headcount is a different decision than AI that performs better and costs the same. Both are wins; the team has to be honest about which one it found.