The conversational AI sales process has a familiar shape. You get a demo. On the screen, the chatbot understands a tricky question, answers it smoothly, handles a follow-up, and sounds almost human. It is impressive, and it is meant to be. The demo is built to show the front end at its best. The front end is also the part of the product that matters least.
Nick Clark, who writes the Service Matters newsletter, has watched a lot of these purchases. His advice on how winning companies choose a conversational AI partner is blunt: the demo is the wrong thing to judge. This article explains why, and gives you the questions to ask instead.
Every modern chatbot demos well
A few years ago, the demo told you something. Some chatbots understood natural language and some did not, and the demo showed the difference. That gap has closed. The language models underneath are now good enough that almost any vendor can show a smooth conversation. The demo no longer separates a strong product from a weak one. It only tells you the vendor can run a demo.
So the common buying instinct, judge the chatbot by how well it talks, now leads you wrong. The thing that used to be the test stopped being the test.
The parts that decide it are the ones you cannot see
What actually decides whether a conversational AI works in your contact centre is the back end: the tools for controlling, correcting, and managing the agent once it is live. Clark calls this the real difference between vendors. It is also the part no demo shows, because it is not exciting to look at.
Three things sit back there. Guardrails: the limits that stop the agent doing something it should not. Controls: how you correct the agent when it gets something wrong, and how fast. Quality tooling: how you check, across thousands of conversations, whether the agent is doing a good job. A vendor can have a polished front end and almost nothing behind it.
Connecting the chatbot to your systems is most of the work
There is a second thing the demo hides. Juan Martin Maglione, also writing in Service Matters, points out that orchestration is not one problem. Connecting the chatbot to your systems, getting it to carry out steps, and getting the model to use those systems correctly are three separate jobs.
A demo runs in a clean setup where all three are already solved. Your setup is not clean. The work of joining the chatbot to your real systems is most of the project, and the demo skips all of it.
The questions to ask instead
Shift the buying conversation away from the front end. Ask the vendor to show you the back end. Five questions do most of the work:
How do I set limits on what the agent can do, and are those limits enforced by the software or only written in instructions? When the agent gets something wrong, how do I correct it, and how long does that take? How do I check the quality of thousands of conversations without reading them all by hand? How does the agent connect to my systems, and who does that work? And what happens to the agent's behaviour when your model provider ships an update?
Where this leaves the demo
The demo still has a small use: it confirms the front end works. The decision, though, sits in the back end: the guardrails, the controls, the quality tooling, and the work of connecting to your systems. Those are what you live with after the demo is over.