How do you test AI before launching it to real customers?

Test AI before launch by replaying a sample of real past contacts through it, by running synthetic edge cases the production system will see (failed lookups, ambiguous intents, mid-conversation language switches), and by red-teaming with adversarial customers (social engineering, prompt injection, deliberately confusing requests). The vendor's demo is the floor; the bar is what your worst hour of contacts looked like last year.

A vendor demo handles a customer asking about an order beautifully. The buyer signs. Production hits in week one: a customer types two requests in one message, the order number has a typo, the language switches mid-sentence. The AI handles none of those gracefully. The demo never included them. Production tests have to.

What people in the field are saying

EmergingAI's "Agentic workflows: the tutorial..." argues that the gap between a well-handled demo and live performance is mostly about edge cases, and the only way to know about them is to confront the system with them before customers do.

What does replay testing look like?

Take a stratified sample of contacts from the previous quarter: routine, complex, edge, angry, multilingual. Run them through the AI. Read what came back. Score for correctness, tone, and escalation behaviour. Most teams find the AI handles 70% of the sample well, 20% with minor issues, and 10% badly. The 10% is where the work is.

What edge cases should you build deliberately?

Failed lookups: the order number is wrong, the customer is not in the database. Ambiguous intents: "I want to change my plan" can mean upgrade, downgrade, cancel. Multiple intents: "refund my last order and update my address." Language switches: a sentence that mixes two languages. Distressed tone: angry, all-caps, repeated. Misinformation: a customer asserting something incorrect about their account.

What does a red-team test look like?

Adversarial inputs designed to break the AI. Social engineering: a customer claiming to be someone else, with details they "remember." Prompt injection: a customer message that tries to make the AI ignore its instructions. Policy probing: a customer asking the AI to bend a rule, to see if it will. Each is a failure mode in production; the red-team's job is to find them before customers do.

What is the pass bar for launch?

Not "the AI handles X% correctly," because the X is unfalsifiable in advance. The bar is: every failure category the testing surfaced has a clean response (escalate, refuse, ask for clarification), and the team has read the AI's responses and signed off. A launch with the testing untouched is a launch in production.

How long should pre-launch testing take?

Two to six weeks for a focused launch, depending on contact-type complexity and regulatory load. Less than the cautious team's six-month pilot, more than the vendor's two-week onboarding. The shorter the test, the more time the team will spend fixing things in production.