The 30-second answer

Without RAG, an AI agent answers from what its model learned during training — months old, generic, often wrong about your specific product. With RAG, the AI first looks up the relevant article in your knowledge base, then writes the reply using that article as the source. The answer is grounded in your real product, not the model's memory.

How it works, in two steps

  • Retrieve. When the customer asks "what's your return policy?", the system searches your knowledge base — help centre articles, internal docs, product specs — for the most relevant passages. Usually it finds your returns help article and any related FAQ.
  • Generate. The AI writes the reply using those passages, often quoting them or citing them. The bot stays inside your product's facts.

The retrieval step uses embedding search — turning the question and every article into numerical vectors, then finding the closest matches. The generation step is a normal large-language-model call, but with the retrieved articles included in the prompt as "here are the relevant sources, answer based on these."

Where it sits in an AI agent

Almost every modern AI-agent vendor uses RAG. Intercom Fin, Decagon, Ada, Forethought, Sierra — all of them. It's not a feature you add; it's the default architecture for a 2026 support AI.

Where they differ is in how well they do it. Better retrieval (finds the right passage); better generation (stays faithful to the source); better fallback (says "I don't know" instead of guessing). The vendor names look interchangeable in the brochure; in practice the quality difference is significant.

Common failure modes

  • The knowledge base is wrong. RAG confidently repeats whatever it finds. Stale or inaccurate articles produce stale, inaccurate answers — at scale, for every customer.
  • The knowledge base is incomplete. The bot finds nothing relevant and either says "I don't know" (the honest case) or hallucinates from training data (the bad case). Most vendors let you tune which.
  • Retrieval misses. Even with a good knowledge base, the bot's search returns the wrong article and answers from the wrong source. Hard to detect without sampling — and most teams never sample.
  • The model paraphrases too freely. The retrieved article says "14-day return window"; the bot replies "about two weeks." Close enough most days, wrong on the edges.

How to evaluate RAG quality in a pilot

Run 50 representative customer questions through the vendor's agent — pointed at your knowledge base, not their sample one. For each answer, check three things:

  • Was the source it cited the right one?
  • Was the answer faithful to that source?
  • Was anything in the answer not in the source — i.e. fabricated?

The vendor will offer a polished demo on their corpus. Insist on yours. The first 50 questions usually surface every failure mode at the same time.

What this means before you buy

Most teams discover their internal docs are messier than they thought once they point an AI at them. Outdated articles, contradictory FAQs, undocumented edge cases. Cleaning the knowledge base is usually a bigger lift than configuring the AI — and is often the project that should happen first regardless of which vendor you pick.