Most companies roll out an AI support agent the same way. For the first few weeks, a person checks its answers. The answers look good, so the checking stops. From then on, the agent works on its own. This feels like normal progress. It is also where a lot of the problems begin.
The mistake is treating the choice as a simple on or off: either a person checks the agent, or nobody does. A better way is to picture a dial. Some tasks are safe to leave fully to the agent. Some need a person to look before anything happens. Most sit somewhere in between. The dial sets how much checking each task gets. This article explains how to set that dial, and how to keep it right as you add more agents.
Match the checking to what a mistake would cost
Start with one question for every task the agent can do. If it gets this wrong, how bad is the result, and how hard is it to undo?
Take two examples. The agent tells a customer the wrong opening hours. That is easy to spot and easy to fix. The agent issues a large refund to the wrong person. That is expensive and hard to claw back. These two tasks carry very different risk. Giving them the same amount of checking makes no sense. Watch the refund task closely. Leave the opening-hours task alone.
That is the whole idea behind the dial. Checking should follow risk, task by task. Nick Clark, who writes the Service Matters newsletter, makes the same point about managing AI agents: an agent is not simply on or off. It can do a little or a lot, and you decide how much.
Three levels of checking
Three levels are enough to start. Sort every task the agent can do into one of them.
Low risk. Cheap to get wrong, easy to undo. Answering a question, finding a help article, checking the status of an order. Let the agent do these on its own. Read a sample afterwards to make sure it stays accurate.
Medium risk. A real cost if it goes wrong, but a limited one. Moving an appointment, applying a small credit, changing a contact detail. Let the agent do these too, but record each one so a person can review it quickly and undo it if needed.
High risk. Expensive, hard to reverse, or emotional. Large refunds, closing an account, anything involving a complaint or an upset customer. Here the agent prepares the action, and a person approves it before it happens.
Clark argues that this kind of work, setting limits and checking them, is the interesting part of the job, not a chore. He is right. Sorting tasks into three levels is what lets you give the agent real freedom on the safe tasks without losing sleep over the risky ones.
Build the limits into the system, not the instructions
There are two ways to tell an AI agent what it may do. One is the prompt: the written instructions you give it, in plain English. The other is permissions: what the software actually allows it to touch.
The prompt is only guidance. A clever customer can talk an agent out of its instructions, or the agent can simply drift past them. Permissions are firm. If the agent has no tool that can issue a refund above a set amount, it cannot issue one, whatever the conversation.
So the rule is simple. Put each limit where the software enforces it, in the agent's tools and permissions, rather than in the written instructions. Kevin Davis, who writes the KD Be Schemin newsletter, makes the same point about AI controls in general: the ones that matter are enforced automatically, the moment the action happens, not the ones filed away in a policy document.
Things change, so check again over time
A task that was safe last month can stop being safe. The model gets updated. The product changes. A new group of customers arrives with questions the agent handles badly. The setting you chose once was chosen for conditions that have since moved on.
When you have a few agents, a person can watch them. Once you have many, that stops working, and the agents need to be watched automatically. Clark suggests tracking a few simple signals: questions the agent failed to understand, whether finished tasks were done correctly, the customer's mood through the chat, and any change in behaviour after a model update. When the warning signs rise for a task, move that task to a stricter level of checking until you know why.
Where to start
Start cautious. Put every new task at the medium level, recorded and reviewable, even if it looks safe, because you do not yet know how it will fail. A task earns more freedom by building up a clean record over time, not by having a quiet week or two.
Setting and adjusting the dial is more work than flipping a switch, and it does not show well in a demo, which is why many teams skip it. It is also the approach that holds up once real customers and real money are involved.