What is the right model-update cadence for AI customer service?

The right model-update cadence balances drift (the model getting stale relative to current language, behaviour patterns, and your own policy) against disruption (every update changes the AI's behaviour, which the team has to learn and audit). The empirical pattern is quarterly major updates, weekly prompt updates, monthly knowledge-source updates. Each cadence has a different cost; each is tuned to a different thing.

A vendor pushes a model update on a Tuesday. By Thursday, the team notices the AI is now resolving cases differently, sometimes worse on a contact type that mattered. The vendor's tests were on their benchmark, not on the team's contact mix. The update was good for the average and worse for the team's specific cases. This pattern repeats unless the cadence is managed deliberately.

What people in the field are saying

kdschemin's "Reliability is the product" argues that AI reliability is a continuous engineering discipline, and model updates are the most disruptive event in that discipline. The cadence has to be managed, not just received from the vendor.

Why not just take every vendor update?

Because each update changes behaviour you have not tested. The vendor's evaluation set is not your contact set. An update that is net better on the vendor's benchmark can be net worse on your specific work. Without a buffer between the vendor's release and your production, you absorb the regression.

What cadence works for what?

Major model updates: quarterly, with a one- or two-week shadow-mode test on your real traffic before promoting. Prompt and procedure updates: weekly, low-stakes, easy to roll back. Knowledge-source updates: continuous as content changes, with a content owner. Each has its own risk profile; collapsing them into one cadence ignores the differences.

What is shadow mode?

Running the new model in parallel with the current one, on the same incoming traffic, without serving its responses to customers. The team reads the diffs (where do the two models answer differently, and which is better). Differences that look like regressions get investigated before the new model is promoted. The cost is the parallel compute; the value is catching the regressions before customers do.

Where does this go wrong?

Teams that take every vendor update absorb regressions monthly. Teams that block updates entirely fall behind on real model improvements that would have helped them. Neither extreme works. The middle (shadow test, then promote) takes process discipline that not every team has.

What is the practical first step?

Lock the vendor's auto-update behaviour to your release cadence, not theirs. Set a quarterly major-update calendar; schedule the shadow window for the two weeks before. Make rollback a one-command operation that anyone on the team can execute. The discipline pays back the first time you catch a regression before customers see it.