What is the hidden cost of labelling AI training data?

Labelling training data for AI customer service has a cost the vendor's pricing rarely mentions: someone has to tag conversations with outcomes (resolved or not, customer was satisfied or not), correct transcript errors, and mark the cases where the AI was wrong. The work is real headcount, often more than the team expected, and it is the work that decides whether the AI gets better or drifts.

A team launches AI with the vendor's stock model. After a month, they want to fine-tune it on their own data. The vendor offers fine-tuning at a clear price per million tokens. What the price does not include is the labelling: someone has to mark which conversations represent good outcomes worth learning from. That work, done well, takes hours per hundred conversations.

What people in the field are saying

kdschemin's "The foundation of intelligence" series argues that the data layer underneath AI is where most value is made and lost, and the labelling effort is the part of that data layer that gets under-counted.

What does labelling actually require?

Three categories. Outcome labelling: did the conversation resolve the customer's issue, or did they come back. Quality labelling: was the AI's reply correct, was it on-brand, was it appropriate to the customer's tone. Error labelling: where the AI got something wrong, what specifically was wrong. Each requires someone with judgement reading the transcript carefully; none can be fully automated.

How much does it actually cost?

For a contact centre handling 100,000 contacts a month, fine-tuning quarterly on a 5% sample is 5,000 conversations to label. At 10 minutes per careful labelling (read, judge, tag), that is 830 hours, or about half an FTE. The vendor's invoice will not include this. The CS budget will, eventually.

Who does the labelling?

Three options. Internal CS team members part-time, which is the highest-quality and lowest-velocity option. Dedicated labellers (internal or external), which is the highest-velocity and lower-quality option (they do not know the product). AI-assisted labelling with human review of low-confidence cases, which is the middle option and the one most teams converge on after the first round.

What is the trade-off?

More labelling means the AI keeps improving and stays on-brand. Less labelling means the AI drifts faster and the per-contact quality slips a few percent every quarter. Neither is "wrong"; the team has to know which one it is choosing. The teams that under-invest in labelling tend to discover the cost six months later, in customer complaints rather than in budgets.

What is the practical pattern?

Budget for the labelling work as a real line item, separate from the vendor cost. Start with internal team members for the first quarter to build the labelling rubric; move to a hybrid (AI plus human review) once the rubric is stable. Treat the labelling pipeline as the AI's actual operating cost, because in many deployments it is.