Every support conversation scored against your own rubric, not a two percent sample

What a two percent sample cannot see

Take a subscription software company with fifteen support agents over two shifts. Email runs through Zendesk and in-app chat through Intercom, about 8,000 conversations a month between them, and one QA lead owns quality: the scorecard, the calibration sessions, the coaching.

Working carefully, she grades around 160 conversations a month, two percent, picked at random or from bad CSAT ratings. That tells her something about the average and little about the rest: a refund promised outside policy at ten at night, an account change for someone who never proved they could make it, a customer who wrote 'we are looking at alternatives' and got a macro.

Bad ratings are a weak filter anyway: a refund the customer was not owed earns five stars, and many one-star ratings are about the product, not the agent.

This build is an AI workflow that grades every conversation against her rubric, quotes the evidence for each score, and sends her the conversations worth her time. It is a fixed pipeline, not an agent, and its scores exist for coaching.

The rubric, rewritten so a model can be held to it

Scorecards say 'accurate answer' or 'showed empathy' and leave the rest to the grader. A model needs each criterion as data: a scale, the evidence it must cite, and who checks it. Six rows from the scenario's rubric:

A five-star conversation that broke the refund policy

The scorecard the agent sees, and the calls that produced it. The model only ever knows the agent as A-07.

Eight steps from closed ticket to scorecard

One model call does the scoring. What the model sees, what counts, and who may see a score are decided by ordinary code.

Calibrating the grader before anyone sees a score

The grader earns trust the way a new QA hire would: by grading the same conversations as the lead and being compared, with the methods in LLM evaluation frameworks compared.

1. Build a calibration set of 200 to 300 past conversations across channels, languages and outcomes, with the hard cases in on purpose: angry customers, partial refunds, handovers, replies written in a second language.
2. Have the QA lead and a team lead grade it independently. Where they disagree, fix the criterion's wording before blaming the model.
3. Run the grader on the same set. A criterion goes live only when the model agrees with the humans about as often as they agree with each other.
4. Shadow for a month: everything is scored, and only the QA lead sees the results, next to her own sample.
5. Report at team level first: trends per criterion and knowledge gaps, no individual scorecards.
6. Open agent scorecards criterion by criterion, with disputes available from the first day.

The set is rerun weekly and whenever the model, prompt or rubric changes. A criterion whose agreement slips goes back to shadow automatically until someone finds out why; the review patterns are in human in the loop AI.

Where automated scoring goes quietly wrong

A grader rarely fails loudly. It is wrong in one direction, consistently, until someone checks.

Before the first score reaches an agent

In Germany, a system like this needs the works council before it goes live. Section 87 of the Works Constitution Act gives it co-determination over technical systems designed to monitor employees' behavior or performance, and per-agent scoring is that. The act also says a works council assessing AI may bring in an outside expert without having to show one is needed, and Austria and the Netherlands have comparable consent rights. I bring the rubric, data flow and calibration results before the shadow month; the works agreement then settles who sees individual scores, and for how long.

The EU AI Act points the same way. Annex III lists AI intended to monitor and evaluate the performance and behavior of workers as high-risk, and scores per agent are profiling, which rules out the exemptions for narrow and preparatory tasks. After the Digital Omnibus those obligations apply from 2 December 2027. I design for them now: a person confirms anything that affects someone, every score is logged with its versions, the calibration set documents accuracy, and staff are told before it starts.

The Act has also banned AI that infers workers' emotions since February 2025. It defines that through biometric data such as voice and faces, so grading what an agent wrote falls outside it, while guessing an agent's mood from call audio would not. This build reads text and transcripts only, and its tone criterion judges the wording of a reply, never the state of the person who wrote it. Under GDPR it also needs a documented legal basis and a data protection impact assessment; GDPR-compliant AI development covers the mechanics.

Zendesk QA, MaestroQA, or a grader of your own?

Buy first if your rubric is close to the standard categories. Zendesk QA, the product that used to be Klaus, reviews every conversation with AutoQA, lets you describe custom categories in your own words, and flags churn risk, escalations and knowledge gaps. MaestroQA covers similar ground with AI analysis, custom evaluation and coaching, and connects to Zendesk, Salesforce, Gladly and Freshworks. Intercom's Pro add-on analyzes conversations too, though mainly to improve Fin. For one helpdesk and a conventional scorecard, one of these is a better answer than a build.

A custom grader earns its cost when the rubric depends on your own systems: refund limits checked against billing, identity rules tied to your product's roles, accuracy checked against your help center and known issues. It also wins when conversations span Zendesk and Intercom, when QA should be joined with CRM and churn data instead of living in another tool, and when a works council or data protection officer needs to see exactly what the model reads.

There is a point on the other side. Under the AI Act, a company that has a high-risk system built for its own use counts as its provider, with the documentation and conformity duties that carries from December 2027, while buying leaves most of those with the vendor. If your rubric is ordinary, that tips the decision toward buying. If it is not, the calibration records, logs and versioned rubric this build keeps are much of what those duties ask for.

How you would know it is working

A blueprint has no results to report, so here is what I would measure from the first week instead, on your own data.

Agreement with human graders, per criterion

The share of calibration conversations where model and humans agree, next to how often two humans agree. The gap decides which criteria are live.

Flag precision

Of the conversations flagged for policy, legal or churn risk, the share the QA lead confirms.

Disputes upheld

How often agents dispute a score and are right. Each upheld dispute is a rubric or grader bug and joins the calibration set.

Hours from flag to action

For missed data requests and legal threats, the time until someone acts.

Knowledge gaps closed

Articles written or fixed from the gap list, and whether conversations on those topics then need fewer handovers.

What a build like this costs

This is built as AI Workflow Automation, which runs $3.5K - $60K overall. A build like this one usually lands in the multi-step workflow tier: $15K - $30K, 3-5 weeks. The first working version runs on your real data well before the end of that window.

What it costs to run

One scoring call per conversation over a redacted transcript, typically one to a few cents, so 8,000 conversations a month is a modest line. A capable pinned model is worth its price here: a cheap grader that disagrees with the QA lead costs more of her time than it saves.

What moves the price

• How many helpdesks and channels feed the grader, and whether chat and email need separate rubrics
• How many criteria need your own data: billing for refund policy, product roles for identity, the help center for accuracy
• The size of the calibration set and the languages it covers
• Reporting: a Slack summary only, or scores joined with CSAT, CRM and churn data in Looker Studio or Metabase

Who this is for

• Support teams of ten to fifty agents with one QA lead and a manual sample
• Companies running Zendesk for email and Intercom for chat that want one rubric across both
• Teams whose refund limits and identity checks are specific enough that generic QA categories miss the breaches that matter
• Companies in Germany or Austria that need a QA system their works council can agree to

Questions people ask about this