Quality assurance for AI support responses

Teams that would never ship code without tests routinely deploy AI answers into a live support queue with no quality process at all, then use complaints as the feedback loop. Complaints are a terrible instrument: they arrive late, they are dominated by the angriest fraction of cases, and they say nothing whatsoever about the answers that were quietly mediocre — which is most of the ones that matter.

AI answer QA borrows almost everything from human support QA, with two adjustments. The volume is higher, so you sample harder. And the failure mode is different: AI errors are confident and systematic rather than tired and random, which makes them easier to find deliberately and considerably more damaging when you do not look.

Reviewing every AI response is neither achievable nor useful. The purpose of QA is to estimate quality and find patterns, and a well-chosen sample does both better than an exhausted attempt at full coverage.

Stratify rather than sampling uniformly. Take a genuinely random slice across all categories to get an unbiased baseline number, then oversample where failure is likely and expensive: categories recently promoted to autonomous mode, answers the system itself flagged as low confidence, anything touching access or money, and everything that reopened or escalated. The random slice tells you where quality is. The targeted slices tell you why, and they are where the fixes come from.

Fix a cadence and a sample size you can sustain indefinitely. A consistent twenty reviews a week beats an ambitious two hundred that happen in month one and never again, because the entire value of QA is the trend line, and a trend line with a gap in it is an anecdote.

Score each sampled response against four independent criteria, each recorded as a plain yes or no plus a note. Independence is the point — collapsing them into a single quality percentage destroys exactly the information you need to act.

Correct: is the answer factually right for this requester’s situation, not merely right in general. Grounded: is it supported by a source the system actually retrieved, or did it fill a gap from general knowledge. Complete: does it resolve the whole request, including the second thing the person asked halfway down. Appropriately escalated: did it hand off when it should have — and equally, did it escalate something it was perfectly capable of handling, which is a real cost that nobody logs.

Grounded is the criterion people leave out and the one that predicts future failures. An answer that is right without a retrieved source is right by luck, and it will keep being right until the underlying fact changes, at which point it becomes confidently wrong with no warning. Keep the notes as well as the scores: the aggregate tells you whether quality moved, the notes tell you what to fix.

Before a category goes autonomous, run it in suggest mode and measure what humans do with the draft: send it unchanged, edit it, or discard it and start over. That distribution is a quality signal generated by work people were already doing, at no additional cost and across far more tickets than you could ever review by hand.

Read it with two cautions. Agreement measures acceptability, not correctness — an agent who accepts a wrong answer produces high agreement and a bad outcome, which is why sampled rubric review has to continue alongside it rather than being replaced by it. And agreement inflates under time pressure: a busy queue converts review into rubber-stamping, and the rate climbs for reasons that have nothing to do with the model. Track edit rate and discard rate separately from acceptance, because a category where drafts are accepted but heavily rewritten is not ready however good the headline looks.

Used properly, agreement becomes a promotion gate. A category earns autonomy by sustaining a high acceptance rate with low edit volume across enough tickets to mean something, backed by rubric scores that agree. That is a decision with evidence behind it rather than a decision made because the demo went well.

Rubric scores are informed opinions held by your own team. Reopen rate is behavior — the requester telling you the answer did not work, unprompted, at their own inconvenience. It is the closest thing to ground truth available in support, and it should be the number that governs how much autonomy a category is allowed to hold.

Split it by resolution path and compare AI-resolved against human-resolved within the same categories. A comparison across different category mixes is meaningless, because the automation handles the routine band and routine tickets reopen less for reasons that have nothing to do with the AI. Like for like is the only comparison that carries information.

Watch the direction of divergence as much as the level. A rising reopen rate on AI-resolved tickets in one category is a specific, findable problem — usually one article, one policy change, or one class of question the system should be declining. That specificity is the genuine advantage of automated resolution over a large human team: when quality slips, there is exactly one place to look, and it is the same place for every affected ticket.

Quality rarely decays smoothly. It steps down on a date, because something changed: an article was rewritten, a policy shifted, a system was renamed, a bulk import dropped a few hundred stale documents into the index. The single most valuable QA practice is being able to line a quality change up against a change log — which means keeping a change log for knowledge and configuration, not only for code.

Maintain a frozen regression set: fifty to a hundred past tickets with known-good answers, spanning your main categories, including the awkward ones. Re-run it after any significant knowledge or configuration change. It takes minutes, it catches obvious breakage before requesters do, and it turns the AI seems worse lately into a specific list of what broke and when.

When drift is confirmed, the response order matters: roll the affected category back to suggest mode, fix the source rather than the prompt, re-run the regression set, then restore autonomy. Rolling back is the step teams skip because it feels like an admission of failure. It is the opposite — per-category autonomy exists precisely so a quality problem in one area does not require switching the whole system off, and a team that can demote a category calmly ends up trusting the system far more than one that cannot.

Every practice above depends on two things being cheap: changing how much autonomy a category has, and reconstructing what happened on a ticket after the fact. Where both are expensive, QA degrades into reading complaints again.

FlowTux is built for that loop. Suggest, approve, and autonomous modes are set per category, so promoting and demoting a category is a setting rather than a deployment. Answers are grounded in your linked codebase and resolved ticket history, which makes the grounded criterion checkable instead of a judgment call — the reviewer can see what the answer was built from. And every triage step, source, approval, and resolution is written to the ticket timeline as a full audit trail, which is what makes sampling, drift investigation, and reopen analysis possible weeks after the ticket closed.