QA metrics that actually matter for AI-driven QA teams

The short answer: traditional QA metrics — pass rate, defect count, test coverage percentage — don't translate cleanly to AI systems, because AI output isn't deterministic. The metrics that actually matter measure something different: whether the system's judgment is trustworthy, not whether it matches one fixed expected output.

A 98% test pass rate sounds reassuring for traditional software. For an AI system, it can be meaningless if the 2% of failures are concentrated in your highest-value use case, or if the passing tests were graded with exact-match assertions too loose to catch real quality problems. Traditional QA metrics assume determinism — same input, same expected output, pass or fail. AI systems break that assumption by design, so the metrics have to measure something else: whether the system's behavior stays within an acceptable range of quality, not whether it hits one exact target.

If you're using model-graded evaluation (a second LLM scoring your system's outputs, as covered in our framework for testing AI agents), the grader itself needs a quality metric: how often does it agree with human judgment on a sampled set of cases? Below roughly 85-90% agreement, the grader isn't reliable enough to trust at scale, and every other metric built on top of it inherits that unreliability. This is the metric most AI QA teams skip, and it's the one that silently invalidates everything downstream when neglected.

For any system using retrieval, measure whether the correct source material was actually retrieved — separately from whether the final answer was good. A wrong answer with correct retrieval is a generation problem; a wrong answer with failed retrieval is a different problem entirely, and conflating the two wastes engineering time fixing the wrong layer. Track retrieval hit rate as its own number, not folded into overall answer quality.

Every time a prompt, model, or pipeline changes, run the full golden dataset and track what percentage of previously-passing cases now fail. This is the AI equivalent of a regression suite, and it's the single most important discipline for teams that iterate quickly — without it, a "small" prompt tweak that improves one behavior and quietly breaks three others goes undetected until a customer notices.

Track how often real production failures (caught via monitoring, covered in our guide to monitoring AI agents in production, if published) were something your offline evaluation would have caught. A growing gap between what your eval suite catches and what actually fails in production means your golden dataset has stopped reflecting real usage — a strong signal it needs fresh cases added, not that the system suddenly got worse.

For the slice of outputs reviewed by a human on a rotating basis, track what percentage a domain expert would rate as fully acceptable. This is slower and more expensive than automated metrics, which is exactly why it should be sampled rather than exhaustive — but it's the ground-truth check that catches subtle quality issues automated graders miss, and it should validate the automated metrics above periodically, not replace them.

Raw pass/fail percentage against loosely-defined criteria gives false confidence — a system can pass a badly-written test suite at 100% and still fail users constantly. Similarly, tracking only "did it crash" metrics (uptime, error rate) misses the entire category of AI-specific failure: confidently wrong, unhelpful, or subtly degraded output that returns a perfectly healthy HTTP response.

These metrics only work as a system, not individually — grader agreement validates your automated scoring, regression rate protects against silent breakage, retrieval accuracy isolates where RAG systems actually fail, and the production-to-eval gap keeps your test suite honest over time. This is the exact discipline our QA and software testing team builds into every AI implementation — treating AI quality as a measurable engineering practice, not a subjective judgment call made after the fact.