Traditional QA assumes one thing Al agents refuse to give you: deterministic output. The
same input produces the same result, so you write a test, assert the expected value, and
move on. An LLM-based agent breaks that contract on purpose. Ask it the same question
twice and you may get two different — both acceptable — answers. Most teams respond in
one of two bad ways: they either skip testing entirely and ship on vibes, or they force exact-
match assertions that fail constantly and get deleted within a month.
After testing Al systems across ERP, CRM, and customer-support integrations, we've
settled on a framework that actually holds up. Here's how it works.
Start with a golden dataset, not test cases
Instead of asserting exact outputs, build a golden dataset: 50-200 real, representative inputs paired with reference answers written or approved by a domain expert. For a support agent, that's real (anonymized) customer queries with what a great human answer looks like. For a document-intelligence system, it's real documents with the correct extracted fields.
The dataset is the single most valuable QA asset in an Al project, and it's the one most teams never build. It forces the stakeholder conversation about what "correct" even means — which, in our experience, is where most Al projects quietly fail before a line of code is written.
Score with graders, not string matching
With a golden dataset in place, evaluation becomes a scoring problem. Three grader types cover most systems:
- Programmatic checks for anything objective: did the agent return valid JSON, did it include the required disclaimer, did the extracted invoice total match, did it stay under the length limit. These are cheap, fast, and should run on every build.
- Model-graded evaluation for subjective quality: use a second LLM as a judge, comparing the agent's answer against the reference answer on defined criteria (accuracy, completeness, tone). This scales to thousands of cases, but calibrate it first — run the judge over a sample a human has already scored and check they agree before trusting it.
- Human review for the highest-stakes slice: a rotating sample of production outputs reviewed weekly by someone who knows the domain. No Al system we've shipped runs without this loop, because model-graded evals drift and programmatic checks can't see everything.
Test the workflow, not just the model
Agents don't just answer — they act. They call tools, query databases, chain steps, and hand off to humans. Each of those seams is a failure point that model-quality testing never touches. Agentic workflow QA means testing the full path: what happens when the tool call times out, when the database returns zero rows, when the user sends an input in a language the system wasn't scoped for, when two steps disagree.
The most common production failure we see isn't a bad model answer — it's a workflow that has no defined behavior for an edge the happy-path demo never hit.
Regression-test prompts like code
Every prompt change is a code change. Teams version their application code religiously and then edit production prompts in a dashboard with no history and no tests. Put prompts in version control, and run the golden dataset against every prompt change before it ships. A one-line prompt tweak that improves one behavior routinely degrades three others; without regression evals you find out from customers.
Monitor in production, because evals aren't enough
Offline evals catch what you thought to test. Production monitoring catches the rest: track user-visible failure signals (thumbs-down rates, human-handoff rates, retry loops), sample real traffic into your review queue, and feed every confirmed failure back into the golden dataset so it can never regress silently again. The dataset should grow every month the system is live.
The takeaway
Testing Al agents is not impossible — it's just a different discipline. Golden datasets instead of exact assertions, graders instead of string matches, workflow testing instead of model- only testing, prompts under version control, and a production feedback loop. Teams that build this from day one ship faster, not slower, because they can change things without fear.
If you're building an agent and QA is the part nobody on the team owns, that's usually the highest-risk gap in the project — our QA and software testing team wires this discipline into Alimplementations from the first sprint.
