Beyond Vibes: A Structured QA Framework for AI Chat Agent Optimisation

An AI chat agent goes live. The first week looks promising — users are getting answers, support tickets are dropping, and leadership is satisfied. By week four, a pattern emerges: some responses are wrong, some are right but unhelpful, and a few are confidently citing sources that don't exist. Nobody knows how many.

This is the default state of AI agent quality assurance in most organisations. The agent is evaluated by anecdote — someone noticed a bad answer, someone else said it was great, and the team operates somewhere between those two data points. There is no systematic measurement, no repeatability, and no way to tell whether the agent is getting better or worse over time.

If you can't measure response quality across multiple dimensions, you can't improve it. And if you can't reproduce the measurement, you can't trust it.

This article presents a five-stage framework for building a structured, repeatable QA pipeline for AI chat agents. The framework transforms real customer conversations into a living test suite, automates multi-dimensional evaluation, and creates a feedback loop that improves both the agent and the questions people ask it.

The Pipeline

The framework connects five stages into a continuous loop. Each stage feeds the next, and the output of the final stage feeds back into the first.

The output of Stage 5 — variance analysis and quality improvements — feeds back into Stage 2 as updated topic instructions and into Stage 1 as new edge-case queries. The pipeline is designed to run repeatedly, with each pass producing a measurably different (and ideally better) result.

Stage 1: Historical Query Harvesting

The foundation of the framework is real customer queries — not synthetic test cases, not hypothetical scenarios, but the actual questions that users have already asked through support channels, forums, internal messaging platforms, and ticketing systems.

Real queries are superior to synthetic ones for three reasons:

• Natural language complexity. Real users use abbreviations, skip context, ask multi-part questions, and phrase things in ways that no test-case author would predict. "Hey whats the deal with the thing from last week that broke prod" is a real query. A synthetic test would never generate it.
• Authentic intent distribution. The frequency of topics in real queries reflects actual user needs, not what the QA team guesses users might ask about. If 40% of queries are about billing and 2% are about API rate limits, the test suite should reflect that ratio.
• Edge case coverage. The queries that break agents — ambiguous intent, compound questions, references to context the agent doesn't have — occur naturally in real data. They're expensive and unreliable to manufacture.

Stage 2: Topic Categorisation & Intent Mapping

Raw queries need structure before they can be evaluated. This stage classifies each query into a topic category and defines a Topic-Action-Mapping — a prescription for how each category should be handled.

A Topic-Action-Mapping answers three questions for each category:

1. Which sources should the agent consult? — A curated list of knowledge base articles, documentation pages, or reference materials relevant to the topic.
2. In what order? — The priority sequence for source consultation. Primary sources first, supplementary sources second.
3. What action should the agent take? — Answer directly, escalate to a human, request clarification, or redirect to a specific tool or workflow.

Topic Category	Primary Sources	Action
Billing & Invoicing	Pricing documentation, billing FAQ, plan comparison matrix	Answer directly with link to billing portal
API Integration	API reference docs, SDK guides, changelog	Answer with code examples; escalate if version-specific
Account Access	Auth troubleshooting guide, SSO documentation	Provide self-service steps; escalate if MFA-related
Feature Requests	Product roadmap (public), feature voting board	Acknowledge, log request, redirect to feedback channel
Outage / Status	Status page, incident history	Provide current status; never speculate on ETAs

The Topic-Action-Mapping serves a dual purpose. For the QA pipeline, it defines what a correct response looks like — the evaluation rubric. For the agent itself, it provides scoped instructions that narrow the inference window and reduce the chance of hallucination. This is where quality and cost optimisation converge, but more on that later.

Stage 3: Automated Evaluation Pipeline

With a categorised query set and topic-level instructions in place, the next step is automation. The goal is an orchestration workflow that can process the entire query set without manual prompting.

The pipeline works as follows:

1. Iterate over the full query set, one query at a time
2. Inject the appropriate topic-level instructions based on the query's category
3. Pass the query to the agent as a prompt
4. Capture the complete response, including citations, confidence scores (if available), and any recommended follow-up actions
5. Store the query-response pair alongside metadata (topic, timestamp, batch ID) for downstream analysis

The batch ID is critical. Every execution run gets a unique identifier so that results from different runs can be compared — which becomes essential in Stage 5.

Stage 4: Multi-Dimensional Scoring

Response quality is not a single number. An answer can be accurate but unhelpful, helpful but hallucinated, or well-cited but answering the wrong question. Each dimension needs its own metric.

Metric	What It Measures	How to Score
Numerical Accuracy	Are quantitative claims (prices, dates, limits, percentages) correct?	Binary per claim: correct or incorrect. Aggregate as % accuracy across all numerical claims in the response.
Helpfulness	Does the response resolve the user's underlying need?	3-point scale: fully resolves (3), partially resolves (2), does not resolve (1). Requires rubric aligned to Topic-Action-Mapping.
Intent Recognition	Did the agent correctly identify what was being asked?	Binary: correct intent or misidentified. For multi-part queries, score per sub-intent.
Hallucination Detection	Does the response contain fabricated information?	Binary per factual claim: supported by source or unsupported. Aggregate as hallucination rate.
Source Citation	Are references valid and properly attributed?	Check each cited URL/document: exists (binary), relevant to claim (binary), current (binary).
Consistency	Are responses stable across repeated runs?	Measured in Stage 5 via cross-batch comparison. Scored as semantic similarity coefficient.

The first five metrics are evaluated per response. Consistency is evaluated across responses to the same query in different batch runs, which is what makes Stage 5 necessary.

Stage 5: Batch Consistency Testing

This is the stage most teams skip, and it's arguably the most revealing.

LLM-powered agents are non-deterministic by default. The same input can produce different outputs across runs — different phrasing, different citations, different confidence levels, occasionally different factual claims. For a support agent, this is a problem. A customer who asks the same question twice and gets two different answers loses trust fast.

Batch consistency testing works by running the identical query set through multiple separate executions (minimum three batches) and comparing the outputs.

What to measure across batches

• Content variance — How much does the substance of the response change? Use semantic similarity scoring (e.g., cosine similarity of embeddings) rather than exact string matching, since phrasing will naturally vary.
• Citation drift — Do the same queries produce the same source references? Citation instability often signals that the agent's retrieval layer is inconsistent, not just its generation layer.
• Confidence stability — If the agent reports confidence scores, do they remain consistent? A query that scores 0.9 confidence in one batch and 0.4 in another is a red flag regardless of whether the content is correct.
• Structural consistency — Does the response format remain stable? A response that's a structured list in one run and a prose paragraph in another suggests the agent lacks clear formatting instructions.

The Cost Story

Structured QA isn't just about quality — it directly reduces inference costs.

Without topic-level instructions, an agent processes each query against the broadest possible context. It searches the entire knowledge base, considers every possible intent, and generates a response informed by everything it has access to. This is expensive: more tokens consumed, more API calls to retrieval systems, higher latency, and a wider surface area for hallucination.

The Topic-Action-Mapping from Stage 2 changes this dynamic:

• Scoped retrieval. Instead of searching the full knowledge base, the agent receives curated reference links for the query's topic category. This reduces the number of documents retrieved and the tokens consumed by context.
• Narrowed inference. Topic-level instructions constrain the agent's response space. "Answer this billing question using these three sources" is a cheaper prompt to process than "answer this question using everything you know."
• Reduced follow-ups. Better first responses mean fewer clarification rounds. Each follow-up is an additional API call. A response that resolves the query on the first pass is both better quality and cheaper.

The cost savings are not marginal. In deployments where topic-scoping is applied rigorously, token usage per query can drop by 30–50% while relevance scores simultaneously improve. Quality and cost are not trade-offs in this framework — they move in the same direction.

The Feedback Loop

The framework's real value is iterative. Each evaluation pass produces two outputs that feed back into the system:

1. QA Improvements for the Agent

Every batch evaluation generates a list of specific, actionable refinements:

• Topic categories where accuracy is low → improve the source documents or add new ones to the Topic-Action-Mapping
• Queries where intent is misidentified → add disambiguation instructions to the agent's topic-level prompt
• Citation drift across batches → tighten the retrieval scope or add explicit source priority ordering
• Low-confidence topics → flag as knowledge base gaps and add to the documentation backlog

2. Improved Prompt Recommendations for Users

Patterns in failed queries reveal how users can ask better questions. If the agent consistently fails on queries that lack a product version number, the system can generate a prompt template: "I'm using [product] version [X] and I need help with [issue]." These templates can be surfaced in the chat interface as suggested formats, reducing intent ambiguity before the agent ever processes the query.

The best QA framework doesn't just fix the agent. It fixes the conversation.

Results: What This Framework Reveals

When this methodology is applied to a production agent, several patterns consistently emerge:

Finding	Implication
Confidence levels vary widely by topic	Not all topics are equally well-covered in the knowledge base. Low-confidence topics become a prioritised backlog for documentation teams.
Moderate content variance across identical batches	The agent's instructions and retrieval scope need tightening. High variance = high ambiguity in the system.
Topic-Action-Mapping reduces irrelevant responses	Scoped instructions demonstrably improve citation accuracy and reduce hallucination rates compared to unscoped prompts.
Numerical accuracy is the weakest dimension	LLMs are not calculators. Any response involving prices, dates, or limits needs explicit source grounding, not generative inference.
Multi-part queries have the highest failure rate	The agent often addresses the first part and ignores subsequent parts. Decomposing multi-part queries before processing improves resolution rates.

How to Apply This

This framework is not tied to any specific LLM provider, orchestration platform, or knowledge base technology. Any team running an AI chat agent in production can implement it. Here's the minimum viable version:

1. Export 100 real customer queries from the past 90 days. Pull from whatever channel users actually use — support tickets, messaging platforms, forums.
2. Categorise them into 5–10 topic groups. Don't overthink the taxonomy. "Billing", "Technical", "Account", "Feature Request", and "Other" is a perfectly fine starting point.
3. Write a Topic-Action-Mapping for each category: which 2–3 source documents should the agent consult, and what should it do (answer, escalate, clarify)?
4. Build a batch runner. A script that iterates over the query set, passes each query (with topic instructions) to the agent, and stores the response with metadata. This can be 50 lines of Python.
5. Score the first batch manually. Use the six metrics from Stage 4. This calibrates your rubric and establishes a baseline.
6. Run three batches with identical inputs. Compare responses across batches for variance.
7. Generate improvements. What topics scored lowest? Where did citations drift? What queries broke intent recognition? Those are the inputs for the next iteration.

What's Next

The five-stage framework is a foundation. Once it's running, several extensions become natural next steps:

• Automated threshold alerts. Set minimum acceptable scores for each metric. When a batch drops below threshold on any dimension, trigger an alert and pause the feedback loop until the root cause is resolved.
• Multi-turn conversation testing. The framework as described evaluates single-turn query-response pairs. Extending it to multi-turn conversations — where context accumulates across messages — requires tracking state and evaluating coherence over the full exchange, not just individual responses.
• Production readiness scoring. Combine the six quality metrics into a composite readiness score. Define what "production-ready" means quantitatively: e.g., >90% intent recognition, <5% hallucination rate, <15% cross-batch variance. An agent that doesn't meet the threshold doesn't ship.
• Regression testing on model updates. When the underlying LLM is updated or fine-tuned, run the full query set against the new version and compare scores to the previous baseline. This catches regressions before they reach users.

The goal is not a perfect agent. It's a measurably improving one. A QA framework that runs monthly, generates specific improvements, and tracks quality over time will outperform any amount of ad-hoc spot-checking — and it gives the team a shared language for what "good" actually means.

Stop evaluating AI agents by vibes. Build the pipeline. Measure the dimensions. Run the batches. Let the numbers tell you what's working and what isn't.