A green pipeline is useful evidence. It is not proof that an AI-enabled product is safe, useful, or ready for every situation in which people will use it.
That distinction matters because AI behavior depends on data, prompts, retrieved context, model versions, permissions, and the surrounding workflow. A deterministic API may return the same result for the same input. A model-backed feature can produce several plausible results, fail only for a particular user group, or become less reliable after its knowledge source changes.
Quality engineering gives teams a way to manage that broader system. It does not replace testing, and it is not a fashionable name for test automation. It connects requirements, design, testing, observability, security, operations, and product decisions so that risks are discovered and acted on throughout the lifecycle.
Testing, automation, and quality engineering
The three activities support one another, but they answer different questions.
- Testing investigates. A tester designs experiments, challenges assumptions, follows surprising behavior, and learns what threatens value.
- Test automation checks known expectations repeatedly. It is excellent for contracts, regression signals, data validation, permissions, latency thresholds, and other conditions that can be expressed clearly.
- Quality engineering coordinates the system of work. It helps the team decide which risks matter, which evidence is needed, who owns a decision, and how production feedback changes future work.
Automation therefore produces evidence; it does not interpret every consequence of that evidence. A dashboard can report a 92% task-success rate. A human still has to ask whether the dataset represents real users, whether the remaining 8% contains severe failures, and whether the metric measures the actual business requirement.
NIST's AI Risk Management Framework treats AI risk management as a lifecycle activity organized around Govern, Map, Measure, and Manage. Its resources emphasize test, evaluation, verification, and validation (TEVV), context, documentation, and continuing assessment. This supports a practical conclusion: evaluation is not one final gate owned by QA. It is a shared, recurring activity.
A five-layer operating model
A small team can use five layers without creating a heavy process.
- Requirements and risk mapping. Describe the intended users, allowed actions, prohibited outcomes, data boundaries, and consequences of failure. Link each important risk to an owner and an observable signal.
- Deterministic software checks. Test APIs, schemas, access rules, integrations, retries, timeouts, logging, and fallback behavior. These checks should be reproducible and suitable for CI.
- AI evaluation and uncertainty. Evaluate representative tasks, groundedness, retrieval quality, refusal behavior, robustness, and relevant fairness or safety properties. Record datasets, prompts, model versions, thresholds, and known limitations.
- Exploratory and socio-technical testing. Let testers, domain experts, security specialists, and representative users probe ambiguous scenarios. Investigate how people interpret outputs and how the AI behaves inside the real workflow.
- Production monitoring and feedback. Observe latency, errors, retrieval drift, permission denials, escalations, user corrections, and incidents. Feed what the team learns back into risk mapping and regression suites.
The layers are not a waterfall. A finding in production may change a requirement. An exploratory session may reveal a new deterministic check. A model update may require a fresh evaluation even when the application code did not change.
Example: an AI support assistant
Imagine a support assistant that retrieves internal knowledge and can prepare an escalation for a human agent. Unit tests pass, API contracts are valid, and the evaluation dataset shows good answer relevance. The release can still fail in important ways.
- Requirement ambiguity: “Resolve requests quickly” does not define when the assistant must stop and escalate.
- Retrieval drift: an indexing change makes outdated policies rank above current ones.
- Permission leakage: a correct-looking answer contains information the current user may not access.
- Unsafe escalation: the assistant creates or routes a ticket without the confirmation required by the business process.
- Latency and cost spikes: long retrieval chains work in a small test environment but fail under real traffic.
- User harm: fluent wording persuades a user to trust an answer outside the assistant's approved domain.
More automation helps with some of these risks. Permission matrices, retrieval regression sets, load tests, and audit-log checks should be automated. But automation alone cannot decide whether “quickly” reflects the stakeholder's intent, whether a refusal is appropriate in context, or whether a plausible answer creates unacceptable harm.
That is where investigative testing and domain judgment remain essential.
Responsibility without title debates
The exact job titles are less important than explicit ownership.
- Product owns intended value, acceptance criteria, and risk priority.
- Engineering owns implementation quality, deterministic controls, and diagnosable failures.
- AI or data specialists own model and dataset documentation, evaluation design, and uncertainty reporting.
- QA owns investigation strategy, testability risks, evidence gaps, and independent challenge.
- Security owns threat analysis, abuse cases, access boundaries, and secret handling.
- Operations owns monitoring, incident signals, rollback readiness, and production feedback.
- Domain and user representatives help judge whether behavior is useful, safe, and understandable.
“Quality is everyone's responsibility” becomes meaningful only when a named person owns each decision and evidence item.
Release evidence checklist
Before release, ask for evidence that matches the product's actual risk. A practical checklist is:
- Important requirements and prohibited outcomes are written in testable language.
- High-impact risks have owners, controls, and observable failure signals.
- Deterministic checks cover interfaces, permissions, error handling, and fallback paths.
- AI evaluations record the dataset, prompts, model version, metrics, thresholds, and limitations.
- Someone independent of the implementation has challenged the most important assumptions.
- Exploratory testing covers ambiguous, adversarial, and realistic end-to-end scenarios.
- Production monitoring can detect regressions that pre-release tests may miss.
- The team has rollback, disablement, and incident-response criteria.
The checklist is not a universal certification. Evidence should be proportional to context. A low-impact writing aid and an assistant that can expose confidential information need different depth, reviewers, and thresholds.
Common anti-patterns
Watch for these warning signs:
- Treating green CI as a release argument without discussing uncovered risks.
- Using one accuracy score as a complete measure of trustworthiness.
- Allowing the development dataset to double as the only evaluation dataset.
- Testing the model while ignoring retrieval, permissions, user interface, and downstream actions.
- Running evaluation once and never after model, prompt, data, or policy changes.
- Collecting production metrics without an owner or response threshold.
- Calling every automated check “testing” and leaving no time for investigation.
These anti-patterns create activity without reliable learning.
A one-week adoption plan
A team can begin without reorganizing the company.
- On day one, choose one AI-enabled feature and write its intended use, prohibited use, and top five risks.
- On day two, map existing automated checks and identify which risks have no evidence.
- On day three, add one deterministic check for the most important uncovered technical risk.
- On day four, create a small versioned evaluation set and record the model and prompt configuration.
- On day five, run a focused exploratory session with QA and a domain expert.
- On day six, verify monitoring, rollback, and incident ownership in a safe environment.
- On day seven, review the evidence, unresolved risks, and next actions with product and engineering.
The goal is not a perfect framework in seven days. It is a visible feedback loop in which evidence changes decisions.
Conclusion
AI does not make disciplined testing obsolete. It makes the boundary between repeatable checking and human investigation more important.
Automate stable expectations. Evaluate model behavior with documented context. Explore what the team does not yet understand. Monitor the real system. Then use quality engineering to connect those signals to accountable decisions.
That is how a team moves from “the pipeline is green” to a more defensible statement: “we understand the important risks, we have relevant evidence, and we know what we will do when the evidence changes.”
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…