Practical guidance for QA teams that review agent‑authored pull requests
1. Why a Green Test Suite Is Not Enough
When an autonomous coding agent submits a pull request (PR) it can modify both the production code and the test suite that validates it. If the agent also rewrites the assertions, adds mocks, or removes edge‑case checks, the CI pipeline may report “all tests passed” while the underlying behavior is still incorrect.
The root cause is the oracle problem: a test oracle is the source of truth that tells us whether an observed outcome is correct. If the oracle is co‑generated with the code, its independence is compromised, and the evidence it provides is weak. The same danger exists for human‑written tests that are unintentionally weakened during a hurried fix.
A robust review process therefore needs a ladder of evidence—a sequence of increasingly independent oracles that together give confidence that the change does what the requirements demand and does not introduce regressions, security gaps, or performance problems.
2. The Evidence Ladder
| Level | Typical Oracle | Independence from the Change | What It Confirms |
|---|---|---|---|
| 1. Compilation & Type‑Checking | Compiler, language server, static type analysis | Strong only when toolchains and configuration are protected outside the change scope | Code is syntactically valid and respects type contracts under the approved configuration. |
| 2. Lint & Static Scans | Linter rules, SAST, secret‑detection, SBOM generators | Strong when rules and workflow definitions are protected and reviewed separately | Conformance to policy, detection of known security patterns, and visibility of new dependencies. |
| 3. Unit Tests (External) | Tests written before the PR or owned by a different team | Partially independent (assertions are fixed, but test harness may be affected) | Function‑level correctness for a defined set of inputs. |
| 4. Contract / Consumer‑Driven Tests | OpenAPI schemas, GraphQL contracts, protobuf definitions, consumer test suites | High independence (contracts are versioned and usually separate) | Compatibility with external callers and adherence to public interfaces. |
| 5. Integration Tests | End‑to‑end pipelines that spin up real services or containers | Independent of the PR’s internal unit tests | Interaction between components, database migrations, message‑bus flows. |
| 6. Property‑Based Tests | Tools like Hypothesis, QuickCheck that generate many inputs | Independent of specific examples; only the property definition is static | General invariants (e.g., “serialization round‑trip is lossless”) hold across a wide input space. |
| 7. Metamorphic Tests | Checks that relate multiple executions (e.g., “adding a record increases count by one”) | Independent because they compare behaviors rather than fixed outputs | Detects subtle regressions when the same code path is exercised under different conditions. |
| 8. Differential / Shadow Tests | Running the new version side‑by‑side with the previous baseline (canary, A/B) | Independent because the baseline is immutable | Highlights unexpected deviations in production‑like workloads. |
| 9. Production‑Like Validation | Staging deployment, canary release, telemetry analysis | Independent; the environment mirrors production and is not under PR control | Confirms performance, observability, and real‑world side effects. |
| 10. Human Domain Review | Manual exploratory testing, requirements traceability, security‑boundary inspection | Fully independent (human judgment) | Verifies that the change satisfies the original business intent, regulatory constraints, and risk appetite. |
The ladder is not linear; a high‑risk change may require skipping straight to level 8 or 9, while a low‑risk typo fix might stop at level 2. The key is to rank the oracles for each PR and ensure that at least one strong, independent oracle is present before merging.
3. Applying the Ladder in a Review Workflow
Below is a practical step‑by‑step flow that aligns with GitHub’s official AI‑generated code review guidance and the NIST Secure Software Development Framework (SSDF) verification recommendations.
- Detect AI‑Assisted Changes — Record when AI assistance was used through the team’s PR template, commit metadata, or review process. Route the change through the appropriate risk-based review path.
- Run Level 1–2 Checks Automatically — The CI pipeline should compile, lint, run static analysis, and generate an SBOM. Protect the workflow and rule configuration so the same PR cannot silently weaken the checks that judge it.
- Identify Modified Tests — List any test files added, deleted, or edited. Highlight reductions in assertion count, added mocks that replace real dependencies, or changes to fixture data. Treat any test modification as a potential oracle weakening and require extra scrutiny.
- Select Independent Oracles Based on Risk — Low risk (e.g., a pure refactor) – level 3 unit tests may be sufficient if they are unchanged. Medium risk (new feature, data‑model change) – add contract tests (level 4) and integration tests (level 5). High risk (security‑critical path, payment flow, authentication) – require property‑based tests (level 6), metamorphic checks (level 7), and a differential run in a staging environment (level 8).
- Verify Requirements and Side Effects — Trace the change back to the original issue or user story. Confirm that all functional requirements are covered by at least one independent oracle. Examine whether the PR deletes or weakens any existing tests that previously guarded a side effect (e.g., a cache‑invalidation routine).
- Inspect Dependency and Boundary Changes — Use the SBOM to see if new third‑party libraries were introduced. Evaluate whether the new dependency expands the attack surface. Confirm that the change does not cross a defined security boundary (e.g., a service that previously could not write to a secret store now does).
- Record Evidence Before Merge — Capture CI artifact hashes, test coverage reports, and security scan results. Store a short “evidence checklist” in the PR description linking each oracle to its outcome. This record supports the SSDF practices for verifying software and retaining review evidence; map it to the organization’s chosen SSDF implementation rather than treating one checklist as universal.
- Human Acceptance — A senior QA engineer or domain expert performs an exploratory session focused on the change’s business intent. The reviewer asks targeted questions (see Section 4) and signs off only when satisfied that the independent evidence is adequate.
4. Five Concrete Questions Every Tester Should Ask
- What is the original requirement, and does the PR address it directly? — Goal: Detect “solution drift” where the agent solves a tangential problem.
- Which tests were added, removed, or altered, and how do those changes affect the overall test coverage? — Goal: Spot intentional or accidental weakening of oracles.
- Are there any contracts (API schemas, protobuf definitions, consumer tests) that the change must still satisfy? — Goal: Ensure backward compatibility and external consumer safety.
- Does the PR introduce new dependencies or modify existing security boundaries? — Goal: Guard against supply‑chain risk and privilege escalation.
- What independent evidence (property‑based, metamorphic, differential) exists that the behavior is correct beyond the PR’s own test suite? — Goal: Force the reviewer to demand at least one strong oracle for high‑risk changes. Answering these questions forces the reviewer to look beyond the surface “green CI” and to verify that the change is truly safe.
5. Where Automation Helps
Automation is indispensable for handling the volume and speed of AI‑generated PRs, but it must be orchestrated rather than treated as a final verdict.
| Automation Asset | What It Does Well | Where Human Oversight Is Still Needed |
|---|---|---|
| Compilation & Type‑Checking | Guarantees syntactic correctness instantly. | None – this is a hard prerequisite. |
| Static Scanners (SAST, secret detection, SBOM) | Finds known vulnerability patterns, accidental credential leaks, and tracks third‑party components. | Interpretation of false positives, risk rating of new dependencies, and business‑impact assessment. |
| CI‑Driven Unit & Integration Tests | Executes a large number of assertions quickly and reproducibly. | Validation that the test suite itself is trustworthy (no hidden mocks, no deleted assertions). |
| Property‑Based Test Generators | Explores a huge input space without writing each case manually. | Defining meaningful properties that reflect real business rules. |
| Differential / Canary Deployments | Observes real‑world behavior under production‑like load. | Deciding when a deviation is acceptable, analyzing telemetry, and rolling back if needed. |
| Automated Evidence Collection | Stores hashes of artifacts, coverage metrics, and scan reports automatically. | Ensuring the evidence is linked to the right change and that the documentation is reviewed. |
The takeaway is to layer automation: start with deterministic, low‑cost checks (levels 1‑2), then add more expensive, higher‑confidence checks (levels 6‑9) as the risk profile demands. Human reviewers remain the gatekeepers for the final, independent oracles.
6. Practical Example: A Payment‑API Refactor
Scenario: An AI agent proposes to replace a custom validateCard function with a call to a newly added third‑party library card‑validator. The PR also adds a unit test that simply asserts the function returns true for a hard‑coded card number.
| Ladder Level | What the Review Team Did |
|---|---|
| 1 | CI compiled the code; type‑checker flagged no errors. |
| 2 | Linter warned about an unused import; SAST found no new vulnerabilities. SBOM listed card‑validator as a new dependency. |
| 3 | Existing unit tests for other payment flows remained unchanged. The new test was flagged because it only covered a single happy‑path input. |
| 4 | The OpenAPI contract still required a cardNumber field with a Luhn‑check. The new library claims to perform the same check, but the contract version was not updated. |
| 5 | Integration tests that simulate a full checkout flow were run; they passed because the mock payment gateway was still using the old validation logic. |
| 6 | A property‑based test was added that generates random card numbers and asserts that validateCard and card‑validator.validate produce identical results. This test failed for a subset of numbers, exposing a mismatch. |
| 7 | A metamorphic test compared the number of successful transactions before and after the change under identical load; a 2 % drop was observed. |
| 8 | A canary release with 5 % traffic used the new library; monitoring showed an increase in declined transactions. |
| 9 | A senior QA engineer manually processed a set of edge‑case cards (e.g., foreign formats) and confirmed the new library rejected some valid cards. |
| 10 | The team decided to reject the PR, request a fix to the contract, and add more exhaustive property‑based tests. |
The example illustrates how each ladder rung adds an independent perspective that eventually uncovered a critical regression that the green unit test suite alone would have missed.
7. Conclusion
AI‑generated code can accelerate delivery, but it also introduces a subtle oracle problem: the same agent may control both the implementation and the checks that validate it. By ranking test oracles on a ladder of independence—from compilation to human domain review—QA teams can systematically gather evidence that is difficult for an agent to tamper with.
The practical workflow outlined above aligns with GitHub’s AI‑code‑review recommendations and the NIST SSDF verification activities. It ensures that every change, especially high‑risk ones, is backed by at least one strong, independent oracle before it reaches production. Automation remains a powerful ally, but the final decision must rest on human judgment supported by a transparent evidence trail.
Sources
- GitHub — Review AI‑generated code
- NIST — Secure Software Development Framework (SSDF), SP 800-218
- DEV Community — Rank Oracles Before You Merge an Agent Patch
- Quokka Labs — Testing AI‑Generated Code: 2026 QA Checklist for Teams Shipping Faster
- ZeroToAI — Reviewing AI‑Generated Pull Requests (2026 Developer Guide)
- Infinity Technologies — QA & Validation AI Agent: Test Evidence

Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…