Practical guidance for QA teams that review agent‑authored pull requests


1. Why a Green Test Suite Is Not Enough

When an autonomous coding agent submits a pull request (PR) it can modify both the production code and the test suite that validates it. If the agent also rewrites the assertions, adds mocks, or removes edge‑case checks, the CI pipeline may report “all tests passed” while the underlying behavior is still incorrect.

The root cause is the oracle problem: a test oracle is the source of truth that tells us whether an observed outcome is correct. If the oracle is co‑generated with the code, its independence is compromised, and the evidence it provides is weak. The same danger exists for human‑written tests that are unintentionally weakened during a hurried fix.

A robust review process therefore needs a ladder of evidence—a sequence of increasingly independent oracles that together give confidence that the change does what the requirements demand and does not introduce regressions, security gaps, or performance problems.


2. The Evidence Ladder

LevelTypical OracleIndependence from the ChangeWhat It Confirms
1. Compilation & Type‑CheckingCompiler, language server, static type analysisStrong only when toolchains and configuration are protected outside the change scopeCode is syntactically valid and respects type contracts under the approved configuration.
2. Lint & Static ScansLinter rules, SAST, secret‑detection, SBOM generatorsStrong when rules and workflow definitions are protected and reviewed separatelyConformance to policy, detection of known security patterns, and visibility of new dependencies.
3. Unit Tests (External)Tests written before the PR or owned by a different teamPartially independent (assertions are fixed, but test harness may be affected)Function‑level correctness for a defined set of inputs.
4. Contract / Consumer‑Driven TestsOpenAPI schemas, GraphQL contracts, protobuf definitions, consumer test suitesHigh independence (contracts are versioned and usually separate)Compatibility with external callers and adherence to public interfaces.
5. Integration TestsEnd‑to‑end pipelines that spin up real services or containersIndependent of the PR’s internal unit testsInteraction between components, database migrations, message‑bus flows.
6. Property‑Based TestsTools like Hypothesis, QuickCheck that generate many inputsIndependent of specific examples; only the property definition is staticGeneral invariants (e.g., “serialization round‑trip is lossless”) hold across a wide input space.
7. Metamorphic TestsChecks that relate multiple executions (e.g., “adding a record increases count by one”)Independent because they compare behaviors rather than fixed outputsDetects subtle regressions when the same code path is exercised under different conditions.
8. Differential / Shadow TestsRunning the new version side‑by‑side with the previous baseline (canary, A/B)Independent because the baseline is immutableHighlights unexpected deviations in production‑like workloads.
9. Production‑Like ValidationStaging deployment, canary release, telemetry analysisIndependent; the environment mirrors production and is not under PR controlConfirms performance, observability, and real‑world side effects.
10. Human Domain ReviewManual exploratory testing, requirements traceability, security‑boundary inspectionFully independent (human judgment)Verifies that the change satisfies the original business intent, regulatory constraints, and risk appetite.

The ladder is not linear; a high‑risk change may require skipping straight to level 8 or 9, while a low‑risk typo fix might stop at level 2. The key is to rank the oracles for each PR and ensure that at least one strong, independent oracle is present before merging.


3. Applying the Ladder in a Review Workflow

Below is a practical step‑by‑step flow that aligns with GitHub’s official AI‑generated code review guidance and the NIST Secure Software Development Framework (SSDF) verification recommendations.

  1. Detect AI‑Assisted Changes — Record when AI assistance was used through the team’s PR template, commit metadata, or review process. Route the change through the appropriate risk-based review path.
  2. Run Level 1–2 Checks Automatically — The CI pipeline should compile, lint, run static analysis, and generate an SBOM. Protect the workflow and rule configuration so the same PR cannot silently weaken the checks that judge it.
  3. Identify Modified Tests — List any test files added, deleted, or edited. Highlight reductions in assertion count, added mocks that replace real dependencies, or changes to fixture data. Treat any test modification as a potential oracle weakening and require extra scrutiny.
  4. Select Independent Oracles Based on Risk — Low risk (e.g., a pure refactor) – level 3 unit tests may be sufficient if they are unchanged. Medium risk (new feature, data‑model change) – add contract tests (level 4) and integration tests (level 5). High risk (security‑critical path, payment flow, authentication) – require property‑based tests (level 6), metamorphic checks (level 7), and a differential run in a staging environment (level 8).
  5. Verify Requirements and Side Effects — Trace the change back to the original issue or user story. Confirm that all functional requirements are covered by at least one independent oracle. Examine whether the PR deletes or weakens any existing tests that previously guarded a side effect (e.g., a cache‑invalidation routine).
  6. Inspect Dependency and Boundary Changes — Use the SBOM to see if new third‑party libraries were introduced. Evaluate whether the new dependency expands the attack surface. Confirm that the change does not cross a defined security boundary (e.g., a service that previously could not write to a secret store now does).
  7. Record Evidence Before Merge — Capture CI artifact hashes, test coverage reports, and security scan results. Store a short “evidence checklist” in the PR description linking each oracle to its outcome. This record supports the SSDF practices for verifying software and retaining review evidence; map it to the organization’s chosen SSDF implementation rather than treating one checklist as universal.
  8. Human Acceptance — A senior QA engineer or domain expert performs an exploratory session focused on the change’s business intent. The reviewer asks targeted questions (see Section 4) and signs off only when satisfied that the independent evidence is adequate.

4. Five Concrete Questions Every Tester Should Ask

  1. What is the original requirement, and does the PR address it directly? — Goal: Detect “solution drift” where the agent solves a tangential problem.
  2. Which tests were added, removed, or altered, and how do those changes affect the overall test coverage? — Goal: Spot intentional or accidental weakening of oracles.
  3. Are there any contracts (API schemas, protobuf definitions, consumer tests) that the change must still satisfy? — Goal: Ensure backward compatibility and external consumer safety.
  4. Does the PR introduce new dependencies or modify existing security boundaries? — Goal: Guard against supply‑chain risk and privilege escalation.
  5. What independent evidence (property‑based, metamorphic, differential) exists that the behavior is correct beyond the PR’s own test suite? — Goal: Force the reviewer to demand at least one strong oracle for high‑risk changes. Answering these questions forces the reviewer to look beyond the surface “green CI” and to verify that the change is truly safe.

5. Where Automation Helps

Automation is indispensable for handling the volume and speed of AI‑generated PRs, but it must be orchestrated rather than treated as a final verdict.

Automation AssetWhat It Does WellWhere Human Oversight Is Still Needed
Compilation & Type‑CheckingGuarantees syntactic correctness instantly.None – this is a hard prerequisite.
Static Scanners (SAST, secret detection, SBOM)Finds known vulnerability patterns, accidental credential leaks, and tracks third‑party components.Interpretation of false positives, risk rating of new dependencies, and business‑impact assessment.
CI‑Driven Unit & Integration TestsExecutes a large number of assertions quickly and reproducibly.Validation that the test suite itself is trustworthy (no hidden mocks, no deleted assertions).
Property‑Based Test GeneratorsExplores a huge input space without writing each case manually.Defining meaningful properties that reflect real business rules.
Differential / Canary DeploymentsObserves real‑world behavior under production‑like load.Deciding when a deviation is acceptable, analyzing telemetry, and rolling back if needed.
Automated Evidence CollectionStores hashes of artifacts, coverage metrics, and scan reports automatically.Ensuring the evidence is linked to the right change and that the documentation is reviewed.

The takeaway is to layer automation: start with deterministic, low‑cost checks (levels 1‑2), then add more expensive, higher‑confidence checks (levels 6‑9) as the risk profile demands. Human reviewers remain the gatekeepers for the final, independent oracles.


6. Practical Example: A Payment‑API Refactor

Scenario: An AI agent proposes to replace a custom validateCard function with a call to a newly added third‑party library card‑validator. The PR also adds a unit test that simply asserts the function returns true for a hard‑coded card number.

Ladder LevelWhat the Review Team Did
1CI compiled the code; type‑checker flagged no errors.
2Linter warned about an unused import; SAST found no new vulnerabilities. SBOM listed card‑validator as a new dependency.
3Existing unit tests for other payment flows remained unchanged. The new test was flagged because it only covered a single happy‑path input.
4The OpenAPI contract still required a cardNumber field with a Luhn‑check. The new library claims to perform the same check, but the contract version was not updated.
5Integration tests that simulate a full checkout flow were run; they passed because the mock payment gateway was still using the old validation logic.
6A property‑based test was added that generates random card numbers and asserts that validateCard and card‑validator.validate produce identical results. This test failed for a subset of numbers, exposing a mismatch.
7A metamorphic test compared the number of successful transactions before and after the change under identical load; a 2 % drop was observed.
8A canary release with 5 % traffic used the new library; monitoring showed an increase in declined transactions.
9A senior QA engineer manually processed a set of edge‑case cards (e.g., foreign formats) and confirmed the new library rejected some valid cards.
10The team decided to reject the PR, request a fix to the contract, and add more exhaustive property‑based tests.

The example illustrates how each ladder rung adds an independent perspective that eventually uncovered a critical regression that the green unit test suite alone would have missed.


7. Conclusion

AI‑generated code can accelerate delivery, but it also introduces a subtle oracle problem: the same agent may control both the implementation and the checks that validate it. By ranking test oracles on a ladder of independence—from compilation to human domain review—QA teams can systematically gather evidence that is difficult for an agent to tamper with.

The practical workflow outlined above aligns with GitHub’s AI‑code‑review recommendations and the NIST SSDF verification activities. It ensures that every change, especially high‑risk ones, is backed by at least one strong, independent oracle before it reaches production. Automation remains a powerful ally, but the final decision must rest on human judgment supported by a transparent evidence trail.


Sources