GitHub’s deprecation notice for the all-platform CodeQL bundle is a useful reminder: a security-scanning update is not merely a dependency bump. The CodeQL Action, CLI bundle, query packs, runner platform, build mode, and upload permissions form one system. If one boundary changes, a green workflow can still hide missing languages, changed findings, or incomplete evidence.

This article presents a practical QA approach for advanced CodeQL setups. GitHub-hosted default setup is largely managed by GitHub; teams using pinned tools, custom packs, self-hosted runners, or advanced workflows need the strongest migration evidence.

1. Separate the moving parts

Before changing the workflow, distinguish these components:

  1. CodeQL Action version controls the workflow integration. GitHub currently lists v4 as the latest supported major version and v3 as supported. Advanced setups should normally track a supported major version or deliberately maintain pinned revisions.
  2. CodeQL CLI and bundle version identifies the analyzer and the packaged tools used during a run. It is not the same thing as the Action version.
  3. Query packs contain queries and libraries for particular languages. Custom configurations can introduce dependencies that the default suite does not use.
  4. Runner platform includes the operating system, CPU architecture, installed build tools, network access, and cache state.
  5. Build mode determines how CodeQL creates the database. none, autobuild, and manual have different language support and different risks; the correct choice depends on the repository.
  6. SARIF upload publishes analysis results to code scanning. A successful analysis without the expected upload, category, or permissions is not a complete production result.

The all-platform bundle announcement concerns distribution. It does not prove that every repository must make the same workflow change. Start by documenting the configuration you actually use.

2. Inventory the current setup

Create a small migration record for every CodeQL workflow:

  1. Record every github/codeql-action/* reference and whether it uses a major tag, release tag, or commit SHA.
  2. Record any tools input or other explicit CodeQL version selection.
  3. Record the runner labels and architectures, including self-hosted runners.
  4. Record the analyzed languages and the build mode for each language.
  5. Record custom configuration files, query suites, model packs, and downloaded packs.
  6. Record workflow permissions, SARIF categories, artifact retention, and branch protection dependencies.
  7. Record network restrictions, proxies, mirrors, and caches involved in bundle or pack downloads.

This inventory prevents a common mistake: testing a simplified example while production depends on a custom query pack or a restricted runner.

3. Build a baseline that can be compared

Run the existing configuration on a fixed commit and retain enough evidence to explain later differences. Capture:

  1. Workflow conclusion and step conclusions.
  2. Action and CodeQL versions reported in logs.
  3. Languages and build modes actually initialized.
  4. Database creation and analysis completion.
  5. SARIF files or uploaded categories.
  6. Alert identities and affected locations, grouped by rule and severity.
  7. Execution time, download behavior, and relevant cache information.
  8. Logs and artifact hashes needed to reproduce the comparison.

Do not require identical alert counts as the only acceptance rule. A newer analyzer or query pack can legitimately add, remove, or reclassify findings. The correct question is whether each difference is understood and acceptable.

4. Run a controlled candidate

Use the same commit, configuration, test data, and runner class for the candidate whenever possible. Change one migration variable at a time. Start with a non-blocking branch or scheduled canary, then compare the evidence before changing the required production check.

A compact advanced-setup canary can retain SARIF without publishing it:

name: CodeQL canary
on:
  workflow_dispatch:
permissions:
  contents: read
jobs:
  analyze:
    continue-on-error: true
    runs-on: ubuntu-latest
    strategy:
      matrix:
        language: [javascript-typescript, python]
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/init@v4
        with:
          languages: ${{ matrix.language }}
      - uses: github/codeql-action/analyze@v4
        with:
          category: "/language:${{ matrix.language }}"
          output: sarif-results
          upload: never
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: codeql-sarif-${{ matrix.language }}
          path: sarif-results

This fragment is intentionally incomplete: compiled languages may require autobuild or explicit build commands, and a production workflow may need security-events: write. The canary should mirror the real repository instead of blindly copying a generic example.

5. Define acceptance criteria before promotion

A migration is ready only when its evidence satisfies criteria agreed by security, development, and QA. A practical gate includes:

  1. Every required language initializes and produces a database.
  2. The expected query suites and custom packs resolve successfully.
  3. SARIF is generated for every intended matrix entry.
  4. New, missing, or changed findings are triaged and explained.
  5. Runner architecture and download selection match the intended environment.
  6. Execution time and resource use stay inside the team’s documented budget.
  7. Required upload permissions and categories work in a controlled integration run.
  8. Logs and artifacts contain no secrets or unnecessary source data.
  9. The rollback target is known and still available.
  10. A reviewer approves the comparison, not merely the green status icon.

6. Exercise negative paths safely

Negative testing should use a disposable repository, isolated runner, or dry-run configuration. Do not deliberately break the production security workflow.

  1. Select an unsupported or unavailable tool version and verify that setup fails closed with useful diagnostics.
  2. Test a runner architecture for which the expected asset cannot be obtained, using a controlled network stub rather than modifying production URLs.
  3. Reference an incompatible custom pack and confirm the failure identifies pack resolution.
  4. Deny the download in an isolated network test and confirm the workflow neither reports success nor reuses an unverified artifact.
  5. Remove security-events: write from a test workflow and verify that an attempted upload fails while local SARIF evidence remains inspectable.
  6. Corrupt or omit an expected build step and confirm database creation or analysis fails visibly.
  7. Run two matrix entries with distinct categories and verify their results are not silently combined.

Exact error wording is not a stable contract. Assert the failure category, failed step, retained diagnostics, and absence of a false-success signal.

7. Where automation helps

Automation is valuable for repeatable collection and comparison:

Human review remains necessary. A new alert can be a valuable detection improvement; a missing alert can be a dangerous regression. Neither conclusion follows from a count alone.

8. Five reviewer questions

  1. Which CodeQL component changed: Action, CLI bundle, query pack, runner, build mode, or permissions?
  2. Was the baseline produced from the same commit and representative runner environment?
  3. Can every changed finding be traced to a rule, location, and candidate artifact?
  4. Were custom packs, self-hosted runners, and restricted-network paths included?
  5. Is rollback possible without restoring an already unsupported configuration?

9. Troubleshooting guide

10. Release checklist

  1. Inventory reviewed and stored with the change.
  2. Baseline and candidate use the same representative commit.
  3. Required languages, runners, modes, and custom packs tested.
  4. SARIF differences triaged by a qualified reviewer.
  5. Negative paths fail closed and preserve diagnostics.
  6. Permissions and upload categories verified.
  7. Runtime and resource budgets accepted.
  8. Secrets and sensitive source data excluded from evidence.
  9. Rollback target and owner documented.
  10. Production promotion followed by a minimal smoke check.

Conclusion

The safest CodeQL migration is evidence-led. Treat the bundle change as a change to a security control: identify boundaries, compare a fixed baseline with a controlled candidate, test failure paths, and retain artifacts that another reviewer can inspect. Automation collects the evidence; people decide whether changed security behavior is acceptable.

Sources