GitHub’s deprecation notice for the all-platform CodeQL bundle is a useful reminder: a security-scanning update is not merely a dependency bump. The CodeQL Action, CLI bundle, query packs, runner platform, build mode, and upload permissions form one system. If one boundary changes, a green workflow can still hide missing languages, changed findings, or incomplete evidence.
This article presents a practical QA approach for advanced CodeQL setups. GitHub-hosted default setup is largely managed by GitHub; teams using pinned tools, custom packs, self-hosted runners, or advanced workflows need the strongest migration evidence.
1. Separate the moving parts
Before changing the workflow, distinguish these components:
- CodeQL Action version controls the workflow integration. GitHub currently lists
v4as the latest supported major version andv3as supported. Advanced setups should normally track a supported major version or deliberately maintain pinned revisions. - CodeQL CLI and bundle version identifies the analyzer and the packaged tools used during a run. It is not the same thing as the Action version.
- Query packs contain queries and libraries for particular languages. Custom configurations can introduce dependencies that the default suite does not use.
- Runner platform includes the operating system, CPU architecture, installed build tools, network access, and cache state.
- Build mode determines how CodeQL creates the database.
none,autobuild, andmanualhave different language support and different risks; the correct choice depends on the repository. - SARIF upload publishes analysis results to code scanning. A successful analysis without the expected upload, category, or permissions is not a complete production result.
The all-platform bundle announcement concerns distribution. It does not prove that every repository must make the same workflow change. Start by documenting the configuration you actually use.
2. Inventory the current setup
Create a small migration record for every CodeQL workflow:
- Record every
github/codeql-action/*reference and whether it uses a major tag, release tag, or commit SHA. - Record any
toolsinput or other explicit CodeQL version selection. - Record the runner labels and architectures, including self-hosted runners.
- Record the analyzed languages and the build mode for each language.
- Record custom configuration files, query suites, model packs, and downloaded packs.
- Record workflow permissions, SARIF categories, artifact retention, and branch protection dependencies.
- Record network restrictions, proxies, mirrors, and caches involved in bundle or pack downloads.
This inventory prevents a common mistake: testing a simplified example while production depends on a custom query pack or a restricted runner.
3. Build a baseline that can be compared
Run the existing configuration on a fixed commit and retain enough evidence to explain later differences. Capture:
- Workflow conclusion and step conclusions.
- Action and CodeQL versions reported in logs.
- Languages and build modes actually initialized.
- Database creation and analysis completion.
- SARIF files or uploaded categories.
- Alert identities and affected locations, grouped by rule and severity.
- Execution time, download behavior, and relevant cache information.
- Logs and artifact hashes needed to reproduce the comparison.
Do not require identical alert counts as the only acceptance rule. A newer analyzer or query pack can legitimately add, remove, or reclassify findings. The correct question is whether each difference is understood and acceptable.
4. Run a controlled candidate
Use the same commit, configuration, test data, and runner class for the candidate whenever possible. Change one migration variable at a time. Start with a non-blocking branch or scheduled canary, then compare the evidence before changing the required production check.
A compact advanced-setup canary can retain SARIF without publishing it:
name: CodeQL canary
on:
workflow_dispatch:
permissions:
contents: read
jobs:
analyze:
continue-on-error: true
runs-on: ubuntu-latest
strategy:
matrix:
language: [javascript-typescript, python]
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v4
with:
languages: ${{ matrix.language }}
- uses: github/codeql-action/analyze@v4
with:
category: "/language:${{ matrix.language }}"
output: sarif-results
upload: never
- uses: actions/upload-artifact@v4
if: always()
with:
name: codeql-sarif-${{ matrix.language }}
path: sarif-resultsThis fragment is intentionally incomplete: compiled languages may require autobuild or explicit build commands, and a production workflow may need security-events: write. The canary should mirror the real repository instead of blindly copying a generic example.
5. Define acceptance criteria before promotion
A migration is ready only when its evidence satisfies criteria agreed by security, development, and QA. A practical gate includes:
- Every required language initializes and produces a database.
- The expected query suites and custom packs resolve successfully.
- SARIF is generated for every intended matrix entry.
- New, missing, or changed findings are triaged and explained.
- Runner architecture and download selection match the intended environment.
- Execution time and resource use stay inside the team’s documented budget.
- Required upload permissions and categories work in a controlled integration run.
- Logs and artifacts contain no secrets or unnecessary source data.
- The rollback target is known and still available.
- A reviewer approves the comparison, not merely the green status icon.
6. Exercise negative paths safely
Negative testing should use a disposable repository, isolated runner, or dry-run configuration. Do not deliberately break the production security workflow.
- Select an unsupported or unavailable tool version and verify that setup fails closed with useful diagnostics.
- Test a runner architecture for which the expected asset cannot be obtained, using a controlled network stub rather than modifying production URLs.
- Reference an incompatible custom pack and confirm the failure identifies pack resolution.
- Deny the download in an isolated network test and confirm the workflow neither reports success nor reuses an unverified artifact.
- Remove
security-events: writefrom a test workflow and verify that an attempted upload fails while local SARIF evidence remains inspectable. - Corrupt or omit an expected build step and confirm database creation or analysis fails visibly.
- Run two matrix entries with distinct categories and verify their results are not silently combined.
Exact error wording is not a stable contract. Assert the failure category, failed step, retained diagnostics, and absence of a false-success signal.
7. Where automation helps
Automation is valuable for repeatable collection and comparison:
- Parse workflow metadata and analyzer versions into a machine-readable manifest.
- Normalize SARIF and compare rules, fingerprints, locations, and severities.
- Check that every expected language and category produced an artifact.
- Enforce evidence retention and secret scanning on uploaded logs.
- Detect unexpected runtime or download growth.
- Open a review item when findings change.
Human review remains necessary. A new alert can be a valuable detection improvement; a missing alert can be a dangerous regression. Neither conclusion follows from a count alone.
8. Five reviewer questions
- Which CodeQL component changed: Action, CLI bundle, query pack, runner, build mode, or permissions?
- Was the baseline produced from the same commit and representative runner environment?
- Can every changed finding be traced to a rule, location, and candidate artifact?
- Were custom packs, self-hosted runners, and restricted-network paths included?
- Is rollback possible without restoring an already unsupported configuration?
9. Troubleshooting guide
- Bundle download fails: verify the runner platform, network policy, proxy, selected tools input, and the current release assets.
- A language disappears: check the language identifier, matrix expansion, build mode support, and database initialization logs.
- Custom queries fail: verify pack dependencies and test the exact configuration file used in production.
- SARIF exists but no alerts appear: inspect
uploadsettings, permissions, category values, repository code-scanning availability, and upload logs. - Alert counts change: compare SARIF records by rule and location; do not hide the difference by changing the baseline.
- The canary is slower: separate download, build, database, query, and upload time before assigning the cause.
10. Release checklist
- Inventory reviewed and stored with the change.
- Baseline and candidate use the same representative commit.
- Required languages, runners, modes, and custom packs tested.
- SARIF differences triaged by a qualified reviewer.
- Negative paths fail closed and preserve diagnostics.
- Permissions and upload categories verified.
- Runtime and resource budgets accepted.
- Secrets and sensitive source data excluded from evidence.
- Rollback target and owner documented.
- Production promotion followed by a minimal smoke check.
Conclusion
The safest CodeQL migration is evidence-led. Treat the bundle change as a change to a security control: identify boundaries, compare a fixed baseline with a controlled candidate, test failure paths, and retain artifacts that another reviewer can inspect. Automation collects the evidence; people decide whether changed security behavior is acceptable.
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…