How a small change in the GitHub Actions API illustrates the need for tolerant contracts, robust observability, and a balanced mix of manual and automated testing.
Introduction
In continuous‑integration pipelines we often treat API responses as immutable contracts. A dashboard that shows “3 214 workflow runs” today is expected to show the same exact number tomorrow, and an AI‑driven alerting bot may use that figure to decide whether a deployment is safe.
GitHub’s recent change to the workflow‑run query—returning “2 500+” instead of an exact total when more than 2 500 items match a filter—breaks that assumption. The endpoint is still reachable, the schema is unchanged, yet the meaning of the total_count field has shifted from a precise integer to an approximate indicator.
For QA teams this is a reminder that contract testing must go beyond “the field exists and is a number.” It must also encode the behaviour of the contract under realistic load, pagination, and error‑handling scenarios. The following article walks through the practical implications of the GitHub change, shows where automation can help, and outlines a manual‑focused testing strategy that keeps the human judgment loop alive.
The GitHub Workflow‑Run Count Change – A Real‑World Contract Shift
GitHub announced that queries against the Workflow Runs API (and the corresponding UI) will still return paginated results of up to 1 000 items per page, but when a filter matches more than 2 500 runs the total_count field will be rendered as a rounded value such as “2 500+”. The rationale is two‑fold:
- Performance – Large result sets caused time‑outs and placed unnecessary load on the service.
- Accuracy – The previous exact count could be incomplete because the backend stopped counting after a timeout, misleading consumers.
The change is being rolled out to both GitHub.com and GitHub Enterprise Cloud. From a contract‑testing perspective the shape of the response (JSON object with total_count, workflow_runs, pagination links) has not changed, yet the semantic contract—“total_count is an exact integer”—has been altered.
Why Exact Counts Matter for Observability and AI
Many downstream tools rely on the total_count value:
| Consumer | Typical Use | What Breaks When the Count Becomes Approximate |
|---|---|---|
| Dashboard widgets | Show “X workflow runs in the last 24 h”. | Graphs suddenly display “2 500+” and stop updating correctly, causing false‑positive alerts. |
| CI gating scripts | Fail a build if more than N runs are pending. | A script that checks total_count > 2 500 now always passes, potentially allowing a flood of queued jobs. |
| AI‑driven incident triage | Feed the count into a model that predicts queue health. | The model treats “2 500+” as a concrete number, skewing predictions and leading to unnecessary escalations. |
| Cost‑tracking services | Multiply run count by a per‑run price. | Billing reports become inflated or under‑reported because the multiplier is applied to an imprecise figure. |
When an automation layer assumes the count is exact, the invariant it validates against no longer holds, and the failure propagates silently. Human operators may not notice until a downstream symptom (e.g., a sudden spike in alerts) surfaces, at which point the root cause is harder to trace.
Risks of Treating Approximate Metadata as an Exact Invariant
- Silent Contract Drift – The API contract appears unchanged in schema, so automated contract tests that only validate JSON structure will pass, even though the behavioural guarantee has shifted.
- False Confidence in Metrics – Monitoring dashboards that aggregate exact numbers will produce misleading trends, eroding trust in observability data.
- AI Model Degradation – Models trained on precise counts will receive noisy inputs, degrading prediction quality without an obvious error flag.
- Incorrect Business Logic – Rate‑limiting or quota enforcement that uses the count as a hard threshold may become ineffective, leading to resource exhaustion.
- Regression Blind Spots – Teams may ship new features that rely on the exact count, assuming the contract is stable, only to discover failures in production.
These risks highlight why contract testing must incorporate tolerance and why manual exploratory testing remains essential for spotting semantic mismatches that automated schema checks miss.
A QA Strategy for Tolerant Contracts
Below is a practical, test‑design‑oriented approach that blends manual exploration, consumer‑driven contracts, and selective automation.
1. Define a Tolerance Layer in the Contract
Instead of “total_count is an integer”, express the contract as:
total_count:
type: string | integer
description: |
Exact integer when ≤ 2 500, otherwise a string ending with '+' (e.g., "2 500+").
pattern: ^(\d+|\d{1,4}\+)$Why it helps: The contract now explicitly allows the “+” suffix, preventing false failures in schema validation while still flagging unexpected formats.
2. Test Pagination Limits
Manual test:
- Issue a request with a filter that returns just under 1 000 items. Verify that the
Linkheader contains anextURL. - Increase the filter to return just over 1 000 items. Confirm that the first page contains 1 000 items and the second page starts where the first left off.
Automation: Use a CI job that iterates over page numbers until the next link disappears, asserting that each page size never exceeds the documented maximum.
3. Boundary Tests Around the 2 500 Threshold
| Test | Expected total_count |
|---|---|
| 2 499 matching runs | Exact integer “2 499” |
| 2 500 matching runs | Exact integer “2 500” |
| 2 501 matching runs | String “2 500+” |
| 10 000 matching runs | String “2 500+” |
Exploratory angle: Create synthetic data (e.g., using a test repository with many workflow runs) and manually verify the UI displays the same representation as the API. Look for inconsistencies between UI and raw JSON.
4. Filter Narrowing Guidance
The GitHub guidance recommends adding date ranges or other narrowing criteria. QA should:
- Verify that adding a
createddate filter reduces the result set below the 2 500 limit and restores an exact count. - Document the minimal filter set that guarantees an exact count for a given repository size.
- Include these recommendations in the consumer documentation and in the contract’s “usage notes”.
5. Consumer‑Driven Contract (CDC) Workflow
- Consumer writes expectations – The team that consumes the Workflow Runs API records the tolerated formats (
integerorstring with '+'). - Provider validates – GitHub’s internal tests (or a mock provider for internal tooling) run a Pact‑style verification against the consumer contract.
- Broker publishes – Contracts are stored in a Pact Broker (or similar) so that any change to the provider triggers a CI failure for all registered consumers.
This CDC loop ensures that any change to the contract’s semantics is caught early, even if the OpenAPI schema remains unchanged.
6. Observability Guardrails
- Metric sanity checks – Add a Grafana/Prometheus rule that flags when a
total_countmetric switches from numeric to string. - Log enrichment – Include the raw
total_countvalue in structured logs, allowing downstream log‑analysis tools to detect the “+” suffix. - Feature flag for tolerant handling – Deploy a flag that toggles between strict numeric parsing and tolerant parsing, enabling a safe rollout.
7. Safe Rollout Practices
- Canary deployment – Enable the new tolerant parsing for a small percentage of consumers first.
- Shadow traffic – Run the old strict parser in parallel, compare results, and alert if they diverge beyond a configurable delta.
- Rollback plan – Keep the previous version of the contract (exact count) available for consumers that cannot tolerate the change, and communicate deprecation timelines clearly.
Where Automation Helps
Automation is not a silver bullet, but it excels at:
| Area | Automation Benefit | Manual Complement |
|---|---|---|
| Schema validation | Fast CI checks for JSON structure, data types, and regex patterns. | Exploratory verification of real‑world data variations. |
| Pagination crawling | Scripts can exhaust all pages in seconds, guaranteeing coverage. | Human inspection of edge‑case pages (e.g., empty last page). |
| Boundary generation | Parameterised tests can automatically hit 2 499, 2 500, 2 501, etc. | Manual creation of realistic test data that mirrors production load. |
| Observability alerts | Automated thresholds detect when a metric changes type. | Contextual analysis of why the change occurred (e.g., new filter added). |
| Contract publishing | CI pipelines can push updated CDC files to a broker on every merge. | Review of contract change rationale and impact on downstream teams. |
By letting automation handle the repetitive verification steps, testers can focus on interpretation, risk assessment, and designing realistic scenarios that the automation may never anticipate.
Five Concrete Questions a Tester Can Ask When Evaluating an AI Agent
- What assumptions does the agent make about input data?
E.g., does it treat “2 500+” as a concrete number or as an approximate range?
- How does the agent handle ambiguous or malformed responses?
Does it fall back to a safe default, raise an error, or silently continue?
- What observability signals does the agent expose?
Are there logs, metrics, or traces that reveal when a contract‑drift has been detected?
- Can the agent be configured to operate in a tolerant mode?
Is there a feature flag that switches between strict numeric parsing and flexible string handling?
- How does the agent surface failures to humans?
Does it generate actionable alerts, or does it hide the problem behind a generic “unknown error” message?
Answering these questions during a manual exploratory session uncovers hidden brittleness that automated unit tests would miss.
Conclusion
The GitHub workflow‑run count change is a textbook example of a semantic contract shift that can silently break dashboards, automation scripts, and AI‑driven decision engines. Relying solely on schema‑level contract tests leaves teams vulnerable to such drift.
A robust QA strategy blends tolerant contract definitions, boundary and pagination testing, consumer‑driven contract verification, and observability guardrails. Automation should be employed for the repetitive, deterministic checks, while manual and exploratory testing remains the safety net that validates meaning and risk in real‑world contexts.
By keeping the human judgment loop active—especially when AI agents are involved—teams can detect when “2 500+” stops being a useful number and adapt their pipelines before production incidents arise.
Sources
- GitHub Changelog – Changes to query results in the GitHub Actions API and UI. https://github.blog/changelog/2026-09-25-changes-to-query-results-in-the-github-actions-api-and-ui/
- GitHub REST API reference – Workflow runs. https://docs.github.com/en/rest/actions/workflow-runs
- TotalShiftLeft – API Contract Testing: What It Is & How It Works (2026). https://totalshiftleft.ai/blog/what-is-api-contract-testing
- Medium – Contract Testing for Event‑Driven Architectures (2026). https://medium.com/@chinthakadd/contract-testing-for-event-driven-architectures-can-we-do-we-and-how-do-we-177c8e9acfc7
- Pactflow – The Case for Contract Testing: Cutting Through API Integration Complexity (2026). https://pactflow.io/blog/ai-automation-part-1
- OneUptime – How to Handle Contract Testing Between Services (2026). https://oneuptime.com/blog/post/2026-01-24-contract-testing-services/view

Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…