A Model Context Protocol server can pass every source-level test and still fail after release. The package may omit a runtime file, the container entry point may be wrong, the executable may target a different platform, or a wrapper may write diagnostic text to standard output and corrupt the protocol stream. Those defects are invisible when CI tests only source code with mocks.
The release candidate is therefore not the repository. It is the exact npm tarball, container digest, binary, or archive that users will install. A reliable release gate builds that artifact once, identifies it, and runs black-box tests against the same bytes that will be published.
Why source tests are not enough
Unit and integration tests remain essential. They provide fast feedback and localize failures. Artifact tests cover a different risk boundary: everything that happens between a successful source build and a working installation.
Typical artifact-only failures include:
- packaging rules that exclude configuration, schemas, licenses, or compiled files;
- generated files that are stale or absent;
- entry points that differ from the command used in development;
- undeclared runtime dependencies that were available only on the build agent;
- platform or runtime incompatibilities;
- transport wrappers that mix logs with protocol messages;
- publication of a different file from the one CI validated.
This is a form of differential testing. The source workspace and the published artifact should expose the same documented behavior. Any unexplained difference is a release defect.
A practical artifact release gate
The safest flow is linear and evidence driven.
- Build once and name the candidate. Produce the release artifact in CI and calculate a cryptographic digest such as SHA-256. Do not rebuild it in a later publish job.
- Record provenance when available. An attestation can link an artifact to a repository, workflow, and build. An SBOM can describe included components. Neither proves that the software is secure or correct; both improve traceability.
- Inspect the package before execution. List its contents and compare them with an allowlist of required and forbidden paths. For npm packages,
npm pack --dry-runcan show what would be included, while testing should install the resulting tarball rather than the source directory. - Install in a disposable environment. Use a short-lived container, VM, or sandbox that does not contain the developer checkout. Disable unnecessary network access and mount no personal credentials.
- Start the documented entry point. Exercise the same command and transport that users receive. For stdio, protocol messages must use standard input and standard output, while logs belong on standard error. For Streamable HTTP, test the documented endpoint and authentication behavior.
- Negotiate the supported protocol revision. The test client must follow the lifecycle for the revision the server declares. Older revisions use the initialization flow; newer revisions may use discovery and per-request metadata. The gate should not assume that every server supports the newest revision.
- Verify advertised behavior. Compare advertised tools, resources, prompts, and other capabilities with the release contract. Do not expect names or capabilities that the product never promised.
- Call one deterministic, harmless operation. Choose an operation from the server's own documented interface that has no external side effect. Validate the result schema and values. MCP itself does not guarantee a universal
pingtool, so the fixture must belong to the product or its test configuration. - Exercise failure paths. Send malformed input, an unknown method, invalid arguments, and an unauthorized HTTP request where authentication applies. Test timeouts and cancellation only when the negotiated revision, transport, or product contract defines the relevant mechanism.
- Shut down and retain evidence. Terminate the client and server according to the chosen transport, check for orphaned processes or ports, and store the digest, logs, test result, and attestation reference with the release.
The promotion job should receive the already tested artifact by immutable identifier. If it downloads by a mutable tag such as latest, the chain of evidence is broken.
Minimal provider-neutral test harness
A test does not need to know the server's internal modules. It needs a transport adapter and a product-specific expectation file. The following pseudocode deliberately avoids naming a fictional client library:
artifact = download_by_digest(RELEASE_DIGEST)
assert sha256(artifact) == RELEASE_DIGEST
sandbox = create_disposable_environment(network="restricted")
installation = sandbox.install(artifact)
server = installation.start(documented_command, transport=DOCUMENTED_TRANSPORT)
client = connect_with_supported_mcp_sdk(server)
session = client.negotiate_revision(expected_revision_range)
advertised = session.list_capabilities()
assert advertised satisfies RELEASE_CONTRACT
result = session.call_tool(DETERMINISTIC_FIXTURE_TOOL, FIXTURE_INPUT)
assert validate_schema(result, FIXTURE_RESULT_SCHEMA)
assert result == FIXTURE_EXPECTED_RESULT
assert unknown_method_returns_protocol_error(session)
assert invalid_arguments_return_structured_error(session)
assert protocol_stdout_contains_no_log_lines(server)
client.close()
server.stop()
assert sandbox.has_no_orphan_processes()
archive_evidence(RELEASE_DIGEST, session.logs, test_results)The concrete method names must come from the official SDK and the protocol revision used by the product. The test should fail clearly when the server negotiates an unsupported revision instead of guessing a lifecycle.
Negative cases worth keeping in CI
A compact release suite should include failures that distinguish packaging defects from application defects.
- Clean-install failure: the artifact cannot install without the repository or a global dependency.
- Missing-file failure: the package lacks a schema, template, executable, or generated asset required at runtime.
- Entrypoint failure: the documented command exits early or resolves the wrong path.
- Protocol pollution: a stdio server writes a banner or log line to standard output.
- Contract drift: the artifact exposes a different tool name, input schema, or result shape from the approved contract.
- Authentication boundary failure: a remote endpoint accepts a protected request without valid credentials.
- Secret leakage: tokens or environment values appear in logs, errors, protocol payloads, or archived evidence.
- Shutdown failure: the process, child process, temporary file, or listening port remains after the test.
Each case should state the expected observable result. Avoid inventing universal error strings: MCP servers and SDKs can represent product errors differently within the protocol rules.
CI design without rebuilding
A robust pipeline separates production of the artifact from validation and promotion:
jobs:
build:
outputs:
digest: ${{ steps.digest.outputs.sha256 }}
steps:
- run: ./build-release-artifact
- id: digest
run: ./record-sha256
- run: ./upload-immutable-artifact
validate:
needs: build
steps:
- run: ./download-by-digest "${{ needs.build.outputs.digest }}"
- run: ./inspect-package
- run: ./test-in-disposable-environment
- run: ./archive-validation-evidence
publish:
needs: [build, validate]
steps:
- run: ./promote-the-same-digest "${{ needs.build.outputs.digest }}"The command names are placeholders for the project's scripts, not copy-and-paste product commands. The important invariant is that publish promotes the digest tested by validate.
GitHub's artifact attestations can provide build provenance. Their availability depends on repository visibility and plan, so the pipeline must check the current GitHub requirements rather than silently making attestation a mandatory step. The absence of an optional attestation is not the same as a failed functional test; the organization's policy should decide whether it blocks release.
Release and rollback criteria
Block promotion when the digest differs, installation fails in a clean environment, the documented transport cannot start, negotiation fails for a claimed supported revision, the public contract differs unexpectedly, a deterministic operation returns the wrong result, authentication is bypassed, or secrets leak.
Rollback should point to a previously validated immutable artifact, not trigger a fresh rebuild of an old commit. The rollback record should contain the failed digest, the known-good digest, the reason, and the evidence location.
Where automation helps
Automation is especially valuable for repeatable package inspection, digest comparison, clean installation, protocol smoke tests, schema validation, secret scanning, and evidence retention. It also prevents a common human error: validating one artifact and publishing another.
Human review is still needed to approve the release contract, choose a truly harmless operation, interpret capability changes, and decide whether a provenance or policy exception is acceptable. A green harness proves only what its checks observed.
QA checklist
- [ ] The artifact was built once and referenced by immutable digest.
- [ ] Package contents match the allowlist and contain required runtime files.
- [ ] Installation succeeds without the source checkout or hidden global dependencies.
- [ ] The documented stdio or Streamable HTTP entry point starts successfully.
- [ ] The client negotiates a protocol revision actually supported by the server.
- [ ] Advertised behavior matches the approved product contract.
- [ ] A deterministic, side-effect-free product operation returns the expected result.
- [ ] Malformed and unauthorized requests fail safely.
- [ ] Stdio logs do not pollute the protocol stream.
- [ ] Logs and errors disclose no secrets.
- [ ] Shutdown leaves no orphaned processes, ports, or temporary secrets.
- [ ] The exact validated digest is the artifact promoted to the release channel.
Review questions
- Why can an MCP server pass source tests but fail after publication?
- Which identifier proves that validation and publication used the same artifact?
- Why should a test not assume a universal MCP
pingtool? - What must a stdio server keep out of standard output?
- Which checks require human judgment even when the release gate is automated?
Conclusion
Source tests show that the code can work in the development context. Artifact tests show that the deliverable users receive can be installed, negotiated, exercised, and stopped under realistic conditions. Build once, validate by immutable identity, publish the same bytes, and keep the evidence. That closes a release gap that mocks and repository-level tests cannot see.
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…