A Model Context Protocol server can pass every source-level test and still fail after release. The package may omit a runtime file, the container entry point may be wrong, the executable may target a different platform, or a wrapper may write diagnostic text to standard output and corrupt the protocol stream. Those defects are invisible when CI tests only source code with mocks.

The release candidate is therefore not the repository. It is the exact npm tarball, container digest, binary, or archive that users will install. A reliable release gate builds that artifact once, identifies it, and runs black-box tests against the same bytes that will be published.

Why source tests are not enough

Unit and integration tests remain essential. They provide fast feedback and localize failures. Artifact tests cover a different risk boundary: everything that happens between a successful source build and a working installation.

Typical artifact-only failures include:

This is a form of differential testing. The source workspace and the published artifact should expose the same documented behavior. Any unexplained difference is a release defect.

A practical artifact release gate

The safest flow is linear and evidence driven.

  1. Build once and name the candidate. Produce the release artifact in CI and calculate a cryptographic digest such as SHA-256. Do not rebuild it in a later publish job.
  2. Record provenance when available. An attestation can link an artifact to a repository, workflow, and build. An SBOM can describe included components. Neither proves that the software is secure or correct; both improve traceability.
  3. Inspect the package before execution. List its contents and compare them with an allowlist of required and forbidden paths. For npm packages, npm pack --dry-run can show what would be included, while testing should install the resulting tarball rather than the source directory.
  4. Install in a disposable environment. Use a short-lived container, VM, or sandbox that does not contain the developer checkout. Disable unnecessary network access and mount no personal credentials.
  5. Start the documented entry point. Exercise the same command and transport that users receive. For stdio, protocol messages must use standard input and standard output, while logs belong on standard error. For Streamable HTTP, test the documented endpoint and authentication behavior.
  6. Negotiate the supported protocol revision. The test client must follow the lifecycle for the revision the server declares. Older revisions use the initialization flow; newer revisions may use discovery and per-request metadata. The gate should not assume that every server supports the newest revision.
  7. Verify advertised behavior. Compare advertised tools, resources, prompts, and other capabilities with the release contract. Do not expect names or capabilities that the product never promised.
  8. Call one deterministic, harmless operation. Choose an operation from the server's own documented interface that has no external side effect. Validate the result schema and values. MCP itself does not guarantee a universal ping tool, so the fixture must belong to the product or its test configuration.
  9. Exercise failure paths. Send malformed input, an unknown method, invalid arguments, and an unauthorized HTTP request where authentication applies. Test timeouts and cancellation only when the negotiated revision, transport, or product contract defines the relevant mechanism.
  10. Shut down and retain evidence. Terminate the client and server according to the chosen transport, check for orphaned processes or ports, and store the digest, logs, test result, and attestation reference with the release.

The promotion job should receive the already tested artifact by immutable identifier. If it downloads by a mutable tag such as latest, the chain of evidence is broken.

Minimal provider-neutral test harness

A test does not need to know the server's internal modules. It needs a transport adapter and a product-specific expectation file. The following pseudocode deliberately avoids naming a fictional client library:

artifact = download_by_digest(RELEASE_DIGEST)
assert sha256(artifact) == RELEASE_DIGEST

sandbox = create_disposable_environment(network="restricted")
installation = sandbox.install(artifact)
server = installation.start(documented_command, transport=DOCUMENTED_TRANSPORT)

client = connect_with_supported_mcp_sdk(server)
session = client.negotiate_revision(expected_revision_range)

advertised = session.list_capabilities()
assert advertised satisfies RELEASE_CONTRACT

result = session.call_tool(DETERMINISTIC_FIXTURE_TOOL, FIXTURE_INPUT)
assert validate_schema(result, FIXTURE_RESULT_SCHEMA)
assert result == FIXTURE_EXPECTED_RESULT

assert unknown_method_returns_protocol_error(session)
assert invalid_arguments_return_structured_error(session)
assert protocol_stdout_contains_no_log_lines(server)

client.close()
server.stop()
assert sandbox.has_no_orphan_processes()
archive_evidence(RELEASE_DIGEST, session.logs, test_results)

The concrete method names must come from the official SDK and the protocol revision used by the product. The test should fail clearly when the server negotiates an unsupported revision instead of guessing a lifecycle.

Negative cases worth keeping in CI

A compact release suite should include failures that distinguish packaging defects from application defects.

Each case should state the expected observable result. Avoid inventing universal error strings: MCP servers and SDKs can represent product errors differently within the protocol rules.

CI design without rebuilding

A robust pipeline separates production of the artifact from validation and promotion:

jobs:
  build:
    outputs:
      digest: ${{ steps.digest.outputs.sha256 }}
    steps:
      - run: ./build-release-artifact
      - id: digest
        run: ./record-sha256
      - run: ./upload-immutable-artifact

  validate:
    needs: build
    steps:
      - run: ./download-by-digest "${{ needs.build.outputs.digest }}"
      - run: ./inspect-package
      - run: ./test-in-disposable-environment
      - run: ./archive-validation-evidence

  publish:
    needs: [build, validate]
    steps:
      - run: ./promote-the-same-digest "${{ needs.build.outputs.digest }}"

The command names are placeholders for the project's scripts, not copy-and-paste product commands. The important invariant is that publish promotes the digest tested by validate.

GitHub's artifact attestations can provide build provenance. Their availability depends on repository visibility and plan, so the pipeline must check the current GitHub requirements rather than silently making attestation a mandatory step. The absence of an optional attestation is not the same as a failed functional test; the organization's policy should decide whether it blocks release.

Release and rollback criteria

Block promotion when the digest differs, installation fails in a clean environment, the documented transport cannot start, negotiation fails for a claimed supported revision, the public contract differs unexpectedly, a deterministic operation returns the wrong result, authentication is bypassed, or secrets leak.

Rollback should point to a previously validated immutable artifact, not trigger a fresh rebuild of an old commit. The rollback record should contain the failed digest, the known-good digest, the reason, and the evidence location.

Where automation helps

Automation is especially valuable for repeatable package inspection, digest comparison, clean installation, protocol smoke tests, schema validation, secret scanning, and evidence retention. It also prevents a common human error: validating one artifact and publishing another.

Human review is still needed to approve the release contract, choose a truly harmless operation, interpret capability changes, and decide whether a provenance or policy exception is acceptable. A green harness proves only what its checks observed.

QA checklist

Review questions

  1. Why can an MCP server pass source tests but fail after publication?
  2. Which identifier proves that validation and publication used the same artifact?
  3. Why should a test not assume a universal MCP ping tool?
  4. What must a stdio server keep out of standard output?
  5. Which checks require human judgment even when the release gate is automated?

Conclusion

Source tests show that the code can work in the development context. Artifact tests show that the deliverable users receive can be installed, negotiated, exercised, and stopped under realistic conditions. Build once, validate by immutable identity, publish the same bytes, and keep the evidence. That closes a release gap that mocks and repository-level tests cannot see.

Sources