Changing an AI model, prompt template, temperature, tool definition, or retrieval configuration can alter behavior without changing your application code. A green unit-test suite therefore does not prove that the new configuration is safe. A practical release gate must compare the candidate with a known baseline on the same versioned cases and preserve enough evidence to explain every decision.
This article builds a small provider-neutral Python gate. It checks JSON structure and business rules, repeats cases to expose instability, measures latency, compares candidate metrics with a baseline, and returns a non-zero exit code when the agreed regression budget is exceeded. It does not call a specific commercial API and contains no credentials.
1. Define the decision before writing the runner
Start with an explicit release policy. For example:
- every response must be valid JSON and satisfy its case schema;
- deterministic business rules must pass on every run;
- candidate pass rate may not be lower than baseline pass rate;
- candidate p95 latency may be no more than 20% above baseline and no higher than an absolute ceiling;
- transport errors, timeouts, and rate limits are reported separately from quality failures;
- a human reviews a fixed sample of outputs for meaning, tone, and harmful behavior.
The last item matters. Schema validation can prove that a field exists; it cannot prove that advice is correct or that a summary preserves the important facts. Automated oracles and human evaluation answer different questions.
2. Version the cases and expected behavior
JSON Lines works well for a small evaluation corpus because each line is an independent JSON object. Keep prompts, rules, and case identifiers in version control.
{"case_id":"ticket-001","prompt":"Classify: Payment failed twice","schema":{"type":"object","required":["category","priority"],"properties":{"category":{"type":"string","enum":["billing","technical","other"]},"priority":{"type":"string","enum":["low","medium","high"]}},"additionalProperties":false},"contains":["billing"]}
{"case_id":"ticket-002","prompt":"Classify: How do I change my password?","schema":{"type":"object","required":["category","priority"],"properties":{"category":{"type":"string"},"priority":{"type":"string"}},"additionalProperties":false},"contains":["technical"]}A changed prompt or oracle changes the test asset. Record the Git commit and a stable hash so a later investigation can reproduce exactly what the gate evaluated.
The Python json documentation warns that untrusted JSON can consume significant CPU or memory. Limit both file size and line size before parsing.
import hashlib
import json
from pathlib import Path
from typing import Any
MAX_FILE_BYTES = 1_000_000
MAX_LINE_BYTES = 16_384
def load_cases(path: Path) -> list[dict[str, Any]]:
if path.stat().st_size > MAX_FILE_BYTES:
raise ValueError("case file is too large")
cases: list[dict[str, Any]] = []
with path.open(encoding="utf-8") as source:
for number, line in enumerate(source, start=1):
if not line.strip():
continue
if len(line.encode("utf-8")) > MAX_LINE_BYTES:
raise ValueError(f"line {number} is too large")
case = json.loads(line)
if not {"case_id", "prompt", "schema"} <= case.keys():
raise ValueError(f"line {number} misses required case fields")
cases.append(case)
return cases
def case_hash(case: dict[str, Any]) -> str:
canonical = json.dumps(case, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()3. Keep the provider behind an adapter
The gate should not know whether the model comes from a hosted API, a local server, or an n8n sub-workflow. Give the runner one callable contract: input text in, parsed dictionary out. The adapter owns authentication, timeout handling, response parsing, and provider-specific parameters.
Do not log authorization headers or raw prompts containing personal data. Read secrets from environment variables or the CI secret store, redact sensitive response fields, and retain only evidence needed for debugging.
Use time.perf_counter() for elapsed duration. Python documents it as a high-resolution performance counter suitable for measuring short durations.
import time
from collections.abc import Callable
ModelCall = Callable[[str], dict[str, Any]]
def timed_call(call: ModelCall, prompt: str) -> tuple[dict[str, Any], float]:
started = time.perf_counter()
result = call(prompt)
return result, time.perf_counter() - started4. Apply independent deterministic oracles
JSON Schema checks the response contract. The Python jsonschema package exposes versioned validators including Draft202012Validator and an iter_errors method. Validate the schema itself before using it; otherwise a broken test oracle can create misleading results.
import re
from jsonschema import Draft202012Validator
def deterministic_errors(output: dict[str, Any], case: dict[str, Any]) -> list[str]:
schema = case["schema"]
Draft202012Validator.check_schema(schema)
validator = Draft202012Validator(schema)
errors = [error.message for error in validator.iter_errors(output)]
rendered = json.dumps(output, ensure_ascii=False)
for fragment in case.get("contains", []):
if fragment.casefold() not in rendered.casefold():
errors.append(f"missing required fragment: {fragment}")
pattern = case.get("regex")
if pattern and re.search(pattern, rendered) is None:
errors.append("response does not match the required regex")
return errorsThese checks are independent because the candidate model does not decide whether it passed. Model-based scoring can be useful for style or semantic similarity, but it should be reported as a separate signal with its own calibration and review—not mixed invisibly into deterministic pass/fail logic.
5. Repeat cases and calculate comparable metrics
A single successful call hides variance. Run each case several times using the same configuration for baseline and candidate. Record every failure category rather than silently retrying until green. Limited retries may be appropriate for HTTP 429 or selected 5xx responses, but the original error and retry count remain part of the evidence.
For a small observed sample, an inclusive percentile keeps the calculated p95 between the sample minimum and maximum. Python's statistics.quantiles documentation distinguishes the inclusive and exclusive methods.
from statistics import quantiles
def evaluate(call: ModelCall, cases: list[dict[str, Any]], repeats: int = 5) -> dict:
latencies: list[float] = []
passed = 0
total = 0
evidence: list[dict[str, Any]] = []
for case in cases:
for run_number in range(1, repeats + 1):
total += 1
try:
output, elapsed = timed_call(call, case["prompt"])
problems = deterministic_errors(output, case)
latencies.append(elapsed)
passed += int(not problems)
evidence.append({
"case_id": case["case_id"],
"case_hash": case_hash(case),
"run": run_number,
"elapsed_s": round(elapsed, 4),
"problems": problems,
})
except Exception as exc:
evidence.append({
"case_id": case["case_id"],
"case_hash": case_hash(case),
"run": run_number,
"error_type": type(exc).__name__,
})
p95 = quantiles(latencies, n=100, method="inclusive")[94] if len(latencies) >= 2 else (latencies[0] if latencies else float("inf"))
return {"pass_rate": passed / total, "p95_s": p95, "evidence": evidence}For stochastic text, exact output hashes usually measure sameness rather than correctness. Prefer repeated oracle results and calibrated semantic or human review. If the provider exposes a seed, record it, but do not promise reproducibility that the provider does not guarantee.
6. Compare baseline and candidate with explicit budgets
The gate should compare like with like: same corpus, adapter contract, repeat count, timeout, and environment.
def release_allowed(baseline: dict, candidate: dict) -> tuple[bool, list[str]]:
reasons: list[str] = []
if candidate["pass_rate"] < baseline["pass_rate"]:
reasons.append("candidate pass rate is below baseline")
absolute_limit_s = 2.0
relative_limit_s = baseline["p95_s"] * 1.20
if candidate["p95_s"] > min(absolute_limit_s, relative_limit_s):
reasons.append("candidate p95 latency exceeds the regression budget")
return not reasons, reasonsChoose thresholds from product risk and real historical data. A payment-support classifier and an internal brainstorming tool should not share the same release policy.
Save a sanitized JSON report containing the model/configuration identifiers, case-set commit, hashes, repeat count, environment, metrics, failures, and decision. Do not store secrets or unnecessary customer content.
7. Reuse the cases with pytest
Pytest's @pytest.mark.parametrize runs the same test with several argument sets. Its documentation also notes that parameter values are passed as-is; mutating a dictionary can therefore affect later invocations. Load immutable test data at collection time and do not modify it.
import os
import pytest
CASES = load_cases(Path("tests/cases.jsonl"))
@pytest.fixture
def candidate_model():
return build_model_adapter(
endpoint=os.environ["CANDIDATE_ENDPOINT"],
token=os.environ["CANDIDATE_TOKEN"],
)
@pytest.mark.parametrize("case", CASES, ids=lambda case: case["case_id"])
def test_candidate_contract(case, candidate_model):
output, _ = timed_call(candidate_model, case["prompt"])
assert deterministic_errors(output, case) == []The full comparison runner can execute as a separate CI command. If release_allowed returns false, print concise reasons and exit with code 1. That gives GitHub Actions, Jenkins, Azure DevOps, or n8n a clear machine-readable gate.
8. Failure handling and evidence
Keep failures distinguishable:
- validation failure means the output violated a known contract;
- transport failure means the provider call did not complete;
- timeout and rate-limit events indicate availability or capacity risk;
- infrastructure errors mean the evaluation itself is unreliable;
- semantic review failure means structure passed but meaning did not.
Fail closed when evidence is incomplete for a high-risk release. For a low-risk experiment, the team may choose a warning, but that decision should be visible in policy rather than hidden in code.
9. Release checklist
- Confirm the case corpus, schemas, prompts, and thresholds are reviewed and versioned.
- Run baseline and candidate with the same configuration and environment.
- Verify that secrets are injected securely and logs are redacted.
- Review deterministic failures, transport errors, retries, pass-rate delta, and p95 latency.
- Inspect the required human-review sample for factual quality, safety, and user impact.
- Store the sanitized report with the commit and configuration identifiers.
- Promote the candidate only when every required gate passes.
10. Troubleshooting
If every case fails schema validation, first validate the schema and confirm that the adapter parsed the provider response into the expected object. If p95 rises sharply, separate provider throttling from application changes and compare in the same environment. If results vary at temperature zero, treat the variance as observed behavior; zero temperature is not a universal determinism guarantee. If CI cannot read a secret, verify the secret name and scope without printing its value. If JSONL parsing fails, validate one line at a time and report only the line number—not sensitive content.
Conclusion
A Python regression gate turns an AI-model change into a reviewable engineering decision. Versioned cases provide traceability, independent oracles check contracts, repeated runs expose instability, latency budgets protect user experience, and a non-zero exit code blocks unsafe promotion. The gate is strongest when its limits are explicit: deterministic automation proves only what its oracles measure, while semantic quality and business risk still require calibrated evaluation and human judgment.
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…