Deadlines, Backoff, and Idempotency
1. Why a dedicated polling component matters
AI‑enabled workflows that send a verification email (or a webhook callback) are inherently asynchronous. In a test environment we need to prove that the expected evidence – a “verified” flag, a receipt URL, or a signed token – appears within a reasonable time window, without leaking personal data into logs or flaky timing. A reusable polling service gives us:
- Deterministic deadlines – measured with time.monotonic() so the clock cannot jump backwards.
- Bounded retries – exponential backoff with jitter prevents hammering the provider while still guaranteeing a maximum wait time.
- Idempotent handling – the same message ID is ignored after the first successful validation, which eliminates duplicate processing in noisy inboxes.
- Testability – a fake clock and a fake provider let us verify every branch without real network calls.
The following sections walk through a minimal, dependency‑light implementation that satisfies these goals and can be dropped into a CI pipeline that uses Playwright, pytest, or any other test runner.
2. Data flow overview
Test / CI step
│
├─► ProviderAdapter (e.g. Gmail, SendGrid, mock HTTP)
│ – abstracts transport, returns raw payloads
│
├─► PollingService
│ – drives the deadline loop, backoff, deduplication
│
├─► Validator
│ – parses the payload, checks signatures, extracts DTO
│
└─► EvidenceResult
– success / typed error → Playwright or CI assertionThe provider adapter knows only how to fetch a message; the polling service knows when to stop; the validator knows what constitutes a valid verification.
Separating concerns makes the component reusable across SMTP, REST webhook, or in‑memory mock providers.
3. Core DTOs and typed error model
from dataclasses import dataclass
from typing import Literal, Union
@dataclass(frozen=True)
class VerificationDTO:
message_id: str # stable identifier from the provider
user_email: str
verified_at: float # epoch seconds, monotonic‑compatible
token: str # opaque, never logged
# Typed error outcomes – useful for CI reporting
PollResult = Union[
VerificationDTO,
Literal["timeout"],
Literal["malformed"],
Literal["duplicate"],
Literal["provider_error"]
]The DTO is immutable, making it safe to share between threads or async tasks. The error literals are explicit so a test can assert the exact failure mode.
4. Provider adapter contract
from abc import ABC, abstractmethod
from typing import Optional, Dict, Any
class ProviderAdapter(ABC):
"""Transport‑level abstraction. Implementations must be side‑effect free
for a given `since` timestamp and return at most one payload per call."""
@abstractmethod
def fetch_latest(self, since: float) -> Optional[Dict[str, Any]]:
"""Return the raw payload received after `since` (monotonic seconds)
or `None` if no new message is available."""A concrete Gmail adapter would translate the raw MIME message into a dictionary, but the polling logic never sees the HTTP or IMAP details.
5. Deterministic polling service
import os
import time
import random
import logging
from typing import Set
logger = logging.getLogger("polling")
logger.setLevel(logging.INFO)
class PollingService:
def __init__(
self,
provider: ProviderAdapter,
*,
max_attempts: int = 7,
base_delay: float = 0.5, # seconds
max_delay: float = 5.0,
jitter: float = 0.1, # fraction of delay
deadline_seconds: float = 30.0,
) -> None:
self.provider = provider
self.max_attempts = max_attempts
self.base_delay = base_delay
self.max_delay = max_delay
self.jitter = jitter
self.deadline_seconds = deadline_seconds
self._seen_ids: Set[str] = set()
def _next_delay(self, attempt: int) -> float:
"""Bounded exponential backoff with deterministic jitter."""
exp = min(self.base_delay * (2 ** attempt), self.max_delay)
# deterministic jitter – use a seeded RNG that tests can control
rng = random.Random(attempt) # seed = attempt number
jitter = rng.uniform(-self.jitter, self.jitter) * exp
return max(0.0, exp + jitter)
def poll(self, start_time: float) -> PollResult:
"""Run the deadline‑safe loop. Returns a DTO or a typed error."""
deadline = start_time + self.deadline_seconds
attempt = 0
while True:
now = time.monotonic()
if now >= deadline:
logger.info("polling timeout", extra={"deadline": deadline})
return "timeout"
raw = self.provider.fetch_latest(since=start_time)
if raw:
# Idempotency check – stable message_id must be unique
msg_id = raw.get("message_id")
if not msg_id:
logger.warning("malformed payload missing message_id")
return "malformed"
if msg_id in self._seen_ids:
logger.info("duplicate message ignored", extra={"msg_id": msg_id})
return "duplicate"
self._seen_ids.add(msg_id)
# Delegate validation
dto_or_error = Validator.validate(raw)
if isinstance(dto_or_error, VerificationDTO):
logger.info("verification received", extra={"msg_id": msg_id})
return dto_or_error
else:
logger.warning("payload failed validation")
return dto_or_error
# No payload yet – backoff before next attempt
delay = self._next_delay(attempt)
logger.debug("backoff", extra={"attempt": attempt, "delay": delay})
time.sleep(delay)
attempt += 1
if attempt >= self.max_attempts:
logger.info("max attempts reached without payload")
return "timeout"Key points
- Deadline safety –
deadline = start + deadline_secondsuses time.monotonic(); the loop never overshoots because the check happens before each sleep. - Deterministic jitter – the random generator is seeded with the attempt number, guaranteeing the same sequence in unit tests.
- Idempotency – a set of observed
message_ids prevents processing the same verification twice. - Structured logs –
extradicts contain only identifiers; the actual email body or token never appears in log output. - Secret handling – any credentials required by a concrete provider are read from environment variables (
os.getenv) inside that provider implementation, never inside the polling core.
6. Validation logic
class ValidationError(RuntimeError):
pass
class Validator:
@staticmethod
def validate(raw: dict) -> Union[VerificationDTO, str]:
"""
Expected keys: message_id, user_email, verified_at, token.
Returns a DTO on success or a typed error literal.
"""
required = {"message_id", "user_email", "verified_at", "token"}
if not required.issubset(raw):
return "malformed"
try:
verified_at = float(raw["verified_at"])
# Additional domain‑specific checks could go here,
# e.g. signature verification, email format, etc.
return VerificationDTO(
message_id=raw["message_id"],
user_email=raw["user_email"],
verified_at=verified_at,
token=raw["token"],
)
except (ValueError, TypeError):
return "malformed"The validator stays pure – no I/O, no network calls – which makes it trivial to unit‑test in isolation.
7. Example usage in a CI test
import pytest
def test_email_verification_flow():
# Arrange – fake provider that will emit a payload after 1.2 s
fake_provider = FakeProvider(
schedule=[
(1.2, {
"message_id": "msg-123",
"user_email": "alice@example.com",
"verified_at": str(time.monotonic() + 1.2),
"token": "opaque‑value"
})
]
)
poller = PollingService(fake_provider, deadline_seconds=5.0)
# Act
result = poller.poll(start_time=time.monotonic())
# Assert
assert isinstance(result, VerificationDTO)
assert result.message_id == "msg-123"
assert result.user_email == "alice@example.com"FakeProvider (shown later) respects the same fetch_latest contract but reads from an in‑memory schedule, enabling deterministic tests without real email traffic.
8. Unit‑test suite with a fake clock
class FakeClock:
"""A minimal monotonic clock that can be advanced manually."""
def __init__(self, start: float = 0.0):
self._now = start
def monotonic(self) -> float:
return self._now
def advance(self, seconds: float):
self._now += seconds
class FakeProvider(ProviderAdapter):
"""Emits pre‑programmed payloads based on the fake clock."""
def __init__(self, schedule: list[tuple[float, dict]]):
# schedule = [(delay_from_start, payload), ...]
self.schedule = schedule
self.start = None
def fetch_latest(self, since: float) -> Optional[dict]:
if self.start is None:
self.start = since
elapsed = time.monotonic() - self.start
for delay, payload in self.schedule:
if delay <= elapsed:
# Return once and then remove to simulate single delivery
self.schedule.remove((delay, payload))
return payload
return None
def test_timeout_behavior(monkeypatch):
fake_clock = FakeClock()
monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)
# Provider never yields a payload
provider = FakeProvider(schedule=[])
poller = PollingService(provider, deadline_seconds=2.0, max_attempts=3)
result = poller.poll(start_time=fake_clock.monotonic())
assert result == "timeout"
def test_duplicate_delivery(monkeypatch):
fake_clock = FakeClock()
monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)
payload = {
"message_id": "dup-001",
"user_email": "bob@example.com",
"verified_at": "0",
"token": "t"
}
provider = FakeProvider(schedule=[(0.1, payload), (0.5, payload)])
poller = PollingService(provider, deadline_seconds=5.0)
# First poll consumes the payload
first = poller.poll(start_time=fake_clock.monotonic())
assert isinstance(first, VerificationDTO)
# Advance clock – second poll sees the same message_id
fake_clock.advance(1.0)
second = poller.poll(start_time=fake_clock.monotonic())
assert second == "duplicate"
def test_malformed_payload(monkeypatch):
fake_clock = FakeClock()
monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)
malformed = {"user_email": "carl@example.com"} # missing required keys
provider = FakeProvider(schedule=[(0.2, malformed)])
poller = PollingService(provider, deadline_seconds=3.0)
result = poller.poll(start_time=fake_clock.monotonic())
assert result == "malformed"monkeypatch(pytest fixture) replacestime.monotonicwith the fake clock, guaranteeing deterministic timing.- The fake provider respects the same contract, allowing us to test timeout, duplicate, and malformed cases without external services.
9. QA checklist for the polling component
- Deadline safety: Confirm that
time.monotonic()is used for every elapsed-time calculation. - Backoff limits: Ensure
max_delayis never exceeded, including after jitter. - Deterministic jitter: Inject or seed the random source in unit tests and assert exact sleep intervals.
- Idempotency: Verify that a second payload with the same
message_idreturnsduplicate. - Error classification: Verify the expected
timeout,malformed,duplicate, andprovider_errorresults. - Secret isolation: Read provider credentials from environment configuration and never include them in logs.
- Log hygiene: Log only safe identifiers and timing data, never message bodies, tokens, or personal data.
- Transport separation: Keep deadlines, backoff, and deduplication in
PollingService; keep provider details in adapters. - Test coverage: Cover success, timeout, duplicate, malformed payload, and provider-error paths.
- CI integration: Assert the returned
VerificationDTOdirectly from Playwright or another end-to-end test.
10. Where automation helps
- Rapid regression detection – The deterministic backoff and fake clock let a CI pipeline run the full poll‑validate cycle in milliseconds, catching timing regressions before they reach production.
- Data‑minimal logging – Structured logs can be indexed for alerting without exposing user‑level content, satisfying privacy regulations while still providing traceability.
- Cross‑provider reuse – Because the polling logic is provider‑agnostic, the same test suite validates Gmail, SendGrid, or an internal webhook with only a thin adapter change.
- Scalable test matrix – By parametrizing
deadline_secondsandmax_attempts, teams can generate a matrix of “tight” vs. “lenient” SLA tests, automating risk analysis for different service‑level agreements.
11. Conclusion
A well‑engineered polling component turns an inherently flaky asynchronous verification step into a deterministic, testable building block. By anchoring all time calculations to time.monotonic(), applying bounded exponential backoff with a seeded jitter, and enforcing idempotency through stable message IDs, we obtain:
- Predictable CI runs – no hidden wall‑clock dependencies.
- Clear failure semantics – typed error results make debugging straightforward.
- Privacy‑first observability – logs contain only meta‑information.
Integrating this component into AI‑driven email verification workflows gives QA engineers confidence that the “email verified” signal is both real and reproducible, while keeping the test suite fast, reliable, and maintainable.
12. Review questions
- Why is time.monotonic() preferred over
time.time()for deadline calculations in a polling loop? - How does seeding the random number generator with the attempt count make jitter deterministic in tests?
- What mechanism prevents the polling service from processing the same verification message more than once?
- List two pieces of information that must never appear in structured logs according to the design.
- In the provided unit tests, how is the provider’s “no payload” scenario simulated without external services?
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…