Deadlines, Backoff, and Idempotency


1. Why a dedicated polling component matters

AI‑enabled workflows that send a verification email (or a webhook callback) are inherently asynchronous. In a test environment we need to prove that the expected evidence – a “verified” flag, a receipt URL, or a signed token – appears within a reasonable time window, without leaking personal data into logs or flaky timing. A reusable polling service gives us:

The following sections walk through a minimal, dependency‑light implementation that satisfies these goals and can be dropped into a CI pipeline that uses Playwright, pytest, or any other test runner.


2. Data flow overview

Test / CI step
   │
   ├─► ProviderAdapter (e.g. Gmail, SendGrid, mock HTTP)
   │        – abstracts transport, returns raw payloads
   │
   ├─► PollingService
   │        – drives the deadline loop, backoff, deduplication
   │
   ├─► Validator
   │        – parses the payload, checks signatures, extracts DTO
   │
   └─► EvidenceResult
            – success / typed error → Playwright or CI assertion

The provider adapter knows only how to fetch a message; the polling service knows when to stop; the validator knows what constitutes a valid verification.

Separating concerns makes the component reusable across SMTP, REST webhook, or in‑memory mock providers.


3. Core DTOs and typed error model

    from dataclasses import dataclass
    from typing import Literal, Union

    @dataclass(frozen=True)
    class VerificationDTO:
        message_id: str          # stable identifier from the provider
        user_email: str
        verified_at: float       # epoch seconds, monotonic‑compatible
        token: str               # opaque, never logged

    # Typed error outcomes – useful for CI reporting
    PollResult = Union[
        VerificationDTO,
        Literal["timeout"],
        Literal["malformed"],
        Literal["duplicate"],
        Literal["provider_error"]
    ]

The DTO is immutable, making it safe to share between threads or async tasks. The error literals are explicit so a test can assert the exact failure mode.


4. Provider adapter contract

    from abc import ABC, abstractmethod
    from typing import Optional, Dict, Any

    class ProviderAdapter(ABC):
        """Transport‑level abstraction. Implementations must be side‑effect free
        for a given `since` timestamp and return at most one payload per call."""

        @abstractmethod
        def fetch_latest(self, since: float) -> Optional[Dict[str, Any]]:
            """Return the raw payload received after `since` (monotonic seconds)
            or `None` if no new message is available."""

A concrete Gmail adapter would translate the raw MIME message into a dictionary, but the polling logic never sees the HTTP or IMAP details.


5. Deterministic polling service

    import os
    import time
    import random
    import logging
    from typing import Set

    logger = logging.getLogger("polling")
    logger.setLevel(logging.INFO)

    class PollingService:
        def __init__(
            self,
            provider: ProviderAdapter,
            *,
            max_attempts: int = 7,
            base_delay: float = 0.5,          # seconds
            max_delay: float = 5.0,
            jitter: float = 0.1,              # fraction of delay
            deadline_seconds: float = 30.0,
        ) -> None:
            self.provider = provider
            self.max_attempts = max_attempts
            self.base_delay = base_delay
            self.max_delay = max_delay
            self.jitter = jitter
            self.deadline_seconds = deadline_seconds
            self._seen_ids: Set[str] = set()

        def _next_delay(self, attempt: int) -> float:
            """Bounded exponential backoff with deterministic jitter."""
            exp = min(self.base_delay * (2 ** attempt), self.max_delay)
            # deterministic jitter – use a seeded RNG that tests can control
            rng = random.Random(attempt)          # seed = attempt number
            jitter = rng.uniform(-self.jitter, self.jitter) * exp
            return max(0.0, exp + jitter)

        def poll(self, start_time: float) -> PollResult:
            """Run the deadline‑safe loop. Returns a DTO or a typed error."""
            deadline = start_time + self.deadline_seconds
            attempt = 0

            while True:
                now = time.monotonic()
                if now >= deadline:
                    logger.info("polling timeout", extra={"deadline": deadline})
                    return "timeout"

                raw = self.provider.fetch_latest(since=start_time)
                if raw:
                    # Idempotency check – stable message_id must be unique
                    msg_id = raw.get("message_id")
                    if not msg_id:
                        logger.warning("malformed payload missing message_id")
                        return "malformed"
                    if msg_id in self._seen_ids:
                        logger.info("duplicate message ignored", extra={"msg_id": msg_id})
                        return "duplicate"
                    self._seen_ids.add(msg_id)

                    # Delegate validation
                    dto_or_error = Validator.validate(raw)
                    if isinstance(dto_or_error, VerificationDTO):
                        logger.info("verification received", extra={"msg_id": msg_id})
                        return dto_or_error
                    else:
                        logger.warning("payload failed validation")
                        return dto_or_error

                # No payload yet – backoff before next attempt
                delay = self._next_delay(attempt)
                logger.debug("backoff", extra={"attempt": attempt, "delay": delay})
                time.sleep(delay)
                attempt += 1
                if attempt >= self.max_attempts:
                    logger.info("max attempts reached without payload")
                    return "timeout"

Key points


6. Validation logic

    class ValidationError(RuntimeError):
        pass

    class Validator:
        @staticmethod
        def validate(raw: dict) -> Union[VerificationDTO, str]:
            """
            Expected keys: message_id, user_email, verified_at, token.
            Returns a DTO on success or a typed error literal.
            """
            required = {"message_id", "user_email", "verified_at", "token"}
            if not required.issubset(raw):
                return "malformed"

            try:
                verified_at = float(raw["verified_at"])
                # Additional domain‑specific checks could go here,
                # e.g. signature verification, email format, etc.
                return VerificationDTO(
                    message_id=raw["message_id"],
                    user_email=raw["user_email"],
                    verified_at=verified_at,
                    token=raw["token"],
                )
            except (ValueError, TypeError):
                return "malformed"

The validator stays pure – no I/O, no network calls – which makes it trivial to unit‑test in isolation.


7. Example usage in a CI test

    import pytest

    def test_email_verification_flow():
        # Arrange – fake provider that will emit a payload after 1.2 s
        fake_provider = FakeProvider(
            schedule=[
                (1.2, {
                    "message_id": "msg-123",
                    "user_email": "alice@example.com",
                    "verified_at": str(time.monotonic() + 1.2),
                    "token": "opaque‑value"
                })
            ]
        )
        poller = PollingService(fake_provider, deadline_seconds=5.0)

        # Act
        result = poller.poll(start_time=time.monotonic())

        # Assert
        assert isinstance(result, VerificationDTO)
        assert result.message_id == "msg-123"
        assert result.user_email == "alice@example.com"

FakeProvider (shown later) respects the same fetch_latest contract but reads from an in‑memory schedule, enabling deterministic tests without real email traffic.


8. Unit‑test suite with a fake clock

    class FakeClock:
        """A minimal monotonic clock that can be advanced manually."""
        def __init__(self, start: float = 0.0):
            self._now = start

        def monotonic(self) -> float:
            return self._now

        def advance(self, seconds: float):
            self._now += seconds

    class FakeProvider(ProviderAdapter):
        """Emits pre‑programmed payloads based on the fake clock."""
        def __init__(self, schedule: list[tuple[float, dict]]):
            # schedule = [(delay_from_start, payload), ...]
            self.schedule = schedule
            self.start = None

        def fetch_latest(self, since: float) -> Optional[dict]:
            if self.start is None:
                self.start = since
            elapsed = time.monotonic() - self.start
            for delay, payload in self.schedule:
                if delay <= elapsed:
                    # Return once and then remove to simulate single delivery
                    self.schedule.remove((delay, payload))
                    return payload
            return None

    def test_timeout_behavior(monkeypatch):
        fake_clock = FakeClock()
        monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)
        # Provider never yields a payload
        provider = FakeProvider(schedule=[])
        poller = PollingService(provider, deadline_seconds=2.0, max_attempts=3)

        result = poller.poll(start_time=fake_clock.monotonic())
        assert result == "timeout"

    def test_duplicate_delivery(monkeypatch):
        fake_clock = FakeClock()
        monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)

        payload = {
            "message_id": "dup-001",
            "user_email": "bob@example.com",
            "verified_at": "0",
            "token": "t"
        }
        provider = FakeProvider(schedule=[(0.1, payload), (0.5, payload)])
        poller = PollingService(provider, deadline_seconds=5.0)

        # First poll consumes the payload
        first = poller.poll(start_time=fake_clock.monotonic())
        assert isinstance(first, VerificationDTO)

        # Advance clock – second poll sees the same message_id
        fake_clock.advance(1.0)
        second = poller.poll(start_time=fake_clock.monotonic())
        assert second == "duplicate"

    def test_malformed_payload(monkeypatch):
        fake_clock = FakeClock()
        monkeypatch.setattr(time, "monotonic", fake_clock.monotonic)

        malformed = {"user_email": "carl@example.com"}   # missing required keys
        provider = FakeProvider(schedule=[(0.2, malformed)])
        poller = PollingService(provider, deadline_seconds=3.0)

        result = poller.poll(start_time=fake_clock.monotonic())
        assert result == "malformed"

9. QA checklist for the polling component


10. Where automation helps


11. Conclusion

A well‑engineered polling component turns an inherently flaky asynchronous verification step into a deterministic, testable building block. By anchoring all time calculations to time.monotonic(), applying bounded exponential backoff with a seeded jitter, and enforcing idempotency through stable message IDs, we obtain:

Integrating this component into AI‑driven email verification workflows gives QA engineers confidence that the “email verified” signal is both real and reproducible, while keeping the test suite fast, reliable, and maintainable.


12. Review questions

  1. Why is time.monotonic() preferred over time.time() for deadline calculations in a polling loop?
  2. How does seeding the random number generator with the attempt count make jitter deterministic in tests?
  3. What mechanism prevents the polling service from processing the same verification message more than once?
  4. List two pieces of information that must never appear in structured logs according to the design.
  5. In the provided unit tests, how is the provider’s “no payload” scenario simulated without external services?

Sources