Fault Correlation Ticket Routing Automation

On-Call Handoff with PagerDuty Events API Webhooks

The handoff is the moment an automated pipeline stops reasoning about an incident and asks a human to own it. It is also the moment most likely to silently blow the mean time to acknowledge (MTTA) budget. If the page never lands, the on-call engineer never acknowledges, and the escalation clock ticks toward a breach that no dashboard predicted. Worse, a naive integration that fires a fresh page on every retry or re-fire produces alert fatigue — the responder mutes the noisy service, and the next page, the real one, goes unseen. Handing correlated incidents to PagerDuty through its Events API, with a stable dedup key and a disciplined acknowledge/resolve lifecycle, is what keeps a single fault mapped to a single alert, a single page, and a single acknowledgement — so the MTTA measured against the SLA reflects real human response time rather than integration artifacts.

Schema Alignment and Taxonomy Anchor

This page details the responder-handoff mechanics that Escalation Routing invokes whenever it assigns or promotes an incident. That stage decides who should be paged and when; this reference covers how the page is delivered, acknowledged, and resolved without duplicating alerts or losing the acknowledgement signal that stops the escalation clock. The incident carried into the handoff is the same enriched object the correlation tier produced, so its dedup_key is the identity that Ticket Deduplication already established — and reusing that exact key as the Events API deduplication key is what binds one fault to one alert end to end. PagerDuty is named here as the on-call platform; the pattern applies to any Events-API-style incident interface.

The Events API models three lifecycle actions against a single alert identity: trigger opens or re-fires an alert, acknowledge records that a human owns it, and resolve closes it. The critical property is that all three carry the same dedup_key, so a re-fire updates the open alert rather than opening a second one.

Trigger, acknowledge, and resolve lifecycle over one dedup keyA correlated incident passes through severity mapping and dedup-key reuse into a trigger event sent to the PagerDuty Events API. The API opens one alert and pages the on-call responder. An acknowledge webhook returns from the responder and stops the escalation clock in the router. A repeated trigger carrying the same dedup key updates the existing alert rather than opening a second one. A resolve event, also carrying the same dedup key, closes the alert. The single dedup key threads through every action so one fault maps to exactly one alert.Correlatedincidentseverity + dedup_keyMap + triggerevent_action=triggerdedup_keyPagerDutyEvents APIone alert perdedup_keyOn-callresponderpagedacknowledge → stop escalation clockre-fire: same dedup_key updates alertResolvecloses alert

Production Code: Events API Dispatch

The dispatcher below is the sink the routing stage drains its handoff queue into. It maps the incident’s internal severity to the Events API severity vocabulary, reuses the incident dedup_key verbatim as the Events API deduplication key, and issues trigger, acknowledge, and resolve actions over an async HTTP client. Every call is non-blocking, every failure is retried with backoff against the same idempotent key, and the response is validated so a partial success is never mistaken for a delivered page.

import asyncio
import logging
from enum import Enum

import httpx  # async HTTP client
from pydantic import BaseModel, Field, ConfigDict

# asyncio timeout + retry composition: https://docs.python.org/3/library/asyncio-task.html
logger = logging.getLogger("oncall_handoff")

EVENTS_API_URL = "https://events.example-oncall.internal/v2/enqueue"


class InternalSeverity(str, Enum):
    CRITICAL = "CRITICAL"
    MAJOR = "MAJOR"
    MINOR = "MINOR"
    INFO = "INFO"


# Map the internal fault severity onto the Events API vocabulary. Keeping this
# explicit prevents a P1 transport fault ever paging as low severity.
SEVERITY_MAP: dict[InternalSeverity, str] = {
    InternalSeverity.CRITICAL: "critical",
    InternalSeverity.MAJOR: "error",
    InternalSeverity.MINOR: "warning",
    InternalSeverity.INFO: "info",
}


class EventPayload(BaseModel):
    """A validated Events API v2 payload. dedup_key threads identity through
    trigger, acknowledge, and resolve."""
    model_config = ConfigDict(strict=True, extra="forbid")

    routing_key: str = Field(min_length=8)
    event_action: str                  # trigger | acknowledge | resolve
    dedup_key: str = Field(min_length=8)
    payload: dict | None = None        # required for trigger, omitted otherwise


class OnCallDispatcher:
    def __init__(self, routing_key: str, client: httpx.AsyncClient,
                 max_retries: int = 4):
        self._routing_key = routing_key
        self._client = client
        self._max_retries = max_retries

    def _build_trigger(self, incident: dict) -> EventPayload:
        sev = InternalSeverity(incident["severity"])
        return EventPayload(
            routing_key=self._routing_key,
            event_action="trigger",
            dedup_key=incident["dedup_key"],   # reuse the correlated identity
            payload={
                "summary": incident["summary"][:1024],
                "source": incident["affected_service"],
                "severity": SEVERITY_MAP[sev],
                "custom_details": {
                    "priority": incident["priority"],
                    "correlation_id": incident["dedup_key"],
                },
            },
        )

    async def _post(self, body: EventPayload) -> str:
        """POST one action with bounded retries. The same dedup_key makes every
        retry idempotent — a duplicated trigger updates the one alert."""
        delay = 0.5
        for attempt in range(1, self._max_retries + 1):
            try:
                resp = await self._client.post(
                    EVENTS_API_URL,
                    json=body.model_dump(exclude_none=True),
                    timeout=5.0,      # never block the loop indefinitely
                )
                if resp.status_code == 202:
                    return "DELIVERED"
                # 429/5xx are retryable; 4xx (bad payload) is not.
                if resp.status_code < 500 and resp.status_code != 429:
                    logger.error("non-retryable %s for %s: %s",
                                 resp.status_code, body.dedup_key, resp.text[:200])
                    return "REJECTED"
            except (httpx.TimeoutException, httpx.TransportError) as exc:
                logger.warning("transient handoff error for %s: %s",
                               body.dedup_key, exc)
            await asyncio.sleep(delay)
            delay = min(delay * 2, 8.0)   # exponential backoff, capped
        return "UNDELIVERED"

    async def trigger(self, incident: dict) -> str:
        return await self._post(self._build_trigger(incident))

    async def acknowledge(self, dedup_key: str) -> str:
        return await self._post(EventPayload(
            routing_key=self._routing_key, event_action="acknowledge",
            dedup_key=dedup_key))

    async def resolve(self, dedup_key: str) -> str:
        return await self._post(EventPayload(
            routing_key=self._routing_key, event_action="resolve",
            dedup_key=dedup_key))

Because trigger, acknowledge, and resolve all key off the same dedup_key, the integration is idempotent by construction: a retried trigger, a re-fired alarm, or a duplicated resolve all converge on the single alert that identity represents. That is precisely why the routing stage can retry a failed handoff aggressively without fear of paging the responder twice.

Async Processing Hook

The dispatcher slots into the routing stage’s handoff_worker — the coroutine that drains promotion and assignment events without blocking the timer loop. The hook below shows a worker consuming a bounded queue, dispatching each action, and feeding the acknowledge signal back into the router so the escalation clock stops the instant the responder owns the incident.

async def handoff_worker(queue: asyncio.Queue, dispatcher: OnCallDispatcher,
                         router) -> None:
    """Drain routing handoff events into the Events API. One slow POST never
    stalls the next incident because dispatch is awaited here, off the timer
    loop, and failures are bounded by the retry cap."""
    while True:
        event = await queue.get()
        try:
            action = event["action"]
            if action in ("REASSIGN", "TRIGGER", "PAGE_SECONDARY"):
                status = await dispatcher.trigger(event["incident"])
                if status != "DELIVERED":
                    # Undelivered page is an SLA risk: re-queue once at the
                    # front so the next drain retries before newer, lower work.
                    logger.error("page %s undelivered (%s); re-queueing",
                                 event["incident"]["dedup_key"], status)
                    await queue.put(event)
            elif action == "ACK":
                await dispatcher.acknowledge(event["dedup_key"])
                await router.acknowledge(event["dedup_key"])  # stop the clock
            elif action == "RESOLVE":
                await dispatcher.resolve(event["dedup_key"])
        except Exception:
            logger.exception("handoff worker error for %s", event.get("dedup_key"))
        finally:
            queue.task_done()

The acknowledge path is the one that protects MTTA directly: the moment PagerDuty reports the responder acknowledged, the worker calls back into the router to cancel the acknowledgement timer, so the escalation clock stops at the true human-response time rather than running on to a spurious promotion.

Mitigation and Hardening

  1. Dedup-key discipline. Reuse the correlated dedup_key verbatim for every action; never generate a per-attempt key. A fresh key on retry opens a second alert and double-pages the responder. Alert when the count of distinct open alerts diverges from the count of open incidents — divergence means key drift.
  2. Undelivered-page detection. Treat anything other than a 202 acceptance as an SLA risk. Re-queue undelivered triggers ahead of newer work, and if a page remains undelivered past a hard cap, fall back to a secondary notification channel and raise a monitoring alert — an unacknowledged page that never arrived is invisible to MTTA otherwise.
  3. Severity-map integrity. Keep the severity mapping explicit and total, with a fail-closed default that maps unknown severities to critical rather than info. A silently under-mapped P1 is worse than a slightly over-paged P3.
  4. Poison-payload isolation. A 4xx from the Events API is a malformed payload, not a transient fault; do not retry it. Route it to a dead-letter queue with the validation detail so the summary or custom-details bug is fixed once, rather than hammered against the API.

Operational Hardening Notes

Reuse a single pooled httpx.AsyncClient across all dispatches rather than constructing one per call; connection setup dwarfs the request itself and a per-call client will exhaust file descriptors under a promotion storm. Keep the per-request timeout tight — five seconds is generous for an enqueue endpoint — so a stalled connection surfaces as a fast retry rather than a blocked worker holding an SLA-critical page. Size the backoff so the total retry budget fits comfortably inside the MTTA budget of the lowest priority you page: four attempts at capped exponential backoff span well under a minute, leaving the acknowledgement clock intact even for a P1. Run several handoff_worker coroutines against one queue for throughput, but keep the acknowledge and resolve actions idempotent so concurrent workers processing a re-fire and its acknowledgement cannot corrupt the alert state. With a pooled client, tight timeouts, and idempotent keys, the handoff sustains thousands of actions per minute during a storm while holding the p95 page-delivery latency under two seconds.

Frequently Asked Questions

Why reuse the correlated dedup key as the Events API deduplication key?

Because it binds one fault to exactly one alert across every action. When trigger, acknowledge, and resolve all carry the same key, a re-fired alarm or a retried request updates the single open alert instead of opening a second one, so the responder is paged once and the acknowledgement maps back to the right incident. Generating a fresh key per attempt would double-page the responder and break the link between the alert and the escalation clock.

How does the acknowledge webhook keep MTTA under budget?

When the responder acknowledges in PagerDuty, the handoff worker receives that signal and immediately calls back into the router to cancel the incident acknowledgement timer. The escalation clock stops at the true human-response time, so no spurious promotion fires and the recorded MTTA reflects real response rather than an integration delay. Without that callback the clock would keep running even after a human had already taken ownership.

What happens when a page fails to deliver?

Anything other than a 202 acceptance is treated as an SLA risk. The worker re-queues an undelivered trigger ahead of newer work and retries with capped exponential backoff against the same idempotent key. If it stays undelivered past a hard cap, the integration falls back to a secondary notification channel and raises a monitoring alert, because an unacknowledged page that never arrived would otherwise be invisible to the MTTA measurement.

Should a malformed-payload rejection be retried?

No. A 4xx rejection means the payload itself is wrong — an oversized summary or a bad custom-details field — and retrying it only wastes the retry budget while the page still never lands. Those are routed to a dead-letter queue with the validation detail so the bug is fixed once. Only timeouts, transport errors, rate-limit 429s, and 5xx server errors are retried, because only those are transient.