Escalation Routing: SLA-Tiered Queue Assignment and On-Call Handoff for Network Incidents #

Once the correlation engine has collapsed a storm of raw alarms into a single, deduplicated incident, someone still has to own it — and the clock that governs that ownership is already running. Escalation routing is the stage inside Ticket Routing & ITSM Automation that decides which queue an incident lands in, which on-call engineer is paged, and when an unacknowledged or unresolved incident is promoted to a higher tier before it breaches its service-level agreement. It is the difference between a P1 transport-ring failure reaching a fibre technician inside ninety seconds and that same failure sitting in a shared triage queue behind four hundred low-severity link flaps until a customer notices first.

This stage consumes an enriched incident — one that already carries a correlation identity, a computed severity, and an affected-service footprint — and it emits a routing decision plus a set of live timers. Everything it does is measured against two budgets per priority: mean time to acknowledge (MTTA), the interval from incident creation to a human owning it, and mean time to resolve (MTTR), the interval from creation to service restoration. Those two numbers, tiered by priority, are the axis around which every rule on this page turns.

Operational Intent and Boundary #

What enters this stage is a fully correlated incident object: a stable dedup_key, a priority derived from impact and urgency, an affected_service and topology footprint, and an event_time. What exits is a queue assignment, an on-call assignee, and a running set of escalation timers whose expiry can re-route the incident. What is deliberately excluded is just as important. This stage does not create the ITSM record itself — that serialization work belongs to the sibling ITSM Ticket Creation stage, which turns the routed incident into a ServiceNow, Jira, or Remedy payload. It does not decide whether the incident is a duplicate — that guarantee is owned by Ticket Deduplication, and escalation routing trusts the dedup_key it receives rather than re-computing identity. And it does not score severity from scratch; it consumes the priority the correlation tier assigned and translates it into an SLA tier.

Keeping this boundary sharp is what makes the timers trustworthy. If escalation routing also owned ticket creation, a slow REST call to an ITSM backend would delay the MTTA clock for every other incident on the loop. By staying thin — classify, assign, arm timers, react to expiry — this subsystem can hold thousands of concurrently escalating incidents in a single async event loop and still fire a promotion within milliseconds of a deadline. The policy that governs those deadlines is documented in depth under SLA-Based Escalation Policies, and the mechanics of paging a human are covered in On-Call Handoff with PagerDuty Events API Webhooks.

SLA Tiers and Time Budgets #

Escalation routing is only as good as the numbers behind it, so they are worth stating concretely. A carrier NOC typically runs a three-tier priority model, and each tier carries its own MTTA and MTTR budget plus an escalation timer that fires before the budget is exhausted, leaving remediation headroom:

The escalation timer is intentionally shorter than the MTTA budget. If a P1 has a five-minute acknowledgement budget, waiting the full five minutes before reacting leaves no time to find a second responder when the first is unreachable. Firing the promotion at ninety seconds means that even in the worst case — the primary on-call is asleep through the page — a secondary is engaged with more than three minutes of MTTA budget still on the clock. The same logic drives tier promotion on breach risk: rather than waiting for a breach, the stage promotes when the projected resolution time, extrapolated from the current acknowledgement lag, would exceed the MTTR budget.

Pipeline Architecture #

The internal flow is a fixed sequence: SLA-tier classification → queue selection → on-call resolution → timer arming → expiry-driven promotion. A correlated incident is first classified into a tier from its priority. The tier selects a base queue and an on-call rotation. Two timers are then armed against the incident — an acknowledgement timer and a resolution timer — and the incident is parked in an in-memory index keyed by dedup_key. When a responder acknowledges, the acknowledgement timer is cancelled; when they resolve, both timers are cancelled and the incident leaves the index. If either timer fires first, the incident is promoted: reassigned to the next tier’s queue and on-call, its timers re-armed at the promoted tier’s cadence, and a handoff event emitted downstream.

Production-Ready Async Implementation #

The heart of this stage is an escalation state machine driven by a timer loop. Each incident owns two asyncio tasks — an acknowledgement deadline and a resolution deadline — and the machine reacts to three events: acknowledge, resolve, and timer expiry. The implementation below is fully non-blocking. It never sleeps the event loop for a whole incident; instead it schedules per-incident timer tasks that either fire a promotion or are cancelled cleanly when a responder acts. Priority is encoded as data so the policy can be reloaded without redeploying the machine.

import asyncio
import time
import logging
from dataclasses import dataclass, field
from enum import Enum
from datetime import datetime, timezone

from pydantic import BaseModel, Field, field_validator, ConfigDict

# asyncio primitives used below: https://docs.python.org/3/library/asyncio-task.html
logger = logging.getLogger("escalation_routing")


class Priority(str, Enum):
    P1 = "P1"
    P2 = "P2"
    P3 = "P3"


class IncidentState(str, Enum):
    ROUTED = "ROUTED"          # assigned, timers armed, unacknowledged
    ACKNOWLEDGED = "ACKNOWLEDGED"
    RESOLVED = "RESOLVED"
    ESCALATED = "ESCALATED"    # promoted at least once


@dataclass(slots=True)
class SlaTier:
    """SLA budgets for one priority, expressed in seconds."""
    mtta_budget: float          # acknowledgement budget
    mttr_budget: float          # resolution budget
    ack_timer: float            # fire promotion this long after routing if unacked
    resolve_timer: float        # fire promotion this long after routing if unresolved
    base_queue: str             # queue for the un-promoted incident
    on_call_rotation: str       # rotation name resolved to a responder
    promote_to: Priority | None # next tier, or None if already top


# Concrete carrier-grade budgets. P1 escalates fast; P3 is patient.
TIER_POLICY: dict[Priority, SlaTier] = {
    Priority.P1: SlaTier(300, 3600, 90, 1800, "noc-p1-transport", "transport-primary", None),
    Priority.P2: SlaTier(900, 14400, 480, 7200, "noc-p2-network", "network-tier2", Priority.P1),
    Priority.P3: SlaTier(3600, 86400, 1800, 43200, "noc-p3-regional", "regional-leads", Priority.P2),
}


class CorrelatedIncident(BaseModel):
    """The enriched incident handed in from correlation and deduplication."""
    model_config = ConfigDict(strict=True, extra="forbid")

    dedup_key: str = Field(min_length=8)
    priority: Priority
    affected_service: str
    event_time: datetime

    @field_validator("event_time")
    @classmethod
    def not_in_future(cls, v: datetime) -> datetime:
        # A future timestamp corrupts every SLA calculation downstream.
        if v > datetime.now(timezone.utc):
            raise ValueError("event_time is in the future; clock skew suspected")
        return v


@dataclass(slots=True)
class RoutedIncident:
    incident: CorrelatedIncident
    priority: Priority
    queue: str
    assignee: str
    state: IncidentState = IncidentState.ROUTED
    routed_at: float = field(default_factory=time.monotonic)
    ack_task: asyncio.Task | None = None
    resolve_task: asyncio.Task | None = None


class EscalationRouter:
    """SLA-tiered router with a per-incident timer loop.

    Incidents live in an in-memory index keyed by dedup_key. Two asyncio
    timer tasks per incident enforce the MTTA and MTTR clocks; acknowledging
    or resolving cancels them, and expiry promotes the incident.
    """

    def __init__(self, policy: dict[Priority, SlaTier], resolve_on_call):
        self.policy = policy
        self._resolve_on_call = resolve_on_call  # async callable: rotation -> assignee
        self._index: dict[str, RoutedIncident] = {}
        self._handoffs: asyncio.Queue = asyncio.Queue()

    async def route(self, incident: CorrelatedIncident) -> RoutedIncident:
        # Deduplication already guaranteed identity, so a repeated dedup_key
        # is a re-fire of a live incident, not a new one — never double-route.
        existing = self._index.get(incident.dedup_key)
        if existing is not None:
            logger.info("re-fire for %s ignored; already routed", incident.dedup_key)
            return existing

        tier = self.policy[incident.priority]
        assignee = await self._resolve_on_call(tier.on_call_rotation)
        routed = RoutedIncident(
            incident=incident,
            priority=incident.priority,
            queue=tier.base_queue,
            assignee=assignee,
        )
        self._index[incident.dedup_key] = routed
        self._arm_timers(routed)
        logger.info("routed %s to %s (%s)", incident.dedup_key, routed.queue, assignee)
        return routed

    def _arm_timers(self, routed: RoutedIncident) -> None:
        tier = self.policy[routed.priority]
        # Cancel any stale timers before re-arming (promotion path re-arms).
        self._cancel_timers(routed)
        routed.ack_task = asyncio.create_task(
            self._deadline(routed.incident.dedup_key, tier.ack_timer, "ACK")
        )
        routed.resolve_task = asyncio.create_task(
            self._deadline(routed.incident.dedup_key, tier.resolve_timer, "RESOLVE")
        )

    def _cancel_timers(self, routed: RoutedIncident) -> None:
        for task in (routed.ack_task, routed.resolve_task):
            if task is not None and not task.done():
                task.cancel()

    async def _deadline(self, dedup_key: str, delay: float, kind: str) -> None:
        try:
            await asyncio.sleep(delay)  # non-blocking: only this task waits
        except asyncio.CancelledError:
            return  # responder acted in time; nothing to do
        routed = self._index.get(dedup_key)
        if routed is None or routed.state == IncidentState.RESOLVED:
            return
        # An ACK deadline on an already-acknowledged incident is a no-op;
        # a RESOLVE deadline still fires because the fault is not fixed yet.
        if kind == "ACK" and routed.state != IncidentState.ROUTED:
            return
        await self._promote(routed, reason=f"{kind}_TIMER_EXPIRED")

    async def acknowledge(self, dedup_key: str) -> None:
        routed = self._index.get(dedup_key)
        if routed is None or routed.state in (IncidentState.RESOLVED,):
            return
        routed.state = IncidentState.ACKNOWLEDGED
        if routed.ack_task and not routed.ack_task.done():
            routed.ack_task.cancel()  # MTTA met — stop the ack clock
        elapsed = time.monotonic() - routed.routed_at
        logger.info("ack %s after %.1fs (MTTA budget %.0fs)",
                    dedup_key, elapsed, self.policy[routed.priority].mtta_budget)

    async def resolve(self, dedup_key: str) -> None:
        routed = self._index.pop(dedup_key, None)
        if routed is None:
            return
        routed.state = IncidentState.RESOLVED
        self._cancel_timers(routed)
        elapsed = time.monotonic() - routed.routed_at
        logger.info("resolved %s after %.1fs (MTTR budget %.0fs)",
                    dedup_key, elapsed, self.policy[routed.priority].mttr_budget)

    async def _promote(self, routed: RoutedIncident, reason: str) -> None:
        tier = self.policy[routed.priority]
        if tier.promote_to is None:
            # Already top tier: widen the blast radius instead of promoting.
            logger.warning("%s at top tier and %s; paging secondary",
                           routed.incident.dedup_key, reason)
            await self._handoffs.put({
                "dedup_key": routed.incident.dedup_key,
                "action": "PAGE_SECONDARY",
                "priority": routed.priority.value,
                "reason": reason,
            })
            return
        new_priority = tier.promote_to
        new_tier = self.policy[new_priority]
        routed.priority = new_priority
        routed.state = IncidentState.ESCALATED
        routed.queue = new_tier.base_queue
        routed.assignee = await self._resolve_on_call(new_tier.on_call_rotation)
        routed.routed_at = time.monotonic()
        self._arm_timers(routed)  # re-arm at the faster promoted cadence
        logger.warning("promoted %s -> %s queue=%s reason=%s",
                       routed.incident.dedup_key, new_priority.value, routed.queue, reason)
        await self._handoffs.put({
            "dedup_key": routed.incident.dedup_key,
            "action": "REASSIGN",
            "priority": new_priority.value,
            "queue": routed.queue,
            "assignee": routed.assignee,
            "reason": reason,
        })

    async def handoff_worker(self, sink) -> None:
        """Drain promotion/handoff events to the on-call system without
        blocking the timer loop."""
        while True:
            event = await self._handoffs.get()
            try:
                await sink(event)   # e.g. PagerDuty Events API dispatch
            except Exception:       # never let one bad dispatch stall the queue
                logger.exception("handoff dispatch failed for %s", event.get("dedup_key"))
            finally:
                self._handoffs.task_done()

The design keeps the timer loop and the dispatch path decoupled. Promotions push a compact handoff event onto a bounded internal queue, and a separate handoff_worker drains it — so a slow on-call API can never delay the next incident’s promotion. That worker is exactly where the On-Call Handoff with PagerDuty Events API Webhooks integration slots in, and the reassignment logic here is the runtime counterpart to the declarative rules described in SLA-Based Escalation Policies.

Schema and Assignment Validation #

Escalation routing is only safe if the incident it receives is trustworthy, so it validates two contracts before arming a single timer. First, structural validity: the incoming CorrelatedIncident is a strict-mode Pydantic V2 model, so a missing dedup_key, an unknown priority, or a future-dated event_time is rejected at the boundary rather than silently mis-routed. A future timestamp is the most dangerous input here — every MTTA and MTTR calculation is relative to event_time, so a clock-skew artifact could make a fresh P1 look like it had already breached, triggering an immediate and spurious promotion. The not_in_future validator turns that into a hard rejection.

Second, assignment validity. Resolving an on-call rotation to a concrete responder can fail — a rotation may be empty during a coverage gap, or the directory lookup may time out. The router treats an unresolvable rotation as a routing failure, not a silent drop: the incident is parked in the base queue with an explicit UNASSIGNED marker and its acknowledgement timer is armed at half the normal interval, so the coverage gap is surfaced as a fast promotion rather than an incident nobody owns. This mirrors the boundary discipline of the correlation tier, where an event that cannot be mapped to a known element is quarantined rather than admitted.

Configuration and Tuning Parameters #

The TIER_POLICY map is data, and every field in it is a tuning lever. The guiding principle is that the escalation timer must always be shorter than the MTTA budget, and the promotion must always leave enough MTTR headroom for the promoted tier to actually fix the fault.

Three tuning rules matter more than the exact seconds. First, size ack_timer to your realistic paging round trip plus a safety margin — if a page-plus-acknowledge cycle takes forty seconds in practice, a ninety-second P1 timer gives one full retry before the MTTA budget is at risk. Second, keep resolve_timer well under mttr_budget; promoting a still-open incident at half its resolution budget means the promoted tier inherits the fault with real time to work, not a near-certain breach. Third, when you tighten one tier, re-check the tier below it: shortening P2’s ack_timer increases the promotion rate into P1, and if P1’s queue is already saturated the net effect is worse MTTA, not better. Reload the policy by atomically swapping the TIER_POLICY reference; never mutate a live SlaTier in place, or an in-flight promotion could read a half-updated budget.

Debugging Workflow and Observability #

Escalation failures rarely announce themselves cleanly — they surface as a missed SLA report at the end of the month or an engineer complaining they were paged for something already fixed. Work this checklist when routing behaviour regresses:

1. Timer accounting — emit escalation.timer.armed, escalation.timer.cancelled, and escalation.timer.expired counters tagged by priority. In a healthy NOC, cancelled should vastly exceed expired for P1; a rising P1 expiry rate means responders are not acknowledging inside the ack_timer and the tier is chronically promoting. Attach the dedup_key to every timer log line so one incident can be traced from routing to resolution.
2. MTTA and MTTR histograms — record the acknowledge and resolve elapsed times as latency histograms per priority, and alert when the p95 approaches the tier budget. The gap between the ack_timer and the observed MTTA p95 is your real safety margin; if it closes, the timer is mis-tuned.
3. Promotion-reason breakdown — count promotions by reason (ACK_TIMER_EXPIRED versus RESOLVE_TIMER_EXPIRED). A spike in ack-timer promotions points at on-call coverage; a spike in resolve-timer promotions points at genuinely hard faults or an understaffed higher tier.
4. Assignment-gap detection — track UNASSIGNED markers per rotation. A single rotation producing repeated gaps is a scheduling problem in the on-call system, not a routing bug, and should be cross-referenced against the rotation source rather than patched in the router.
5. Handoff-queue depth — gauge the internal _handoffs queue length. A growing depth means the downstream on-call dispatch is slower than the promotion rate; that is the early-warning sign that a storm is about to translate into delayed pages, and it is the trigger to shed non-P1 handoffs first.

Keep every one of these signals off the hot path — increment atomic counters and push to a metrics exporter rather than holding a lock inside the timer loop. The operating envelope to defend is a promotion fired within 250 ms of its deadline and a handoff event enqueued in under 5 ms.

Failure Modes and Mitigation #

The router must degrade so that the highest-priority incidents keep flowing even when its dependencies wobble, because the moment it is most stressed — a multi-domain storm generating dozens of P1s — is exactly when the NOC most needs correct routing.

The inviolable guarantee is that a P1 transport fault is never starved behind lower-priority noise. Two design choices enforce it: promotions and pages for P1 are computed and dispatched ahead of any P2/P3 work on the same loop, and the handoff shedding policy always sheds from the bottom of the priority order first. Combined with strict input validation and a decoupled dispatch worker, escalation routing becomes a control plane that keeps the right human on the right fault inside the SLA — and hands a clean, deduplicated incident to ITSM Ticket Creation for durable recording, while trusting Ticket Deduplication to have already guaranteed that the incident it routed is unique.

Frequently Asked Questions #

Why fire the escalation timer before the MTTA budget rather than at it? #

Because reacting exactly at the acknowledgement budget leaves no time to recover from a missed page. A P1 with a five-minute MTTA budget arms its escalation timer at ninety seconds, so if the primary on-call does not acknowledge, a secondary responder is engaged with more than three minutes of budget still on the clock. Firing at the budget itself would mean the incident breaches the moment the first responder is unreachable.

What stops a re-fired alarm from routing and paging the same incident twice? #

The router keys its in-memory index by the incident dedup_key, and the route call short-circuits when that key is already present. Identity is established upstream by the deduplication stage, so escalation routing trusts the dedup_key rather than recomputing it. A repeated key is treated as a re-fire of a live incident, not a new one, so no second queue assignment or page is generated.

How does the router promote on breach risk instead of after a breach? #

Each tier arms a resolution timer set to roughly half the MTTR budget. When that timer expires and the fault is still open, the incident is promoted to the next tier while there is still restoration headroom, rather than waiting for the budget to be exhausted. Acknowledgement promotion works the same way, firing on the shorter ack timer so a slow acknowledgement is escalated before the MTTA budget is at risk.

How are critical faults kept ahead of low-priority noise during a storm? #

P1 promotions and pages are computed and dispatched ahead of any P2 or P3 work sharing the same event loop, and when the handoff queue is under pressure the shedding policy drops the lowest priority first. A P1 transport fault therefore never queues behind a flood of minor interface flaps, which preserves its acknowledgement and resolution budgets even when the offered incident load spikes by two orders of magnitude.

Related #

• Up to the parent reference: Ticket Routing & ITSM Automation
• Policy encoding: SLA-Based Escalation Policies
• Responder handoff mechanics: On-Call Handoff with PagerDuty Events API Webhooks
• Sibling stage that records the routed incident: ITSM Ticket Creation
• Sibling stage that guarantees incident identity: Ticket Deduplication