Fault Correlation Ticket Routing Automation

SLA-Based Escalation Policies for Network Incidents

An escalation policy is where a service-level agreement stops being a contract clause and becomes running code. The failure it prevents is subtle and expensive: a correlated incident is created, routed to a queue, and then simply waits because no rule watched its acknowledgement clock. By the time a human notices, the mean time to acknowledge (MTTA) budget is gone, the mean time to resolve (MTTR) budget is under threat, and a P1 transport-ring outage has been sitting behind a wall of minor interface flaps. A well-encoded policy turns each SLA budget into an active timer that promotes the incident before it breaches — reassigning it to a faster tier and a fresh responder while there is still headroom to fix the fault. The measurable payoff is direct: NOCs that move from static, manually watched queues to timer-driven policies routinely cut P1 MTTA from minutes to under ninety seconds and shave double-digit percentages off MTTR by eliminating the dead time between creation and ownership.

Schema Alignment and Taxonomy Anchor

This page is the policy-encoding detail of Escalation Routing, the stage that owns queue assignment and on-call handoff. That stage supplies the runtime — the incident index, the timer loop, and the promotion mechanics — and this reference specifies the rules those mechanics execute: which budgets apply per priority, how a breach-risk projection is computed, and how reassignment preserves priority order. The incident objects a policy reasons over are the same strict-mode events produced by the correlation tier, so priority, dedup_key, and event_time carry the same guarantees they had when Ticket Deduplication established the incident’s identity. A policy never recomputes severity or identity; it consumes the priority it is handed and translates it into time budgets.

P1 escalation timeline against MTTA and MTTR budgetsA horizontal timeline for a P1 incident. At time zero the incident is created and both the acknowledgement and resolution clocks start. An acknowledgement escalation timer fires at ninety seconds, promoting an unacknowledged incident to a secondary responder; the contractual MTTA budget of five minutes sits further right, showing the headroom the early promotion preserves. On a second track, a resolution escalation timer fires at thirty minutes if the fault is still open, and the contractual MTTR budget of sixty minutes sits at the end, showing the restoration headroom the promotion leaves for the higher tier.MTTA track — acknowledgement clockt = 0incident created90 spromote → secondary5 minMTTA budget (breach)MTTR track — resolution clockt = 030 minpromote if still open60 minMTTR budget (breach)

Production Code: Encoding the Policy

A policy has two jobs. It maps a priority to its budgets and timers, and it decides — given how long an incident has already waited — whether to promote now on projected breach risk rather than waiting for a timer to fire. The code below expresses both as pure, testable functions over a Pydantic V2 policy model, then wires them into the async timer that Escalation Routing runs per incident. Everything is non-blocking: the projection is arithmetic, and the only wait is a cancellable asyncio.sleep on the incident’s own timer task.

import asyncio
import time
import logging
from enum import Enum

from pydantic import BaseModel, Field, field_validator, ConfigDict

# asyncio.sleep and task cancellation: https://docs.python.org/3/library/asyncio-task.html
logger = logging.getLogger("sla_policy")


class Priority(str, Enum):
    P1 = "P1"
    P2 = "P2"
    P3 = "P3"


class EscalationPolicy(BaseModel):
    """MTTA/MTTR budgets and timers for one priority tier, in seconds."""
    model_config = ConfigDict(strict=True, extra="forbid", frozen=True)

    priority: Priority
    mtta_budget: float = Field(gt=0)      # contractual acknowledgement ceiling
    mttr_budget: float = Field(gt=0)      # contractual resolution ceiling
    ack_timer: float = Field(gt=0)        # promote-if-unacked interval
    resolve_timer: float = Field(gt=0)    # promote-if-unresolved interval
    breach_risk_factor: float = Field(ge=0.5, le=0.95)  # fraction of budget = risk line
    promote_to: Priority | None = None
    queue: str

    @field_validator("ack_timer")
    @classmethod
    def ack_before_budget(cls, v: float, info) -> float:
        # The escalation timer must fire BEFORE the acknowledgement budget,
        # or there is no headroom to engage a second responder.
        budget = info.data.get("mtta_budget")
        if budget is not None and v >= budget:
            raise ValueError("ack_timer must be shorter than mtta_budget")
        return v


# Concrete carrier-grade policy set. P1 is aggressive; P3 is patient.
POLICIES: dict[Priority, EscalationPolicy] = {
    Priority.P1: EscalationPolicy(priority=Priority.P1, mtta_budget=300, mttr_budget=3600,
                                  ack_timer=90, resolve_timer=1800, breach_risk_factor=0.75,
                                  promote_to=None, queue="noc-p1-transport"),
    Priority.P2: EscalationPolicy(priority=Priority.P2, mtta_budget=900, mttr_budget=14400,
                                  ack_timer=480, resolve_timer=7200, breach_risk_factor=0.8,
                                  promote_to=Priority.P1, queue="noc-p2-network"),
    Priority.P3: EscalationPolicy(priority=Priority.P3, mtta_budget=3600, mttr_budget=86400,
                                  ack_timer=1800, resolve_timer=43200, breach_risk_factor=0.85,
                                  promote_to=Priority.P2, queue="noc-p3-regional"),
}


def mtta_breach_risk(policy: EscalationPolicy, waited: float) -> bool:
    """True when the acknowledgement clock has crossed the risk line.

    Rather than waiting for a hard breach, we promote once the incident has
    consumed breach_risk_factor of its MTTA budget — the point past which a
    normal paging round trip could no longer land inside the budget.
    """
    return waited >= policy.mtta_budget * policy.breach_risk_factor


def projected_mttr_breach(policy: EscalationPolicy, waited: float,
                          typical_fix_seconds: float) -> bool:
    """True when finishing at the current tier would blow the MTTR budget.

    If the time already elapsed plus a typical fix at this tier would exceed
    the resolution budget, the fault should move to a faster tier now.
    """
    return waited + typical_fix_seconds > policy.mttr_budget

The two predicates capture the whole idea of escalating on breach risk. mtta_breach_risk promotes an unacknowledged incident once it has burned a configurable fraction of its acknowledgement budget, and projected_mttr_breach promotes a still-open incident once finishing at the current tier would overrun the resolution budget. Neither waits for an actual breach — the whole point is to reassign while headroom remains.

Async Processing Hook

The policy predicates are evaluated inside the same per-incident timer task the routing stage arms. The hook below shows how a single incident’s acknowledgement clock is watched without blocking any other incident: the task sleeps on the ack_timer, and on expiry it re-checks breach risk against the wall clock before promoting. Acknowledgement cancels the task cleanly.

async def watch_ack_clock(dedup_key: str, policy: EscalationPolicy,
                          routed_at: float, promote, is_acknowledged) -> None:
    """One cancellable timer task per incident. Never blocks the loop for
    more than this incident's own deadline."""
    try:
        await asyncio.sleep(policy.ack_timer)
    except asyncio.CancelledError:
        return  # a responder acknowledged in time — clock stops, no promotion

    if await is_acknowledged(dedup_key):
        return  # raced with a late ack; nothing to escalate

    waited = time.monotonic() - routed_at
    if mtta_breach_risk(policy, waited):
        # Reassignment keeps P1 transport ahead of noise: a promotion pushes
        # the incident UP the priority order, never down.
        target = policy.promote_to or policy.priority
        logger.warning("MTTA breach risk on %s after %.0fs; promoting to %s",
                       dedup_key, waited, target.value)
        await promote(dedup_key, target, reason="MTTA_BREACH_RISK")


async def reassign_preserving_priority(dedup_key: str, current: Priority,
                                       target: Priority, queues) -> str:
    """Move an incident to the target tier's queue. P1 transport faults are
    inserted ahead of lower-priority work already waiting."""
    if target == Priority.P1:
        await queues[target].put_front(dedup_key)   # jump the P1 queue head
    else:
        await queues[target].put(dedup_key)
    logger.info("reassigned %s from %s to %s", dedup_key, current.value, target.value)
    return queues[target].name

Because each incident owns exactly one cancellable timer task, ten thousand incidents cost ten thousand cheap sleeping coroutines, not ten thousand threads — the event loop only wakes for the next deadline. The put_front call on P1 reassignment is the concrete mechanism that keeps a promoted transport fault ahead of the low-priority noise already queued, so a storm of minor flaps can never delay a genuine outage.

Mitigation and Hardening

A policy engine that fires timers is a small piece of code with several sharp failure edges. Each has a defined containment path:

  1. Clock-skew poisoning. Every budget is relative to elapsed time, so use a monotonic clock (time.monotonic) for elapsed-time arithmetic, never wall time, and reject any incident whose event_time is in the future at the ingestion boundary. A future-dated incident would otherwise appear pre-breached and trigger an instant, spurious promotion.
  2. Promotion loops. A misconfigured policy where promote_to cycles back to a lower tier could bounce an incident forever. Enforce a strict monotonic promotion order at load time — a promotion may only move toward P1 — and cap the number of promotions per incident, after which it pages a duty manager instead of re-queuing.
  3. Budget starvation under storm. When hundreds of P2s promote into P1 at once, the P1 queue itself can breach. Alert on the promotion rate into each tier, and when it exceeds the tier’s drain capacity, apply the same admission discipline the ingestion tier uses so the correlation and routing stages are not overwhelmed — shed or aggregate the lowest priority first, never the top.
  4. Policy reload safety. Treat POLICIES as immutable data and swap the whole map atomically; the frozen=True model config makes an accidental in-place mutation raise rather than silently corrupt an in-flight timer.

Operational Hardening Notes

The performance profile of a policy engine is dominated by timer-task count, not by the predicates, which are trivial arithmetic. Keep the per-incident footprint to two cancellable tasks and nothing else — no per-incident background polling loop, which would turn N incidents into N busy coroutines. Set breach_risk_factor from measured paging latency rather than intuition: if your page-plus-acknowledge round trip has a p95 of forty seconds against a three-hundred-second P1 budget, a factor around 0.75 leaves a full round trip of headroom after the risk line is crossed. Re-derive it whenever the on-call tooling or rotation structure changes, because a slower paging path silently erodes the margin. Finally, keep the reassignment path allocation-free on the hot edge: pre-resolve queue handles once at startup so a promotion is a queue insertion, not a directory lookup, and the whole promote decision stays well inside the 250-millisecond budget the routing stage defends.

Frequently Asked Questions

Why promote on breach risk instead of waiting for the SLA to actually breach?

Because a breach is already a failure — promoting after it is missed only changes who is blamed, not the outcome. Promoting on breach risk fires when the incident has consumed a configurable fraction of its budget, at the point past which a normal paging round trip could no longer land inside the budget. That leaves real headroom for a second responder to acknowledge a P1 or for a faster tier to restore service before the resolution budget is exhausted.

How do the acknowledgement and resolution timers differ?

The acknowledgement timer watches the MTTA clock and fires when an incident stays unacknowledged too long, engaging a fresh responder. The resolution timer watches the MTTR clock and fires when an incident is acknowledged but still open as the resolution budget approaches, moving the fault to a tier with the capacity to fix it. They are independent, so an incident that is acknowledged quickly but hard to fix still escalates on the resolution track.

How does reassignment keep a P1 transport fault ahead of low-priority noise?

Promotion always moves an incident toward the top of the priority order, never down, and a P1 reassignment is inserted at the head of the P1 queue rather than the tail. A promoted transport outage therefore jumps ahead of any minor interface flaps already waiting, so a storm of low-severity events cannot delay acknowledgement or resolution of a genuine service-affecting fault.

Can a policy misconfiguration bounce an incident between tiers forever?

Not if promotion order is enforced at load time and promotions are capped. The engine requires that every promote target moves strictly toward P1, which forbids a cycle, and it limits the number of promotions per incident. Once that cap is reached the incident pages a duty manager directly instead of re-queuing, so a bad policy surfaces as a human escalation rather than an infinite loop.