Fault Correlation Ticket Routing Automation

Suppressing Downstream Alarms with Adjacency Graphs

When an aggregation router loses power, every device behind it goes unreachable in the same instant — and every one of them screams. The access switches downstream lose their uplink and raise linkDown; the CPEs beyond them stop responding and trip reachability alarms; the services riding those links breach their probes. The management plane sees hundreds of alarms, but there is exactly one fault: the aggregation router. An engine that opens a ticket per alarm sends the NOC chasing dead CPEs while the real outage — the router that stranded them — sits in the same undifferentiated flood. The symptomatic alarms are not noise to be discarded; they are real, but they are downstream consequences of a single upstream root, and they must be suppressed as children of that root rather than routed as independent incidents.

The goal here is to walk a live adjacency graph at alarm time, classify each alarm as either a root cause or a downstream symptom of another alarm, and suppress the symptoms into the root’s incident. Done correctly, a power failure that would have generated three hundred tickets produces one root incident with the symptomatic alarms attached as suppressed children — cutting ticket volume by an order of magnitude during exactly the major-outage moments when the NOC can least afford noise, and preserving MTTA for the one incident that actually needs a human.

Schema Alignment and Taxonomy Anchor

This pattern is an application of Topology-Aware Correlation, the adjacency-validation stage inside the Fault Correlation & Rule Engines pipeline that proves two events are actually connected before asserting a relationship between them. Suppression is the natural consequence of that proof: once the graph shows that alarm B’s device is only reachable through alarm A’s device, and A’s device is down, B is a symptom of A. The binding assumes normalized input — every alarm carries a canonical device_id, an alarm_type, and a reconciled event_time, exactly as the upstream schema contracts require.

The topology model this walks is a directed adjacency graph whose construction is the subject of Building Graph-Based Fault Trees in Python: nodes are network elements, and a directed edge from an upstream element to a downstream one means the downstream element depends on the upstream for reachability. Suppression consumes that graph read-only; it never mutates the topology, only walks it. The taxonomy is deliberately small — an alarm is either a root (no upstream alarmed ancestor) or a symptom (reachable only through an alarmed ancestor) — and the walk is what assigns each alarm its class.

Reachability Pruning and Root-versus-Symptom Classification

The core operation is a reachability walk. Given the set of currently-alarmed devices, the engine asks of each: is this device reachable from the management plane by any path that does not pass through another alarmed device? If yes, the alarm is a root — the outage does not fully explain it. If no — every path to it is severed by some other alarmed device upstream — it is a symptom of that ancestor, and it is suppressed into the ancestor’s incident.

This is why a naive “same time window, same region” heuristic fails and a graph walk succeeds: temporal coincidence groups alarms that merely happened together, whereas reachability pruning proves that one alarm’s device sits behind another’s on the actual forwarding path. The walk runs breadth-first from each candidate root down the directed edges, marking every reachable alarmed descendant as a suppressed symptom, and stops descending as soon as it leaves the alarmed set — so a healthy device beyond the blast radius is never pruned, and an alarm on a device reachable by an independent path correctly survives as its own root.

Diagram: a single root fault severs reachability to a subtree; the walk classifies the root and suppresses its reachable alarmed descendants as symptoms, while an alarm on an independently-reachable branch survives.

Downstream alarm suppression over an adjacency graphA directed adjacency graph flows top to bottom. At the top a management-plane node connects down to two aggregation routers. The left aggregation router is alarmed and marked as the root cause; the two access switches and the CPE beneath it are all alarmed but, because every path to them passes through the down root, they are classified as suppressed symptoms and drawn with a dashed suppressed style. The right aggregation router is healthy, and a single access switch beneath it carries an independent alarm that is reachable without crossing the root, so it survives as its own separate root incident. A legend distinguishes root, suppressed symptom, and healthy nodes.Management planeAgg router AROOT · power downAgg router BhealthyAccess sw 1symptom · suppressedAccess sw 2symptom · suppressedCPEsymptom · suppressedAccess sw 3separate rootroot causesuppressed symptomhealthy

Production Suppression Worker

The following worker classifies a batch of concurrent alarms against a live adjacency graph. It builds the alarmed set, walks reachability from each candidate root breadth-first, and emits one SuppressionResult naming the roots and the symptoms suppressed under each. The topology resolver is injected and async so a cold subgraph fetch can await the inventory backend without blocking the loop; the hot path reads a warm in-memory adjacency cache. It performs no blocking I/O and is copy-paste runnable.

import asyncio
import logging
from collections import defaultdict, deque

from pydantic import BaseModel, ConfigDict, Field

logger = logging.getLogger("downstream_suppressor")


class Alarm(BaseModel):
    model_config = ConfigDict(frozen=True)

    alarm_id: str
    device_id: str
    alarm_type: str
    event_time: float


class SuppressionResult(BaseModel):
    root_device: str
    root_alarm_id: str
    suppressed_alarm_ids: list[str] = Field(default_factory=list)
    suppressed_devices: list[str] = Field(default_factory=list)
    symptom_count: int = 0


class AdjacencyGraph:
    """Directed reachability graph: edge u -> v means v depends on u to be reached.

    Held in memory and warmed from inventory. children() is the hot-path read; a
    cold miss awaits the backing store so the event loop never blocks on I/O.
    """

    def __init__(self, edges: dict[str, set[str]], upstreams: dict[str, set[str]]):
        self._children = edges           # device -> devices directly downstream
        self._upstreams = upstreams      # device -> devices directly upstream

    async def children(self, device: str) -> set[str]:
        await asyncio.sleep(0)           # placeholder for a cache-miss subgraph fetch
        return self._children.get(device, set())

    def upstreams(self, device: str) -> set[str]:
        return self._upstreams.get(device, set())


class DownstreamSuppressor:
    def __init__(self, graph: AdjacencyGraph):
        self.graph = graph

    async def suppress(self, alarms: list[Alarm]) -> list[SuppressionResult]:
        # Index alarms by device; one alarm per device is assumed post-dedup.
        by_device: dict[str, Alarm] = {a.device_id: a for a in alarms}
        alarmed = set(by_device)

        # A device is a root only if no upstream neighbour is also alarmed.
        roots = [
            dev for dev in alarmed
            if not (self.graph.upstreams(dev) & alarmed)
        ]

        results: list[SuppressionResult] = []
        claimed: set[str] = set()        # devices already suppressed under some root
        for root_dev in roots:
            root_alarm = by_device[root_dev]
            symptoms = await self._reachable_symptoms(root_dev, alarmed, claimed)
            claimed |= symptoms
            results.append(SuppressionResult(
                root_device=root_dev,
                root_alarm_id=root_alarm.alarm_id,
                suppressed_alarm_ids=[by_device[d].alarm_id for d in sorted(symptoms)],
                suppressed_devices=sorted(symptoms),
                symptom_count=len(symptoms),
            ))
            logger.info("root=%s suppressed=%d", root_dev, len(symptoms))
        return results

    async def _reachable_symptoms(self, root: str, alarmed: set[str],
                                  claimed: set[str]) -> set[str]:
        # Breadth-first walk down the graph, pruning as soon as we leave the
        # alarmed set: a healthy device stops the descent so beyond-blast-radius
        # elements are never suppressed.
        symptoms: set[str] = set()
        queue: deque[str] = deque([root])
        seen: set[str] = {root}
        while queue:
            node = queue.popleft()
            for child in await self.graph.children(node):
                if child in seen:
                    continue
                seen.add(child)
                if child in alarmed and child not in claimed:
                    # Reachable only via an alarmed ancestor -> a symptom.
                    symptoms.add(child)
                    queue.append(child)      # keep descending through the outage
                # A non-alarmed child is a live path; do not descend past it.
        return symptoms

Async Ingestion Hook

Suppression runs on a windowed batch of concurrent alarms rather than one alarm at a time, because classification is only meaningful across the set that fired together. A consumer collects alarms from the ingestion tier into a short grouping window, hands the batch to suppress, and forwards each root incident — with its symptoms attached — to routing. Because the graph walk touches only the alarmed subset and prunes at the first healthy node, the cost scales with the size of the outage, not the size of the network.

async def run_suppressor(batches: "asyncio.Queue[list[Alarm]]",
                         suppressor: DownstreamSuppressor,
                         emit: "asyncio.Queue[SuppressionResult]") -> None:
    while True:
        batch = await batches.get()
        try:
            for result in await suppressor.suppress(batch):
                await emit.put(result)   # backpressure: blocks if router is behind
        except Exception:
            logger.exception("suppression_failed batch_size=%d", len(batch))
        finally:
            batches.task_done()

The grouping window that assembles each batch is the same mechanism described in Temporal Windowing Strategies; suppression consumes a sealed group and classifies it. When ticket routing falls behind, the bounded emit queue applies backpressure so the suppressor drains in bounded micro-batches rather than collapsing under head-of-line blocking.

Mitigation and Hardening

The worker degrades toward over-reporting rather than over-suppressing, because a missed ticket is worse than a duplicate one. Concrete failure paths and mitigations:

  1. Stale topology. If the adjacency graph predates a recent re-home, the walk can suppress an alarm whose device no longer sits behind the root. Stamp every SuppressionResult with the topology version used, and when the graph is known-stale, degrade to marking symptoms suppression_unverified and routing them for review rather than silently hiding them.
  2. Cold-cache miss. A subgraph absent from the warm cache forces an inventory fetch mid-walk. Bound that fetch with a timeout; on timeout, treat the missing branch as not proven downstream and let its alarm survive as an independent root, so an inventory hiccup never suppresses a real fault.
  3. Cyclic or redundant paths. A device reachable by two upstreams, only one of which is alarmed, must survive — it still has a live path. The seen set and the “descend only through alarmed ancestors” rule guarantee a redundantly-reached device is never classified as a pure symptom.
  4. Idempotent sealing. Derive a deterministic incident key from the root device_id and the grouping-window epoch so a worker restart or active-active failover never opens the same root incident twice or re-suppresses an already-attached symptom.
  5. Dead-letter isolation. An alarm whose device_id is absent from the graph entirely routes to a dead-letter queue with the reason attached, preserved for replay once inventory catches up, never dropped.

Operational Hardening Notes

  • Warm the adjacency cache from inventory-change events, not a fixed timer. Cold subgraph fetches during the first minutes of a major outage are the leading cause of both latency spikes and suppression_unverified results; invalidate on topology change so the graph is hot exactly when a storm hits.
  • Prune at the first healthy node. The walk must stop descending the moment it leaves the alarmed set; this is what bounds the walk to the blast radius and keeps a fabric-wide graph from being traversed on every batch.
  • Cap the walk depth and node budget. A pathological topology or a graph loop should not run the walk unbounded; enforce a maximum visited-node count and, on exceeding it, emit what was found and flag the batch for review rather than blocking.
  • Prefer over-reporting under uncertainty. When topology confidence is low, surface a symptom as an unverified child rather than fully hiding it, so a suppression bug can never make a real fault invisible to the NOC.
  • Watch the suppression ratio as a live signal. A sudden drop in the symptoms-per-root ratio during an outage usually means the cache went cold and the walk is failing to reach real descendants — page on it before the noise reaches tickets.

Frequently Asked Questions

How is graph-based suppression better than grouping alarms by time and region? A time-and-region heuristic groups alarms that merely coincided, which both over-suppresses unrelated faults that happened together and misses causal chains that span regions. A reachability walk proves that a symptom’s device sits behind the root on the actual forwarding path, so it suppresses only genuine downstream consequences and lets an independently-reachable alarm survive as its own root even if it fired in the same instant.

What stops the engine from suppressing a fault that is not actually downstream? Two guards. The walk descends only through devices that are themselves alarmed and prunes at the first healthy node, so it never crosses a live path to reach an unrelated device; and a device reachable by any upstream that is not alarmed keeps a live path and is never classified as a pure symptom. When topology confidence is low the worker degrades to marking symptoms unverified and routing them for review rather than hiding them.

Why classify on a batch of alarms instead of one alarm as it arrives? Root-versus-symptom classification is only meaningful across the set of alarms that fired together, because whether an alarm is a symptom depends on whether its upstream neighbours are also alarmed at the same time. A single alarm in isolation cannot be classified, so the worker consumes a sealed grouping window and walks the whole alarmed set at once.

How does cache warm-up affect suppression accuracy during a major outage? Suppression accuracy depends on the adjacency graph being both current and resident in memory when the storm hits. A cold cache forces mid-walk inventory fetches that time out and leave real descendants unreached, collapsing the suppression ratio and letting symptom alarms leak into tickets. Warming the cache from inventory-change events, rather than a periodic timer, keeps the graph hot exactly when a large outage needs it most.