Fault Correlation Ticket Routing Automation

Correlating BGP Flaps with Interface Down Events

In carrier-grade IP/MPLS cores and data center interconnect fabrics, a single physical or logical interface degradation routinely triggers a cascading control-plane alarm storm. When an interface drops, the BGP sessions riding that link reset, the RIB recalculates, and downstream peers withdraw routes — each transition surfacing as a discrete syslog message, SNMPv3 trap, or streaming telemetry payload. Without deterministic correlation logic, the NOC sees five, ten, or fifty alarms for one root cause: ticket duplication, misrouted incidents, and an MTTR figure inflated by the time engineers waste reconciling symptoms by hand. The operational mandate here is narrow and specific — bind every BGP flap to the interface-down event that produced it, before an ITSM ticket is ever generated, so one physical fault becomes one actionable, topology-aware incident.

This page works through that exact two-signal binding pattern: the temporal window that aligns the streams, the confidence model that ranks the binding, and the async worker that runs it inline without blocking ingest. A correct binding here typically collapses an 8–15 alarm storm into a single ticket and shaves minutes off MTTA, because L1 transport teams receive a pre-correlated interface_down → bgp_flap payload rather than a queue of orphaned BGP Idle notifications.

Schema Alignment and Taxonomy Anchor

This pattern is the canonical two-signal case of Cross-Source Event Linking, the deterministic binding stage inside the Fault Correlation & Rule Engines pipeline. The linking layer assumes clean, already-normalized input: every event arriving here carries a canonical device_fqdn/device_id, an if_index, a severity, and an epoch timestamp, exactly as the contracts in Event Schema Design require. This page does not re-parse Cisco syslog lines or decode trap OIDs — that work belongs to upstream parsers, and trap fields specifically are standardized by Configuring SNMPv3 Trap Receivers in Python before they reach the correlator.

The taxonomy is deliberately small. There are exactly two event classes in play — an interface_down state change (the candidate root cause) and a bgp_state_change / bgp_notification flap (the candidate symptom) — plus the if_index → neighbor_ip mapping that proves they belong on the same link. Topology enrichment maps if_index to BGP neighbor IPs via LLDP/CDP or an inventory API, eliminating false associations across multi-homed peers; the adjacency-validation machinery this depends on is formalized in Topology-Aware Correlation.

Temporal Alignment and Stream Normalization

Effective correlation requires aligning asynchronous event streams on a unified timeline. Interface-down notifications typically arrive via gNMI state-change subscriptions or SNMP traps, while BGP session resets propagate through BGP FSM transitions or telemetry counters. Transport protocols, broker buffering, and polling intervals all introduce variable latency, so strict equality matching on event_timestamp is consistently unreliable — the two halves of the same fault rarely share an exact millisecond.

Instead, production systems implement a backward-looking sliding window that evaluates events within a configurable temporal tolerance, typically calibrated to 3–15 seconds depending on telemetry sampling rates and negotiated BGP hold timers. The causal direction matters: the interface drops first, then BGP reacts, so the window looks backward from the interface_down event to gather the BGP flaps it caused — never forward. The tuning of those tolerances as the network and its baselines drift is governed by Threshold Tuning Methods, and the same principle applied to latency thresholds is worked through in Dynamic Threshold Adjustment for Latency Alerts.

Diagram: the backward-looking window. The interface_down event at t0 triggers a scan back across the tuned window, binding only the BGP flaps it caused — an earlier flap outside the window stays unbound.

Backward-looking BGP-flap to interface-down correlation windowA left-to-right time axis carries event markers. Furthest right at t0 sits the interface_down trigger event. A shaded backward window extends left from t0; a dashed arrow labelled "scan backward from t0" runs across it. Three bgp_flap events fall inside the window and are bound as symptoms, while a fourth bgp_flap that occurred before the window opened stays outside and unbound. The window seals one correlated payload, carrying root_cause interface_down, down to a single ITSM ticket.BACKWARD WINDOW — Δ TUNED BY BFD / HOLD-TIMERscan backward from t0timebgp_flappre-window · unboundbgp_flapbgp_flapbgp_flapinterface_downt0 · triggerCorrelated payload → ITSMone ticket · root_cause = interface_down

Production Correlation Worker

The following implementation is a stateful, async-driven correlation worker designed for high-throughput Kafka ingestion. It maintains an in-memory state table keyed by (device_id, if_index, neighbor_ip), enforces concurrency safety via asyncio.Lock, and applies a backward-looking temporal window to attach BGP flaps as child symptoms to an interface fault. It is copy-paste runnable and emits a single CorrelatedPayload when — and only when — a flap and a drop share a link inside the window.

import asyncio
import logging
from dataclasses import dataclass, field
from typing import Deque, Dict, List, Optional
from collections import deque
from enum import Enum

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("bgp_iface_correlator")


class EventType(str, Enum):
    INTERFACE_DOWN = "interface_down"
    BGP_STATE_CHANGE = "bgp_state_change"
    BGP_NOTIFICATION = "bgp_notification"


@dataclass
class NetworkEvent:
    event_id: str
    device_id: str
    if_index: int
    neighbor_ip: Optional[str]
    event_type: EventType
    timestamp: float                       # wall-clock epoch seconds
    metadata: Dict = field(default_factory=dict)


@dataclass
class CorrelatedPayload:
    root_cause: str
    symptoms: List[str]
    device_id: str
    if_index: int
    neighbor_ip: Optional[str]
    confidence_score: float
    correlation_window_ms: float
    event_ids: List[str]
    timestamp: float


class SlidingWindowCorrelator:
    def __init__(self, default_window_ms: float = 5000.0, max_buffer_size: int = 1000):
        self.default_window_ms = default_window_ms
        self.max_buffer_size = max_buffer_size
        # State table: (device_id, if_index, neighbor_ip) -> Deque[NetworkEvent]
        self.event_buffers: Dict[tuple, Deque[NetworkEvent]] = {}
        self.locks: Dict[tuple, asyncio.Lock] = {}

    def _get_lock(self, key: tuple) -> asyncio.Lock:
        # One lock per link key keeps unrelated links fully parallel; only the
        # same (device, ifIndex, neighbor) serializes through a critical section.
        if key not in self.locks:
            self.locks[key] = asyncio.Lock()
        return self.locks[key]

    def _prune_buffer(self, buffer: Deque[NetworkEvent], current_ts: float, window_ms: float):
        cutoff = current_ts - (window_ms / 1000.0)
        while buffer and buffer[0].timestamp < cutoff:
            buffer.popleft()

    async def ingest_event(self, event: NetworkEvent) -> Optional[CorrelatedPayload]:
        key = (event.device_id, event.if_index, event.neighbor_ip or "0.0.0.0")
        async with self._get_lock(key):
            buffer = self.event_buffers.setdefault(
                key, deque(maxlen=self.max_buffer_size)
            )
            buffer.append(event)

            if event.event_type != EventType.INTERFACE_DOWN:
                return None  # only an interface drop triggers a backward scan

            # Query the backward-looking window for BGP symptoms it caused.
            window_ms = self._calculate_dynamic_window(event)
            self._prune_buffer(buffer, event.timestamp, window_ms)

            bgp_flaps = [
                e for e in buffer
                if e.event_type in (EventType.BGP_STATE_CHANGE, EventType.BGP_NOTIFICATION)
                and e.timestamp <= event.timestamp
            ]
            if not bgp_flaps:
                return None

            delta_ms = (event.timestamp - bgp_flaps[0].timestamp) * 1000
            confidence = self._calculate_confidence(event, bgp_flaps, delta_ms)
            payload = CorrelatedPayload(
                root_cause="interface_down",
                symptoms=["bgp_flap"],
                device_id=event.device_id,
                if_index=event.if_index,
                neighbor_ip=event.neighbor_ip,
                confidence_score=confidence,
                correlation_window_ms=delta_ms,
                event_ids=[event.event_id] + [f.event_id for f in bgp_flaps],
                timestamp=event.timestamp,
            )
            logger.info("correlated device=%s ifIndex=%s conf=%.2f",
                        event.device_id, event.if_index, confidence)
            return payload

    def _calculate_dynamic_window(self, event: NetworkEvent) -> float:
        # Expand the window when BFD is absent or the hold-timer is high, since
        # BGP then reacts slowly and the causal gap widens.
        bfd_active = event.metadata.get("bfd_state") == "up"
        hold_timer = event.metadata.get("bgp_hold_timer_sec", 180)
        if bfd_active:
            return 500.0      # tight window for sub-second BFD detection
        if hold_timer <= 30:
            return 15000.0
        return 30000.0        # fallback for legacy hold timers

    def _calculate_confidence(self, iface_event: NetworkEvent,
                              bgp_events: List[NetworkEvent], delta_ms: float) -> float:
        base = 0.75
        if delta_ms < 1000.0:
            base += 0.15
        if iface_event.metadata.get("bfd_state") == "up":
            base += 0.10
        return min(base, 1.0)

Async Ingestion Hook

The correlator is built to sit inline on the hot path without ever blocking the collectors feeding it. A consumer task drains an asyncio.Queue populated by the ingestion tier, hands each event to ingest_event, and forwards any sealed payload to the routing stage. Because every per-link critical section is short and non-blocking, sustained flap storms on one chassis cannot stall correlation for the rest of the fabric.

async def run_worker(queue: "asyncio.Queue[NetworkEvent]",
                     correlator: SlidingWindowCorrelator,
                     emit: "asyncio.Queue[CorrelatedPayload]") -> None:
    while True:
        event = await queue.get()
        try:
            payload = await correlator.ingest_event(event)
            if payload and payload.confidence_score >= 0.80:
                await emit.put(payload)   # backpressure: blocks if router is behind
        except Exception:
            logger.exception("correlation_failed event=%s", event.event_id)
        finally:
            queue.task_done()

When sustained ingest exceeds correlation throughput, the bounded emit queue provides the natural backpressure signal described in Async Batch Processing — the worker drains in bounded micro-batches rather than collapsing under head-of-line blocking. The same non-blocking ingest discipline applied to raw SNMP collection is detailed in Implementing asyncio for High-Volume SNMP.

Protocol-Aware Window Tuning

Static correlation windows fail in heterogeneous routing environments. The engine must explicitly account for BGP graceful restart capabilities (RFC 4724) and BFD session states. When BFD is active, the interface-down event typically precedes the BGP reset by less than 200 ms, so a tight window is correct and a wide one only invites false bindings. When BFD is absent, BGP hold-timer expiry introduces seconds of delay, and the window must stretch to match. The _calculate_dynamic_window method above expands the tolerance based on the neighbor’s advertised hold timer and BFD state captured at event time.

Threshold tuning should be applied per device profile rather than globally. For spine-leaf fabrics with aggressive BGP timers (e.g. hold-time 3, keepalive 1), a 1–2 second window suffices. For legacy PE routers with hold-time 180, the engine must tolerate up to 30 seconds of drift. Implementing adaptive hysteresis prevents a flapping interface from generating duplicate correlation payloads during transient link oscillations.

Severity Scoring and Confidence

Raw event correlation without scoring leads to alert fatigue. A deterministic confidence model evaluates three dimensions:

  1. Temporal proximity — a sub-1s delta yields ~0.95 confidence; a >5s delta drops toward 0.60, reflecting the weaker causal link.
  2. Topology validation — a confirmed neighbor_ip ↔ if_index mapping via LLDP adds +0.10 weight, because it proves the two signals share a physical link.
  3. Protocol state — BFD Down plus BGP Idle together confirm physical-and-logical causality, the strongest possible evidence.

The worker computes a provisional confidence to make routing deterministic, but the authoritative decay functions, source-trust weighting, and SLA-aligned calibration live in Severity Scoring Algorithms — specifically Implementing Weighted Severity Scoring, which the routing tier consults before a ticket is opened.

Mitigation and Hardening

The correlator degrades gracefully rather than dropping events silently. Concrete failure paths and their mitigations:

  1. Dead-letter isolation. An event that fails schema validation or references an unknown if_index/neighbor_ip is routed to a dead-letter queue with the rejection reason attached, never discarded. The DLQ is replayable once the upstream schema or inventory gap is fixed.
  2. False-positive flood control. Maintain a suppression cache keyed by device_id:if_index with a TTL matching the interface flap-dampening threshold (e.g. 60 s). A re-arming link then produces one correlated payload per dampening cycle, not one per oscillation.
  3. Missing topology, not missing binding. If the if_index → neighbor_ip lookup is unavailable, fall back to keying on (device_id, if_index) alone and stamp the payload topology_unverified so the NOC reviews it — preserve the binding rather than dropping the incident.
  4. Idempotent sealing. Derive a deterministic correlation key from device_id, if_index, and the window epoch so a worker restart or active-active failover never emits the same incident twice.
  5. Confidence gating, not silent drops. Payloads below the routing threshold (here 0.80) are logged and surfaced for review rather than discarded, so a borderline binding never vanishes without a trace.

Operational Hardening Notes

Performance and accuracy tuning specific to this pattern:

  • Bound every buffer. The deque(maxlen=...) cap means a single chatty link cannot exhaust memory under a storm; excess events evict by age and surface in eviction-rate metrics. Keep max_buffer_size just above the expected flap count per window.
  • Cache the EventType enum lookups. In a tight ingest loop, repeated string-to-enum coercion is measurable; parse the event type once at the ingestion boundary and pass the enum through, never the raw string.
  • Warm the topology cache. Cold if_index → neighbor_ip lookups after a maintenance window are a leading cause of topology_unverified payloads; invalidate the cache on inventory-change events rather than on a fixed timer alone.
  • Watch p99, not the mean. Alert when p99 correlation latency exceeds ~200 ms — that threshold typically signals lock contention on a hot link key or topology cache misses, well before SLA acknowledgment clauses are at risk.
  • Per-key locks, not a global lock. Locking per (device_id, if_index, neighbor_ip) keeps unrelated links fully parallel; a single global lock would serialize the whole fabric through one flapping chassis.

When a binding is finally sealed, the engine routes one consolidated payload — a BGP peer flap coinciding with a physical interface error becomes a single ticket for the IP/MPLS transport team instead of duplicate routing and transport alarms — normalized for ITSM ingestion (ServiceNow, Jira Service Management, or PagerDuty):

{
  "incident_type": "network_fault",
  "root_cause": "interface_down",
  "symptoms": ["bgp_flap"],
  "device_id": "PE-01-DC2",
  "if_index": 42,
  "neighbor_ip": "10.0.0.2",
  "confidence_score": 0.95,
  "correlation_window_ms": 142.0,
  "routing_metadata": {
    "tier": "L1_PHYSICAL",
    "auto_assign_group": "noc_transport",
    "suppress_child_tickets": true,
    "runbook_url": "https://wiki.internal/runbooks/bgp-flap-interface-down"
  }
}

Ticket-routing automation consumes the routing_metadata block to bypass manual triage queues, and suppress_child_tickets prevents the symptomatic BGP notifications from ever opening their own incidents. AI-driven root-cause pipelines can enrich this payload further by cross-referencing historical incident databases, but the deterministic backward-window engine remains the primary gatekeeper for production-grade fault isolation.

Frequently Asked Questions

Why scan backward from the interface event instead of forward from the BGP flap? Causality runs one way: the interface drops first, then BGP reacts to the lost link. Triggering the backward scan on the interface_down event lets you gather every BGP flap it produced in a single pass. Triggering on the BGP flap would force you to wait for a future interface event that may never arrive, adding latency and missed bindings.

How wide should the correlation window be? Set it by BFD and BGP hold-timer state, not a fixed constant. With BFD active the gap is sub-200 ms, so a ~500 ms window is right; with a 30 s hold timer the engine needs up to 15 s, and legacy 180 s hold timers need up to 30 s. A window that is too tight misses slow BGP reactions; too wide and unrelated flaps bind to the wrong drop.

What stops a flapping interface from generating duplicate correlated tickets? A suppression cache keyed by device_id:if_index with a TTL matching the flap-dampening threshold (e.g. 60 s). Within that TTL the link emits one correlated payload per dampening cycle rather than one per oscillation, and suppress_child_tickets in the routing metadata stops the symptomatic BGP alarms from opening their own incidents.

What happens when the if_index → neighbor_ip topology lookup is unavailable? The worker falls back to keying on (device_id, if_index) alone, seals the binding, and stamps it topology_unverified for manual NOC review. The incident is never dropped — routing precision degrades, but MTTA is preserved until the inventory source recovers.