Correlating Optical Degradation with BGP Route Withdrawals #

An optical fault rarely announces itself as an outage. Long before a wavelength goes dark, the receiver’s pre-forward-error-correction bit-error rate climbs, the optical signal-to-noise ratio erodes, and the coherent line card logs a steady rise in corrected errors. On the control plane, the first visible symptom is entirely different: BGP neighbors ride that degrading link, and as frames start to drop, sessions miss keepalives, hold timers expire, and peers withdraw prefixes. To a NOC watching only the IP layer, this reads as a routing problem — a flurry of bgp_withdrawal events with no obvious cause — and the incident lands on the wrong team. The transport team, meanwhile, has an optical alarm sitting in a separate console that nobody has connected to the routing churn.

The operational goal here is to bind those two layers into one incident before a ticket is opened, so that a rising optical bit-error rate and the BGP withdrawals it caused become a single, transport-attributed incident that names the optical root cause. Done correctly, this collapses a scatter of routing-team tickets into one accurately-routed transport ticket, and it does so early — while the degradation is still corrected errors, hours before the wavelength drops — turning a hard outage into a proactive maintenance dispatch and cutting both MTTA and MTTR for the physical fault.

Schema Alignment and Taxonomy Anchor #

This is a cross-layer instance of Cross-Source Event Linking, the deterministic binding stage inside the Fault Correlation & Rule Engines pipeline. Where the sibling pattern in Correlating BGP Flaps with Interface-Down Events binds two events on the same IP link, this pattern binds across layers — an optical-transport metric to a routing-control-plane event — which makes the cross-layer key the crux of the whole problem. The binding assumes normalized input: every event carries a canonical device_id, the optical event carries an optical_port and a pre_fec_ber reading, and the BGP event carries a neighbor_ip, exactly as the upstream schema contracts require.

The taxonomy is two classes plus a mapping. An optical_degradation event — a pre-FEC bit-error-rate reading that has crossed its rising-trend threshold — is the candidate root cause; a bgp_withdrawal event is the candidate symptom. The mapping that proves they belong together is the cross-layer adjacency optical_port → circuit_id → bgp_neighbor_ip: the physical port carries a circuit, the circuit carries the IP link, and the IP link carries the BGP session. Resolving and trusting that mapping is the job of Topology-Aware Correlation; this pattern consumes it to reject withdrawals that ride a different, healthy path.

Cross-Layer Keying and Temporal Alignment #

Two things make this correlation harder than a same-layer binding. First, the two signals share no field directly — an optical port number and a BGP neighbor IP have nothing in common until the topology graph maps one to the other through the circuit that connects them. Second, the causal delay is long and variable. Optical degradation is gradual: the bit-error rate crosses its threshold, and only once frame loss becomes severe enough to break keepalives does BGP react — a gap governed by the neighbor’s hold timer, often tens of seconds, not the sub-second gap of a hard interface drop.

The binding therefore uses a forward-looking window anchored on the optical event: degradation is detected first, and the engine watches forward for the BGP withdrawals it predicts will follow within the hold-timer horizon. This is the opposite direction to a hard interface fault, where the drop and the flap are near-simultaneous and the scan runs backward. Sizing the forward window to the neighbor’s advertised hold timer plus a margin is what separates a true optical-caused withdrawal from an unrelated routing change that merely happened to occur nearby.

Diagram: a rising optical bit-error rate crosses its threshold at t0, opening a forward window that binds the BGP withdrawals it causes into one transport incident.

Production Correlation Worker #

The following worker binds optical degradation to downstream BGP withdrawals. It buffers events per cross-layer key, resolves the optical_port → bgp_neighbor_ip mapping through an injected topology resolver, and applies a forward window sized to the neighbor’s hold timer. It emits a single OpticalRootCauseIncident only when a withdrawal both falls inside the window and is proven to ride the degrading circuit. It performs no blocking I/O and is copy-paste runnable.

import asyncio
import logging
from collections import defaultdict, deque
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field, field_validator

logger = logging.getLogger("optical_bgp_correlator")


class EventType(str, Enum):
    OPTICAL_DEGRADATION = "optical_degradation"
    BGP_WITHDRAWAL = "bgp_withdrawal"


class NetworkEvent(BaseModel):
    model_config = ConfigDict(frozen=True)

    event_id: str
    device_id: str
    event_type: EventType
    event_time: float                     # reconciled epoch seconds
    optical_port: str | None = None       # set on optical events
    pre_fec_ber: float | None = None      # set on optical events
    neighbor_ip: str | None = None        # set on BGP events
    hold_timer_s: float = 90.0            # advertised BGP hold timer

    @field_validator("pre_fec_ber")
    @classmethod
    def _ber_range(cls, v: float | None) -> float | None:
        if v is not None and not (0.0 <= v <= 1.0):
            raise ValueError("pre_fec_ber must be a fraction between 0 and 1")
        return v


class OpticalRootCauseIncident(BaseModel):
    root_cause: str = "optical_degradation"
    device_id: str
    optical_port: str
    circuit_id: str
    pre_fec_ber: float
    symptom_neighbor_ips: list[str]
    confidence: float
    forward_window_s: float
    event_ids: list[str]
    event_time: float


class TopologyResolver:
    """Maps an optical port to the circuit and BGP neighbors it carries.

    In production this is backed by an inventory API; the warm in-memory cache
    keeps the resolve call off the hot path. It is async so a cold lookup can
    await the backing store without blocking the event loop.
    """

    def __init__(self, table: dict[tuple[str, str], tuple[str, set[str]]]):
        self._table = table   # (device_id, optical_port) -> (circuit_id, {neighbor_ip})

    async def resolve(self, device_id: str, optical_port: str):
        await asyncio.sleep(0)   # placeholder for a cache-miss inventory fetch
        return self._table.get((device_id, optical_port))


class OpticalBgpCorrelator:
    def __init__(self, resolver: TopologyResolver, ber_threshold: float = 1e-4,
                 max_buffer: int = 2000):
        self.resolver = resolver
        self.ber_threshold = ber_threshold
        # Buffer BGP withdrawals per device so an optical event can scan forward.
        self._bgp: dict[str, deque[NetworkEvent]] = defaultdict(
            lambda: deque(maxlen=max_buffer))
        self._locks: dict[str, asyncio.Lock] = defaultdict(asyncio.Lock)

    async def ingest(self, event: NetworkEvent) -> OpticalRootCauseIncident | None:
        async with self._locks[event.device_id]:
            if event.event_type is EventType.BGP_WITHDRAWAL:
                self._bgp[event.device_id].append(event)
                return None
            # Optical event: only a rising BER over threshold triggers a scan.
            if event.pre_fec_ber is None or event.pre_fec_ber < self.ber_threshold:
                return None
            return await self._bind(event)

    async def _bind(self, optical: NetworkEvent) -> OpticalRootCauseIncident | None:
        topo = await self.resolver.resolve(optical.device_id, optical.optical_port or "")
        if topo is None:
            logger.info("TOPOLOGY_MISS device=%s port=%s",
                        optical.device_id, optical.optical_port)
            return None
        circuit_id, neighbor_ips = topo
        window_s = optical.hold_timer_s * 1.5   # forward horizon: hold timer plus margin
        buffer = self._bgp[optical.device_id]
        bound = [
            e for e in buffer
            if e.neighbor_ip in neighbor_ips                       # rides this circuit
            and optical.event_time <= e.event_time <= optical.event_time + window_s
        ]
        if not bound:
            return None
        confidence = self._confidence(optical, bound, window_s)
        incident = OpticalRootCauseIncident(
            device_id=optical.device_id,
            optical_port=optical.optical_port or "",
            circuit_id=circuit_id,
            pre_fec_ber=optical.pre_fec_ber or 0.0,
            symptom_neighbor_ips=sorted({e.neighbor_ip for e in bound if e.neighbor_ip}),
            confidence=confidence,
            forward_window_s=window_s,
            event_ids=[optical.event_id] + [e.event_id for e in bound],
            event_time=optical.event_time,
        )
        logger.info("optical_root_cause device=%s port=%s circuit=%s conf=%.2f",
                    optical.device_id, optical.optical_port, circuit_id, confidence)
        return incident

    def _confidence(self, optical: NetworkEvent, bound: list[NetworkEvent],
                    window_s: float) -> float:
        base = 0.70
        # A BER an order of magnitude over threshold is strong physical evidence.
        if optical.pre_fec_ber and optical.pre_fec_ber > self.ber_threshold * 10:
            base += 0.15
        # Multiple neighbors on the same circuit withdrawing together strengthens it.
        if len({e.neighbor_ip for e in bound}) > 1:
            base += 0.10
        return min(base, 1.0)

Async Ingestion Hook #

The correlator sits inline on the hot path behind a bounded queue. A consumer drains events from the ingestion tier, hands each to ingest, and forwards any sealed incident to routing. Because BGP withdrawals are buffered as they arrive and the forward scan runs only when an optical event lands, the common case — a withdrawal with no matching optical fault — costs a single append and returns immediately, so routine routing churn never stalls the loop.

async def run_correlator(events: "asyncio.Queue[NetworkEvent]",
                         correlator: OpticalBgpCorrelator,
                         emit: "asyncio.Queue[OpticalRootCauseIncident]") -> None:
    while True:
        event = await events.get()
        try:
            incident = await correlator.ingest(event)
            if incident and incident.confidence >= 0.80:
                await emit.put(incident)   # backpressure: blocks if router is behind
        except Exception:
            logger.exception("correlation_failed event=%s", event.event_id)
        finally:
            events.task_done()

When sustained ingest exceeds correlation throughput, the bounded emit queue provides the natural backpressure signal so the worker drains in bounded micro-batches rather than collapsing under head-of-line blocking. The same discipline underpins the same-layer binding in Correlating BGP Flaps with Interface-Down Events, which this pattern complements: one binds a hard drop to its flaps, the other binds a slow optical decay to the withdrawals it foreshadows.

Mitigation and Hardening #

The worker degrades gracefully rather than dropping events. Concrete failure paths and mitigations:

1. Dead-letter isolation. An optical event missing an optical_port, or a BGP event whose neighbor_ip fails schema validation, routes to a dead-letter queue with the rejection reason attached, never discarded. The queue is replayable once the upstream parser or inventory gap is fixed.
2. Topology miss, not silent drop. If the optical_port → circuit_id → neighbor_ip mapping is unavailable, the worker emits no false binding but records a TOPOLOGY_MISS and holds the optical event for a short grace period so a warming inventory cache can complete the mapping before the incident is abandoned.
3. Forward-window false binding. A withdrawal that falls inside the time window but rides a different, healthy circuit is rejected by the neighbor-set membership test, so temporal coincidence alone can never bind an unrelated routing change to the optical fault.
4. Idempotent sealing. Derive a deterministic incident key from device_id, circuit_id, and the optical event’s threshold-crossing epoch so a worker restart or active-active failover never opens the same transport incident twice.
5. Confidence gating, not silent drops. An incident below the routing threshold is logged and surfaced for review rather than discarded, so a borderline early-degradation binding never vanishes without a trace — important precisely because catching the fault early is the entire value of the pattern.

Operational Hardening Notes #

• Warm the topology cache before it is needed. Cold optical_port → neighbor_ip lookups after a maintenance window are the leading cause of TOPOLOGY_MISS; invalidate on inventory-change events rather than on a fixed timer so the mapping is hot when a degradation fires.
• Anchor the window on the hold timer, with margin. BGP reacts to frame loss only when keepalives fail, so a forward window of roughly 1.5 times the advertised hold timer captures the reaction without stretching so far that unrelated churn leaks in.
• Gate on a rising trend, not a single reading. A lone BER spike can be a measurement artifact; require the pre-FEC bit-error rate to hold above threshold across consecutive samples before treating it as a root-cause trigger, which keeps the false-positive rate low without delaying a genuine decay.
• Buffer BGP per device, bounded. The deque(maxlen=...) cap means a chatty peer cannot exhaust memory under a route-churn storm; size it just above the expected withdrawal count within one hold-timer window.
• Watch p99 resolve latency. Alert when the topology resolve p99 climbs — it is the earliest sign of an inventory backend slowdown that will start producing topology misses and silently degrade binding precision.

Frequently Asked Questions #

Why anchor the window on the optical event and scan forward instead of backward? Causality runs from the physical layer up: the optical bit-error rate degrades first, and BGP reacts only once frame loss breaks keepalives, tens of seconds later. Anchoring on the optical threshold crossing and scanning forward across the hold-timer horizon gathers the withdrawals the degradation caused. Scanning backward from a withdrawal would miss the slow-building optical trend that preceded it and misattribute the cause to the routing layer.

How does the engine avoid binding an unrelated BGP withdrawal that just happened nearby? Temporal proximity alone is never sufficient. The worker resolves the optical port to the specific circuit and the set of BGP neighbors that ride it, then binds only withdrawals whose neighbor IP is in that set. A withdrawal that falls inside the time window but rides a different, healthy circuit fails the membership test and stays unbound, so coincidence in time cannot manufacture a false root cause.

What is the value of correlating before the link actually fails? Optical degradation is gradual, so the pre-forward-error-correction bit-error rate crosses its threshold while errors are still being corrected, often hours before the wavelength drops. Binding the early degradation to the first BGP withdrawals it causes lets the engine open a proactive transport maintenance ticket while the fault is still soft, turning a hard outage into a scheduled repair and cutting mean time to resolve for the physical fault.

What happens when the optical-to-neighbor topology mapping is missing? The worker does not fabricate a binding. It records a topology miss and holds the optical event for a short grace period so a warming inventory cache can complete the mapping, then abandons the binding if the mapping never resolves. The optical event itself is preserved for review rather than dropped, so a genuine degradation is never lost just because inventory was briefly unavailable.

Related #

• Up to the parent stage: Cross-Source Event Linking — the deterministic binding layer this cross-layer pattern lives inside.
• Correlating BGP Flaps with Interface-Down Events — the same-layer sibling binding that this optical pattern complements.
• Topology-Aware Correlation — resolves the optical-port-to-neighbor adjacency this binding depends on.
• Temporal Windowing Strategies — the windowing stage that sizes the forward horizon used here.
• Fault Correlation & Rule Engines — the parent reference for the correlation domain.