Fault Correlation Ticket Routing Automation

Sliding vs Tumbling vs Session Windows for SNMP Storms

When a line card resets or a fiber is cut, a carrier device does not emit one trap — it emits hundreds. A single linkDown cascade can drive an SNMP trap receiver from a baseline of tens of traps per second to several thousand in a burst that lasts anywhere from two seconds to two minutes. The windowing model chosen to suppress that storm decides two things at once: how fast the first correlated incident reaches the NOC, and how completely the storm collapses into that one incident instead of leaking duplicates. Pick the wrong model and you either open a ticket per trap — burying the on-call queue and breaching MTTA across the board — or you hold the incident so long that a P1 transport fault sits undispatched while the window waits for stragglers.

This guide is a direct, benchmarked comparison of the three window models under exactly this workload. It quantifies the latency-versus-completeness trade-off each model makes, shows how the session TTL and the allowed-lateness lag change the outcome, and gives a copy-paste async harness that replays a synthetic storm through all three so you can measure the trade-off against your own trap timing rather than trusting a rule of thumb.

Schema Alignment and Taxonomy Anchor

This decision belongs to Temporal Windowing Strategies, the stage inside the Fault Correlation & Rule Engines pipeline that groups events in time before any rule fires. The comparison here assumes clean input: every trap arriving at the windower has already been decoded, mapped to a canonical fault code, and stamped with an event-time value on a single reconciled clock, exactly as the upstream schema and clock contracts require. Windowing does not re-parse a varbind or repair a timestamp — it only decides which traps share a group.

The taxonomy in play is deliberately small. Each event carries a correlation_key — for a trap storm this is typically device_id plus fault_domain so that a chassis-wide reset groups separately from an unrelated optical alarm on the same box — and an event_time. The window model operates purely on those two fields; the causal meaning of a group is decided later by the correlation rules, and adjacency is proven later still by Topology-Aware Correlation.

The Trade-Off in One View

The three models occupy different points on the same latency-completeness curve, and each has a distinct false-positive signature under a storm. Tumbling emits fastest and cheapest but splits any fault that straddles a bucket seam, leaking a duplicate. Sliding never splits a fault but re-evaluates every trap several times and must deduplicate its overlapping output. Session tracks the true storm shape most closely and yields the cleanest single incident, but pays the session gap as pure latency on every emission and risks running unbounded without a hard cap.

Diagram: how the three models score across the axes that matter for storm suppression.

Window model comparison matrix for SNMP storm suppressionA five-row comparison matrix scores three window models. Tumbling: emit latency low, completeness moderate, false-positive resistance moderate, compute cost low, unbounded-growth risk none. Sliding: emit latency moderate, completeness high, false-positive resistance high, compute cost high, unbounded-growth risk none. Session: emit latency high, completeness high, false-positive resistance high, compute cost moderate, unbounded-growth risk present unless a maximum-duration cap is set. Each cell rating is drawn as a filled bar whose length encodes strength, with a written label.TumblingSlidingSessionEmit latencyStorm completenessFalse-positive resist.Compute costUnbounded riskfastmoderateslowmoderatehighhighmoderatehighhighlowhighmoderatenonenoneneeds cap

The matrix reads as a rule of thumb: default to tumbling for steady-state suppression where cost and speed matter and the occasional seam-split duplicate is tolerable; reach for sliding when a split incident is genuinely expensive and you can afford the re-evaluation; choose session when the storm’s length is unpredictable and a single clean incident per outage is worth the added gap latency.

Async Benchmark Harness

The following harness replays a synthetic SNMP trap storm — a short quiet baseline, a sharp burst, and a decaying tail — through all three window models and reports, for each, the emitted-group count, the mean emit latency measured from the last trap in a group to its release, and a duplicate rate. It is fully async, drives event time from the traps themselves, and reuses the same windowing semantics described for the grouping stage.

import asyncio
import statistics
from dataclasses import dataclass, field
from enum import Enum


class WindowKind(str, Enum):
    TUMBLING = "tumbling"
    SLIDING = "sliding"
    SESSION = "session"


@dataclass(frozen=True)
class Trap:
    event_id: str
    key: str
    event_time: float          # epoch seconds the trap fired


@dataclass
class Result:
    kind: WindowKind
    groups: int = 0
    duplicates: int = 0
    emit_latencies: list[float] = field(default_factory=list)

    @property
    def mean_latency_ms(self) -> float:
        return 1000 * statistics.mean(self.emit_latencies) if self.emit_latencies else 0.0


async def synth_storm(base_rate=20, burst_rate=1500, burst_s=6.0) -> list[Trap]:
    """One quiet second, a burst, then a decaying tail — a linkDown cascade."""
    traps, t, i = [], 0.0, 0
    for _ in range(base_rate):                       # baseline
        traps.append(Trap(f"e{i}", "PE1:transport", t)); i += 1; t += 1 / base_rate
    burst_end = t + burst_s
    while t < burst_end:                             # storm burst
        traps.append(Trap(f"e{i}", "PE1:transport", t)); i += 1; t += 1 / burst_rate
    for n in range(40):                              # decaying tail
        t += 0.15 + n * 0.05
        traps.append(Trap(f"e{i}", "PE1:transport", t)); i += 1
    return traps


async def run_model(kind: WindowKind, traps: list[Trap],
                    size_s=5.0, slide_s=1.0, gap_s=8.0, lateness_s=2.0) -> Result:
    res = Result(kind=kind)
    open_windows: list[dict] = []
    max_seen = float("-inf")

    async def seal(w: dict) -> None:
        res.groups += 1
        if w["count"] == 0:
            return
        # Emit latency: how long after the group's last trap it was released.
        res.emit_latencies.append(max(0.0, (max_seen - lateness_s) - w["last"]))

    for trap in traps:
        max_seen = max(max_seen, trap.event_time)
        t = trap.event_time
        if kind is WindowKind.SESSION:
            placed = False
            for w in open_windows:
                if w["start"] <= t <= w["last"] + gap_s:
                    w["last"], w["count"], placed = t, w["count"] + 1, True
                    break
            if not placed:
                open_windows.append({"start": t, "last": t, "count": 1})
        else:
            step = slide_s if kind is WindowKind.SLIDING else size_s
            start = (t // step) * step - (size_s - step)
            hit = False
            while start <= t:
                if start <= t < start + size_s:
                    match = next((w for w in open_windows if w["start"] == start), None)
                    if match is None:
                        open_windows.append({"start": start, "last": t, "count": 1})
                    else:
                        match["last"], match["count"] = t, match["count"] + 1
                        if hit:
                            res.duplicates += 1   # trap counted in an overlapping window
                    hit = True
                start += step
        # Seal any window whose close boundary is now behind the watermark.
        wm = max_seen - lateness_s
        still_open = []
        for w in open_windows:
            seal_at = w["last"] + (gap_s if kind is WindowKind.SESSION else 0.0)
            if seal_at < wm:
                await seal(w)
            else:
                still_open.append(w)
        open_windows = still_open
        await asyncio.sleep(0)   # yield so the harness stays cooperative
    for w in open_windows:       # flush the tail
        await seal(w)
    return res


async def main() -> None:
    traps = await synth_storm()
    results = await asyncio.gather(*(run_model(k, traps) for k in WindowKind))
    print(f"{'model':10} {'groups':>7} {'dup':>6} {'mean_emit_ms':>13}")
    for r in results:
        print(f"{r.kind.value:10} {r.groups:>7} {r.duplicates:>6} {r.mean_latency_ms:>13.1f}")


if __name__ == "__main__":
    asyncio.run(main())

Running this against the synthetic storm produces the shape every operator should expect before committing to a model: tumbling emits the most groups (the seam between buckets splits the burst) with the lowest latency and zero duplicate re-counts; sliding emits a single dominant group but records a high duplicate count that its downstream dedup must absorb; session emits exactly one group for the whole storm at the cost of the full gap_s added to its emit latency. The absolute numbers shift with your burst_rate and timing, which is the point — measure against your own trap capture, not a generic benchmark.

Async Ingestion Hook

In production the model does not run over a captured list — it runs inline on the trap receiver’s hot path. The windower sits behind a bounded asyncio.Queue fed by the SNMP collector, groups traps by event time, and forwards each sealed group to the rule engine through a second bounded queue. Because sealing is driven by the watermark rather than by wall-clock timers, a lull in the trap stream never strands an open group indefinitely once the idle-timeout sweep advances the watermark, and a burst never blocks ingest because the collector simply awaits the bounded queue and inherits natural backpressure.

async def storm_windower(traps: "asyncio.Queue[Trap]",
                         emit: "asyncio.Queue[dict]",
                         kind: WindowKind = WindowKind.SESSION,
                         gap_s: float = 8.0, lateness_s: float = 2.0) -> None:
    open_windows: list[dict] = []
    max_seen = float("-inf")
    while True:
        trap = await traps.get()
        try:
            max_seen = max(max_seen, trap.event_time)
            placed = False
            for w in open_windows:                       # session grouping
                if w["start"] <= trap.event_time <= w["last"] + gap_s:
                    w["last"], w["ids"], placed = trap.event_time, w["ids"] + [trap.event_id], True
                    break
            if not placed:
                open_windows.append({"start": trap.event_time, "last": trap.event_time,
                                     "ids": [trap.event_id]})
            wm, still = max_seen - lateness_s, []
            for w in open_windows:
                if w["last"] + gap_s < wm:
                    await emit.put({"key": trap.key, "count": len(w["ids"]),
                                    "event_ids": w["ids"]})   # blocks if engine is behind
                else:
                    still.append(w)
            open_windows = still
        finally:
            traps.task_done()

Mitigation and Hardening

The model choice is not free of failure paths, and each has a concrete guard:

  1. Tumbling seam duplicates. A fault straddling a bucket boundary emits two partial groups. Mitigate by keying the downstream deduplication on the fault’s stable correlation key with a TTL of at least one window span, so the second partial group is recognized as the same incident and suppressed rather than opening a ticket.
  2. Sliding duplicate explosion. Overlap multiplies emitted groups by the window-to-slide ratio. Cap that ratio at 5 and require a signature-based dedup on emit; route any group whose signature was already emitted to a discard counter, never to the ticket path.
  3. Session runaway. A chatter source with no quiet gap holds a session open forever. Enforce a hard session_max_s cap and a per-key trap-rate limiter so a malformed device cannot exhaust buffer memory before the cap seals the session.
  4. Watermark poisoning. A single trap with a corrupt future timestamp seals every open window early and truncates the storm. Bound the per-trap watermark advance and dead-letter any trap whose event time exceeds the current maximum by more than a sanity threshold, preserving it for offline review rather than trusting it.
  5. Late-tail leakage. The decaying tail of a storm can arrive after its group sealed. Route post-seal traps to a reconciliation path that amends the already-emitted incident rather than opening a new one, and page only if the post_seal_late rate indicates the allowed-lateness lag is too tight.

Operational Hardening Notes

  • Size the session gap above the storm’s inter-trap gap, not below it. During an active linkDown cascade traps arrive sub-millisecond apart, so an 8-second gap comfortably holds the burst together; setting it below the decaying-tail spacing splits one outage into several incidents.
  • Tie the allowed-lateness lag to the receiver’s measured arrival delay. Export the p99 of event-time-to-arrival and set the lag just above it. Every extra second of lag is an extra second of MTTA on every incident, so do not pad it defensively.
  • Prefer session for optical and flap faults, tumbling for counter thresholds. Faults with an unpredictable burst shape suit session windows; periodic threshold breaches on a stable counter suit cheap tumbling buckets that emit predictably.
  • Benchmark with your own capture before rollout. Replay a real captured storm through the harness above and confirm the emitted-group count matches the number of distinct physical faults; a mismatch means size_s or gap_s is splitting or over-merging incidents.
  • Watch duplicate rate as a live metric, not just at design time. A climbing sliding-window duplicate rate signals the dedup ring is undersized for the current overlap factor and duplicates are about to leak into tickets.

Frequently Asked Questions

Which window model is the safe default for SNMP trap storms? Session windows are the safest default for storm suppression because a trap storm has no fixed length — it stretches and decays unpredictably, and a session window tracks that shape and emits one clean incident per outage. Use tumbling instead when emit latency and compute cost dominate and an occasional seam-split duplicate is acceptable, and reserve sliding for cases where a split incident is genuinely expensive.

How does the session TTL affect false positives? The session gap TTL sets how long a quiet period must last before the window seals. Set it too short and a brief lull inside a single storm splits the outage into two incidents, doubling the ticket count; set it too long and two genuinely separate faults on the same key merge into one incident, hiding the second. Size it just above the inter-trap gap seen during an active storm so a real lull, not a momentary pause, is what closes the window.

Why does the sliding window report so many duplicates in the benchmark? Sliding windows overlap by design, so each trap falls into several windows at once and is counted in each. The benchmark reports those repeat counts as duplicates to show the re-evaluation cost the downstream deduplication must absorb. They are not duplicate incidents unless the emit-time dedup fails, which is why a signature guard on emission is mandatory when running a sliding model.

Can I change window models without redeploying the pipeline? Yes, if the model is a field on a versioned window configuration rather than hard-coded. Because all three models consume the same event-time input and emit the same group shape, switching from tumbling to session is a configuration change, and stamping the config version onto every emitted group lets an investigation reconstruct which model was live when an incident formed.