Adaptive Rate Limiting During SNMP Storms #

A static rate limiter is tuned for a world that does not exist during an outage. It is sized for steady state — a few traps per second per element — and when a core router loses a linecard and starts emitting linkDown traps for two hundred sub-interfaces at once, that static ceiling either clips the storm so aggressively that the genuine root-cause trap is shed alongside the noise, or it is set so generously that the storm sails straight through and buries the correlation engine. Neither outcome is acceptable. The first delays the ticket for a real outage; the second turns one fault into a NOC console of forty thousand alarms, and both inflate mean time to acknowledge (MTTA) and mean time to resolution (MTTR) at precisely the moment they matter most.

An SNMP trap storm has a signature a static limiter cannot see: the offered rate from one source jumps ten to a hundred times above that source’s own recent baseline within a second or two. Adaptive rate limiting exploits exactly this. Instead of one fixed ceiling for every element, it learns each source’s normal trap rate, tolerates bursts proportional to that baseline, and tightens only for the sources that are actually storming — while a hard priority lane guarantees that a Critical SERVICE_OUTAGE trap is never counted against any quota. The limiter shapes the flood without ever silencing the signal inside it.

Schema alignment and taxonomy anchor #

This page is the storm-adaptive specialization of Rate Limiting Strategies, the admission-control stage of the Ingestion & Parsing Workflows data plane that sits immediately downstream of normalization and immediately upstream of correlation. That stage composes a token bucket with a sliding window into a priority-aware control plane; this reference makes the token bucket’s refill rate a function of each source’s measured baseline instead of a static constant, so the same primitive adapts to a storm rather than being overwhelmed by one. The burst arithmetic this builds on — bucket depth, refill cadence, and the relationship between tokens_per_sec and max_burst — is worked through in Setting Up Token Bucket Rate Limiters.

Every trap reaching the limiter is a normalized event conforming to the canonical contract from Event Schema Design, carrying asset_id, severity, vendor_alarm_code, and event_time. The limiter reads severity to route Critical and Major traps down the bypass lane, and asset_id to key per-source fairness so one storming element can never consume the admission budget of a healthy neighbor. The severity scale it trusts is the shared one defined in Defining Severity Levels for Telecom Faults, so a MAJOR trap means the same thing here as everywhere else in the pipeline.

A baseline-aware adaptive limiter #

The limiter below keeps a rolling estimate of each source’s normal trap rate using an exponentially weighted moving average (EWMA), and scales that source’s token refill from the baseline. A storm — a current rate many times the baseline — tightens the effective ceiling for that source alone, so its low-severity noise is shed while quiet neighbors are untouched. Critical and Major traps bypass every quota before any of this runs.

import asyncio
import time
from dataclasses import dataclass, field
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field


class Severity(str, Enum):
    CRITICAL = "CRITICAL"
    MAJOR = "MAJOR"
    MINOR = "MINOR"
    INFO = "INFO"
    CLEARED = "CLEARED"


class Trap(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    asset_id: str
    severity: Severity
    vendor_alarm_code: str
    event_time: float = Field(gt=0)


@dataclass
class SourceState:
    """Per-source adaptive state: baseline rate, live bucket, and rate window."""
    baseline_rps: float = 1.0        # EWMA of the source's normal trap rate
    tokens: float = 8.0              # current bucket contents
    last_refill: float = field(default_factory=time.monotonic)
    recent: int = 0                  # traps counted in the live rate window
    window_start: float = field(default_factory=time.monotonic)


class AdaptiveLimiter:
    def __init__(
        self,
        storm_ratio: float = 8.0,    # rate/baseline above this = storm
        ewma_alpha: float = 0.2,     # baseline responsiveness
        burst_multiple: float = 4.0, # bucket depth as a multiple of baseline
        rate_window_sec: float = 1.0,
    ) -> None:
        self.storm_ratio = storm_ratio
        self.ewma_alpha = ewma_alpha
        self.burst_multiple = burst_multiple
        self.rate_window_sec = rate_window_sec
        self._sources: dict[str, SourceState] = {}
        self._locks: dict[str, asyncio.Lock] = {}
        self.metrics = {"allowed": 0, "shed": 0, "bypass": 0}

    def _lock_for(self, asset_id: str) -> asyncio.Lock:
        # Per-source locks: a storm on one element never blocks another's admission.
        lock = self._locks.get(asset_id)
        if lock is None:
            lock = self._locks[asset_id] = asyncio.Lock()
        return lock

    def _observe_rate(self, st: SourceState, now: float) -> float:
        """Fold the last window's count into the EWMA baseline; return live rate."""
        elapsed = now - st.window_start
        if elapsed >= self.rate_window_sec:
            live_rps = st.recent / elapsed
            # A storm should NOT poison the baseline, so only let the baseline
            # track downward-or-modest rates; spikes update it slowly.
            if live_rps <= st.baseline_rps * self.storm_ratio:
                st.baseline_rps = (
                    self.ewma_alpha * live_rps
                    + (1 - self.ewma_alpha) * st.baseline_rps
                )
            st.recent = 0
            st.window_start = now
            return live_rps
        return st.recent / max(elapsed, 1e-6)

    async def admit(self, trap: Trap) -> tuple[bool, str]:
        # Hard guarantee: high-severity traps never touch the quota path.
        if trap.severity in (Severity.CRITICAL, Severity.MAJOR):
            self.metrics["bypass"] += 1
            return True, "PRIORITY_BYPASS"

        async with self._lock_for(trap.asset_id):
            now = time.monotonic()
            st = self._sources.setdefault(trap.asset_id, SourceState())
            st.recent += 1
            live_rps = self._observe_rate(st, now)

            # Refill scales with the learned baseline, not a static constant.
            refill_rps = max(st.baseline_rps, 0.5)
            depth = max(refill_rps * self.burst_multiple, 4.0)
            st.tokens = min(st.tokens + (now - st.last_refill) * refill_rps, depth)
            st.last_refill = now

            storming = live_rps > st.baseline_rps * self.storm_ratio
            if storming and trap.severity in (Severity.INFO, Severity.CLEARED):
                # Under a storm, shed the lowest-value traps from THIS source only.
                self.metrics["shed"] += 1
                return False, "SHED_STORM_LOW_SEV"

            if st.tokens >= 1.0:
                st.tokens -= 1.0
                self.metrics["allowed"] += 1
                return True, "ALLOWED"

            self.metrics["shed"] += 1
            return False, "BUCKET_EMPTY"

The subtle part is that the storm must not corrupt the baseline it is measured against. The _observe_rate method folds a window’s rate into the EWMA only when that rate is within the storm threshold; a genuine spike updates the baseline slowly rather than instantly redefining “normal” as the storm rate. This is what stops a limiter from adapting itself into uselessness — after a five-minute storm a naive EWMA would treat the flood as the new baseline and stop shedding entirely.

Async processing hook #

The limiter slots into the async admission stage exactly where the static one does, and per-source locks keep it non-blocking across sources: a storm on one router never delays admission for a healthy neighbor sharing the event loop.

async def admission_stage(
    limiter: AdaptiveLimiter,
    inbound: asyncio.Queue,      # normalized traps from parsing
    correlation: asyncio.Queue,  # admitted traps to the correlation engine
    dlq: asyncio.Queue,          # shed traps, tagged with the reason
) -> None:
    while True:
        raw = await inbound.get()
        try:
            trap = Trap.model_validate(raw)
        except Exception:
            await dlq.put({"raw": raw, "reason": "MALFORMED"})
            inbound.task_done()
            continue

        allowed, reason = await limiter.admit(trap)
        if allowed:
            await correlation.put(trap.model_dump())
        else:
            # Shedding is observable, never silent: the reason rides to the DLQ.
            await dlq.put({"asset_id": trap.asset_id, "reason": reason})
        inbound.task_done()

Because admit only ever holds a per-source lock briefly around in-memory arithmetic and awaits nothing inside it, the stage shapes tens of thousands of traps per second on a single event loop. Every shed decision carries its reason to the dead-letter queue, so a storm leaves an auditable trail rather than a hole in the record.

Mitigation and hardening #

Adaptive limiting adds power and, with it, new ways to be wrong. Each has a concrete containment path:

1. Baseline poisoning. If a sustained storm is allowed to drag the EWMA upward, the limiter eventually accepts the storm rate as normal and stops protecting the engine. Gating the EWMA update on the storm threshold — updating slowly during spikes — keeps the baseline anchored to true steady state, and pinning a floor under refill_rps stops a source that has been quiet for hours from starting with a zero ceiling.
2. Never shed high severity. Critical and Major traps return PRIORITY_BYPASS before any state is touched, the same inviolable guarantee the parent stage enforces. A storm sheds only INFO and CLEARED traps from the storming source, so the root-cause linkDown that triggered the flood is always admitted.
3. Per-source fairness. Keying buckets and locks by asset_id means one storming element cannot exhaust a shared quota and starve every other source — a failure mode a single global limiter is prone to. A newly provisioned element with no policy binds to a conservative default baseline and emits a fallback metric, mirroring the inventory-gap handling that should be cross-referenced with Error Categorization Pipelines.
4. Security-relevant floods. A trap storm can be a symptom of reconnaissance or a spoofed-source flood rather than a genuine fault. Shed traps are dead-lettered with their asset_id and reason so the boundary-enforcement posture in Security Boundary Mapping can review whether a storming source is a failing element or a hostile one, instead of the flood vanishing unlogged.

Operational hardening notes #

Tuning an adaptive limiter is mostly about three coupled dials. The storm_ratio sets how far above baseline a source must go before it is treated as storming: eight times is a sound default for transport elements whose normal trap rate is low and whose storms are dramatic, but flap-prone BGP edges may need a higher ratio so ordinary churn is not misread as a storm. The ewma_alpha sets how fast the baseline tracks reality — a small alpha near 0.2 makes the baseline stable and slow to be fooled, which is exactly what you want, since a jumpy baseline is a baseline a storm can poison. The burst_multiple sets bucket depth as a function of baseline, so it must be large enough to admit a legitimate correlated burst — a single fault lighting up a node’s protected paths — without admitting the whole flood. Keep every hot-path operation O(1) and lock-free across sources: the per-source lock guards only in-memory arithmetic, nothing awaits inside it, and the rate window is a counter reset rather than a stored history, so memory stays bounded no matter how many sources storm at once. Expose allowed, shed, and bypass as counters and the per-source baseline as a gauge; a healthy storm shows bypass and shed both rising on the storming source while quiet sources hold a flat allowed rate. Held to these controls, adaptive rate limiting keeps a p99 admission decision under 5 ms through a multi-domain trap storm, preserves a zero-percent drop rate on high-severity traps, and hands the correlation engine one actionable signal per fault instead of forty thousand.

Frequently Asked Questions #

How is adaptive rate limiting different from a static token bucket? #

A static token bucket uses one fixed refill rate and burst size for every element regardless of conditions, so it is either too tight for a legitimate correlated burst or too loose to contain a storm. An adaptive limiter learns each source’s normal trap rate as a rolling baseline and scales that source’s ceiling from it, tolerating bursts proportional to normal behavior and tightening only for sources whose current rate has jumped far above their own baseline. It shapes the flood per source instead of applying one blunt ceiling to all.

How does the limiter avoid learning a storm as the new normal? #

The baseline is an exponentially weighted moving average that is updated only when the observed rate stays within the storm threshold of the existing baseline. A genuine spike is recognized as a storm and is deliberately not folded into the baseline at full weight, so a long flood cannot drag the average up until the limiter treats the storm rate as normal. This keeps the baseline anchored to true steady state and preserves the limiter’s ability to shed throughout the event.

Are high-severity traps ever shed during a storm? #

No. Critical and Major traps are routed down a priority bypass lane that returns an allow decision before any quota arithmetic runs, so they never consume or depend on a token bucket. During a storm the limiter sheds only Info and Cleared traps, and only from the specific source that is storming, which means the root-cause high-severity trap that triggered the flood is always admitted to correlation.

How does per-source fairness prevent one element from starving others? #

Every source has its own token bucket and its own async lock, both keyed by asset id. Because quota state is tracked per source rather than as a single shared pool, a storming element can only exhaust its own bucket; it cannot consume the admission budget of a healthy neighbor. The per-source lock also guards only brief in-memory arithmetic and awaits nothing, so shaping one source’s storm never blocks admission decisions for any other source on the same event loop.

Related #

• Up to: Rate Limiting Strategies — the admission-control stage this adaptive limiter specializes
• Setting Up Token Bucket Rate Limiters — the burst-control primitive whose refill this makes baseline-aware
• Async Batch Processing — the downstream stage that aggregates admitted traps
• Error Categorization Pipelines — where shed and fallback-policy traps are triaged
• Defining Severity Levels for Telecom Faults — the shared severity scale the bypass lane trusts