Fault Correlation Ticket Routing Automation

Mitigating DDoS Telemetry Poisoning

A volumetric DDoS flood attacks a fault-correlation platform twice. The first hit is obvious: millions of near-identical flow records per second threaten to exhaust collectors and queues. The second hit is subtler and far more damaging — if those records reach the anomaly engine, they rewrite the statistical baselines that define “normal,” so once the flood subsides the detector has quietly learned that a hundred-fold traffic spike is unremarkable. The next real attack, or the next genuine congestion event, then slips under a baseline the attacker moved. That is telemetry poisoning, and it corrupts detection long after the packets stop. It also inflates mean time to resolution (MTTR) during the flood itself, as the platform buries operators in tens of thousands of duplicate incidents. Mitigating poisoning means shedding volume intelligently at the edge — keeping enough signal to see the attack while denying it the ability to move the baseline.

Schema Alignment and Taxonomy Anchor

This reference is a security-perimeter application of the routing layer described in Security Boundary Mapping, the parent stage that decides which trust zone owns an event before dispatch. That parent stage routes events on the assumption that the volume reaching it is representative; this page owns the narrower job of ensuring a volumetric flood cannot make the stream unrepresentative before correlation ever sees it.

Anchoring the defence to the documented contract keeps it interoperable with the rest of the taxonomy. The reject-and-preserve posture here is the volumetric counterpart of the per-record checks in Detecting Spoofed ASN in BGP Telemetry and Filtering Illegal TCP Flag Combinations at the Ingestion Edge: those filters reject records that are individually impossible, while this one shields the pipeline from records that are individually plausible but collectively poisonous. All three feed the same canonical shape defined in Event Schema Design.

Rate-aware sampling and reputation gating against a floodA high-volume telemetry flood enters a two-stage defence. First a source-reputation gate scores each source and drops or heavily down-samples known-bad sources while passing trusted sources intact. Second a rate-aware sampler admits a representative fraction of the remaining volume, keeping the admitted rate bounded regardless of input rate. The bounded, representative stream reaches the correlation engine and its anomaly baseline. Suppressed volume is counted into an attack-rate metric rather than fed into the baseline, so the flood is measured but cannot poison what the detector considers normal.Telemetry floodmillions/secReputation gatescore per sourcedrop / down-weight badRate-aware sampleradmit bounded raterepresentative fractionCorrelation+ anomaly baselinebounded · representativeSuppressed volume → attack-rate metriccounted, never fed into the baseline

Production Code: Rate-Aware Sampling and Reputation Gating

The defence has two cooperating stages. A source-reputation gate keeps a decaying score per source and admits a shrinking fraction as a source misbehaves, so a flooding source is throttled hard while a trusted source passes intact. A rate-aware sampler then caps the total admitted rate with a token bucket, admitting a representative fraction of whatever survives the gate. Suppressed records are counted into an attack-rate metric, never fed to the baseline. Everything is O(1) per record; the config is enforced with Pydantic V2 (ConfigDict, field_validator classmethods).

import asyncio
import time
from pydantic import BaseModel, ConfigDict, Field, field_validator


class SamplerConfig(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True, extra="forbid")

    admit_rate_per_sec: float = Field(gt=0)   # token bucket refill (admitted/sec)
    burst: float = Field(gt=0)                # bucket capacity
    reputation_floor: float = Field(ge=0, le=1)  # min admit fraction for bad sources
    decay_half_life_s: float = Field(gt=0)    # how fast a bad score recovers

    @field_validator("reputation_floor")
    @classmethod
    def _floor_below_one(cls, v: float) -> float:
        # A floor of 1.0 would disable reputation gating entirely.
        if v >= 1.0:
            raise ValueError("reputation_floor must be < 1.0")
        return v


class ReputationGate:
    """Decaying per-source score in [0, 1]; lower score => admit less."""

    def __init__(self, cfg: SamplerConfig) -> None:
        self._cfg = cfg
        self._score: dict[str, float] = {}
        self._last: dict[str, float] = {}

    def _current(self, src: str, now: float) -> float:
        score = self._score.get(src, 1.0)
        elapsed = now - self._last.get(src, now)
        # Exponential recovery toward 1.0 over the configured half-life.
        recovery = 0.5 ** (elapsed / self._cfg.decay_half_life_s)
        return 1.0 - (1.0 - score) * recovery

    def penalize(self, src: str, now: float, amount: float = 0.2) -> None:
        score = max(0.0, self._current(src, now) - amount)
        self._score[src], self._last[src] = score, now

    def admit_fraction(self, src: str, now: float) -> float:
        score = self._current(src, now)
        self._last[src] = now
        # Never fully mute a source: floor keeps a thread of signal alive.
        return max(self._cfg.reputation_floor, score)


class RateAwareSampler:
    """Token-bucket admission with reputation-scaled sampling."""

    def __init__(self, cfg: SamplerConfig) -> None:
        self._cfg = cfg
        self._gate = ReputationGate(cfg)
        self._tokens = cfg.burst
        self._last = time.monotonic()
        self.suppressed = 0     # exported as the attack-rate metric

    def _refill(self, now: float) -> None:
        self._tokens = min(self._cfg.burst,
                           self._tokens + (now - self._last) * self._cfg.admit_rate_per_sec)
        self._last = now

    def admit(self, src: str, is_anomalous: bool) -> bool:
        now = time.monotonic()
        self._refill(now)
        if is_anomalous:
            self._gate.penalize(src, now)   # flooding sources lose reputation fast
        frac = self._gate.admit_fraction(src, now)
        # Deterministic sampling: hash-free, uses the running token budget.
        if self._tokens >= 1.0 and frac >= self._cfg.reputation_floor:
            # Scale the token cost inversely to reputation: bad sources burn more.
            cost = 1.0 / max(frac, self._cfg.reputation_floor)
            if self._tokens >= cost:
                self._tokens -= cost
                return True
        self.suppressed += 1
        return False

The reputation floor is the key design choice: a flooding source is throttled to a thin trickle but never fully silenced, so the correlation engine keeps just enough samples to recognize the attack while being spared the volume that would move its baseline. The suppressed counter is exported as a metric, turning the shed volume into a measurable attack-rate signal instead of invisible loss.

Async Ingestion Hook

The sampler is pure CPU, so it sits inline in the async pipeline as a coroutine that admits or suppresses each record without blocking. Only admitted records are awaited onto the correlation queue.

import asyncio
import logging

logger = logging.getLogger("ddos_shield")


async def shield_worker(in_q: asyncio.Queue, out_q: asyncio.Queue,
                        sampler: RateAwareSampler) -> None:
    while True:
        rec = await in_q.get()
        src = rec["src_prefix"]
        anomalous = rec.get("rate_flag", False)   # set upstream by a rate detector
        if sampler.admit(src, anomalous):
            await out_q.put(rec)      # bounded rate — correlation stays responsive
        # suppressed records are counted, not forwarded
        in_q.task_done()


async def metrics_loop(sampler: RateAwareSampler, interval_s: float = 5.0) -> None:
    """Publish the shed volume as an attack-rate gauge."""
    prev = 0
    while True:
        await asyncio.sleep(interval_s)
        shed = sampler.suppressed - prev
        prev = sampler.suppressed
        logger.info("ddos suppression", extra={"suppressed_per_interval": shed})

Because admitted volume is capped by the token bucket regardless of input rate, the correlation stage sees a bounded, representative stream even at the peak of a flood — the event loop never drowns, and the same backpressure discipline used across Security Boundary Mapping holds.

Mitigation and Hardening

When volume turns hostile, the system sheds deterministically while preserving signal. These are the concrete failure paths to implement:

  1. Baseline isolation. Suppressed records are never fed to the anomaly baseline — they only increment the attack-rate metric. This is the entire point: the flood is measured and alertable, but it cannot teach the detector that a hundred-fold spike is normal, so detection survives the attack intact.
  2. Reputation floor, never full mute. A flooding source is throttled to the reputation_floor fraction but never silenced, so the engine keeps a thin, continuous sample that lets it recognize and characterize the attack. Muting a source entirely would blind the platform to the very incident it needs to route.
  3. Adaptive suppression, not a fixed cap. The token bucket bounds admitted rate while the decaying reputation score tightens per-source sampling as misbehaviour persists and relaxes it as the source recovers, so a briefly bursty but legitimate source is not penalized like a sustained flooder. Alert when the suppressed-rate metric crosses a threshold — that gauge is the earliest edge-side signal of a volumetric event.
  4. Signal preservation across the edge. Like Detecting Spoofed ASN in BGP Telemetry and Filtering Illegal TCP Flag Combinations at the Ingestion Edge, the shed volume is not discarded blindly; its per-source counts are a security feed that names the flood’s top talkers for downstream mitigation.

Operational Hardening Notes

Keep the per-record cost flat under the worst-case input rate — which, during a flood, is the whole point. The reputation lookup and token-bucket update are both O(1) dictionary and float operations, and the decay is computed lazily on read so idle sources cost nothing. Bound the reputation map by scoping keys to source prefixes rather than individual addresses, and evict entries whose score has fully recovered to 1.0 to cap memory during a wide, spoofed-source flood. Tune admit_rate_per_sec to what the correlation engine can process while still statistically detecting anomalies — often a small fraction of peak input is plenty — and set decay_half_life_s so a source penalized during a flood recovers over minutes, not hours. Shield workers are stateless with respect to event content and horizontally scalable, so a node under load fails over without losing the metric, holding the same sub-2-second SLA ticket-creation budget as the rest of the taxonomy.

Frequently Asked Questions

How does a DDoS flood poison anomaly baselines?

Anomaly detection learns what normal traffic looks like from the stream it observes. If a volumetric flood of plausible-looking records reaches the detector, it rewrites that notion of normal so a hundred-fold spike becomes unremarkable. After the flood ends, the baseline stays shifted, and the next real attack slips under a threshold the attacker effectively moved. Keeping the flood out of the baseline prevents that lasting corruption.

Why sample the flood instead of dropping it entirely?

Because the platform still needs to see the attack to route it. Dropping a source completely blinds the correlation engine to the very incident it must respond to. A reputation floor throttles a flooding source to a thin, continuous trickle, which is enough signal to recognize and characterize the attack while denying it the volume that would move the baseline.

What keeps a briefly bursty but legitimate source from being penalized?

The reputation score decays back toward full trust over a configurable half-life, so a short burst costs a source only a temporary, self-healing penalty. Sustained misbehaviour keeps the score low and the admitted fraction small, while a source that stops flooding recovers within minutes. Adaptive suppression distinguishes a genuine flooder from a legitimate spike this way rather than using a fixed cap.

Where does the suppressed volume go?

Into an attack-rate metric, not into correlation. Every suppressed record increments a counter that is published as a gauge, so the flood is measured, alertable, and attributable to its top source prefixes. That turns shed volume into a security signal instead of silent loss, and the gauge is often the earliest edge-side indication that a volumetric event has begun.