Severity Scoring Algorithms for Telecom Fault Correlation #

In carrier-grade network operations, severity scoring algorithms are the deterministic bridge between normalized telemetry and actionable incident prioritization. Unlike legacy enterprise ITSM frameworks that lean on static impact matrices, telecom environments demand dynamic, multi-dimensional evaluation that survives alarm storms and topology cascades. This scoring layer sits inside the broader Fault Correlation & Rule Engines architecture, operating strictly downstream of event normalization and upstream of dispatch routing, so that correlated fault sets are ranked by operational urgency rather than raw alarm volume.

Operational Intent: What Enters, What Exits, What Is Excluded #

The scoring stage consumes already-normalized, already-correlated events. By the time an alarm reaches the scorer it has passed schema validation against the canonical contract defined in Event Schema Design, and it has been grouped into a fault set by upstream linkage logic. What enters is therefore a structured event carrying a base severity vector, topology flags (redundant path, single point of failure), and a short window of historical scores for the same signature.

What exits is a single floating-point score in the range 0.0–10.0, a deterministic tier label (Watch, Low, High, Critical), and a machine-readable breakdown for observability. The tier directly drives dispatch routing, MTTA targets, and customer-notification SLAs.

What is explicitly excluded matters just as much. The scorer does not create tickets, mutate topology state, deduplicate alarms, or decide which downstream symptoms to suppress — those responsibilities belong to adjacent stages. Suppression of cascaded symptoms is owned by Topology-Aware Correlation; multi-protocol symptom alignment is owned by Cross-Source Event Linking; and boundary calibration of the alert thresholds themselves belongs to Threshold Tuning Methods. Keeping the scorer stateless and side-effect free is what makes it safely re-runnable during an active incident lifecycle.

Scoring Pipeline Architecture #

The scoring engine calculates a composite metric by aggregating technical impact, service-degradation signals, and temporal decay. Each normalized event carries a base severity vector, which is then modulated by topology context and historical baseline deviation. To prevent an upstream equipment failure from artificially inflating downstream node scores, the engine resolves hierarchical dependency flags supplied by the topology layer — guaranteeing that a core router failure does not cascade into false-critical scores for every attached access switch.

The pipeline applies a strict, ordered evaluation:

1. Vector initialization — maps raw alarm codes to a base severity index using a configurable lookup table.
2. Contextual modulation — applies topology-derived multipliers: redundancy discounts where a protected path exists, single-point-of-failure escalations where it does not.
3. Temporal and volume decay — suppresses score inflation during alarm storms using an exponential moving average over the signature’s recent history.
4. Clamp and tier assignment — bounds the result and maps it onto a deterministic dispatch tier.

Production-Ready Async Implementation #

Carrier-grade scoring functions must be stateless, idempotent, and non-blocking so a single worker can fan a correlation window across the event loop without stalling ingestion. The implementation below uses Pydantic V2 for strict inbound validation and asyncio to score a window concurrently. It carries no global mutable state, so re-evaluating an event mid-incident never introduces score drift.

from __future__ import annotations

import asyncio
import time
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field, field_validator


class SeverityTier(str, Enum):
    WATCH = "watch"
    LOW = "low"
    HIGH = "high"
    CRITICAL = "critical"


class ScoringConfig(BaseModel):
    """Immutable scoring policy loaded once per worker and shared across tasks."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    impact_weight: float = Field(1.0, ge=0.0)
    redundancy_discount: float = Field(0.5, ge=0.0, le=1.0)
    spof_escalation: float = Field(1.5, ge=1.0)
    ema_alpha: float = Field(0.15, gt=0.0, le=1.0)  # weight of the newest sample
    max_score: float = 10.0
    min_score: float = 0.0
    hysteresis_cycles: int = Field(3, ge=1)


class ScoredEvent(BaseModel):
    """A normalized, already-correlated event entering the scorer."""

    model_config = ConfigDict(extra="forbid")

    event_id: str
    base_severity: float = Field(ge=0.0, le=10.0)
    is_redundant_path: bool = False
    is_spof: bool = False
    history: list[float] = Field(default_factory=list)  # recent scores, same signature

    @field_validator("event_id")
    @classmethod
    def _non_empty(cls, value: str) -> str:
        if not value.strip():
            raise ValueError("event_id must not be empty")
        return value


class SeverityScorer:
    """Stateless, idempotent severity calculator for the correlation hot path."""

    def __init__(self, config: ScoringConfig) -> None:
        self.config = config

    def _tier(self, score: float) -> SeverityTier:
        if score >= 8.5:
            return SeverityTier.CRITICAL
        if score >= 6.5:
            return SeverityTier.HIGH
        if score >= 3.5:
            return SeverityTier.LOW
        return SeverityTier.WATCH

    async def compute(self, event: ScoredEvent) -> dict:
        cfg = self.config

        # 1. Base vector initialization.
        score = event.base_severity * cfg.impact_weight

        # 2. Contextual modulation from topology flags. A protected (redundant)
        #    path discounts the score; an unprotected single point of failure
        #    escalates it. The two are mutually exclusive by construction.
        if event.is_redundant_path:
            score *= cfg.redundancy_discount
        elif event.is_spof:
            score *= cfg.spof_escalation

        modulated = score

        # 3. Temporal smoothing via EMA to damp alarm-storm spikes without
        #    losing a genuine, sustained escalation.
        if event.history:
            ema = event.history[-1]
            score = cfg.ema_alpha * score + (1 - cfg.ema_alpha) * ema

        # 4. Strict clamp so a misconfigured weight can never breach SLA tiers.
        final = max(cfg.min_score, min(cfg.max_score, score))

        return {
            "event_id": event.event_id,
            "base_raw": event.base_severity,
            "modulated": round(modulated, 2),
            "final": round(final, 2),
            "tier": self._tier(final).value,
            "decay_applied": bool(event.history),
            "scored_at": time.time(),
        }


async def score_window(scorer: SeverityScorer, events: list[ScoredEvent]) -> list[dict]:
    """Fan a correlation window across the event loop without blocking ingestion."""
    return await asyncio.gather(*(scorer.compute(event) for event in events))


async def main() -> None:
    config = ScoringConfig(impact_weight=1.2, spof_escalation=1.6)
    scorer = SeverityScorer(config)
    window = [
        ScoredEvent(event_id="bgp-peer-flap-7", base_severity=6.0, is_spof=True),
        ScoredEvent(event_id="if-down-ge0/3", base_severity=5.0, is_redundant_path=True),
    ]
    for result in await score_window(scorer, window):
        print(result)


if __name__ == "__main__":
    asyncio.run(main())

Because ScoringConfig and ScoredEvent reject unknown fields (extra="forbid"), a malformed enrichment upstream fails loudly at validation rather than silently producing a wrong tier. For the weight-matrix calibration that feeds impact_weight and the topology multipliers, see Implementing Weighted Severity Scoring.

Topology and Schema Validation #

A score is only as trustworthy as the flags that modulate it. Two validation boundaries protect the scorer from producing false criticals.

First, schema validation guarantees field presence and type safety before any arithmetic runs. The Pydantic models above enforce that base_severity is within 0.0–10.0 and that event_id is non-empty; the same strict-mode discipline used for inbound protocol payloads is documented in Validating NetFlow Events with Pydantic. An event missing its is_spof resolution must never default to “critical by omission” — it should default to the conservative redundant-path branch or be routed to a dead-letter queue for re-enrichment.

Second, topology validation ensures the is_redundant_path and is_spof flags are mutually consistent with the live network graph. If both flags arrive true, that is a topology classification error, and the scorer must treat it as a defect rather than silently picking a branch. The authoritative adjacency and dependency resolution that sets these flags lives in Topology-Aware Correlation; the scorer consumes its verdict but never re-derives it. This separation is what lets a core-transport SPOF escalate to Critical while the hundreds of downstream access alarms it caused are suppressed before they ever reach the scorer.

Configuration and Tuning Parameters #

Every tunable in ScoringConfig maps to an operational lever with a concrete rationale:

• impact_weight (default 1.0) — global gain on the base vector. Raise it for service-tier customers whose SLA carries financial penalties; keep it at 1.0 for best-effort segments.
• redundancy_discount (default 0.5) — halves the score for protected paths. A faulted link with a hot-standby that absorbed traffic with zero packet loss rarely warrants immediate dispatch; 0.4–0.6 is the practical band.
• spof_escalation (default 1.5) — multiplies the score for unprotected single points of failure. Values above 1.8 tend to push too many High events into Critical, inflating war-room activations; 1.4–1.7 keeps the Critical tier rare and meaningful.
• ema_alpha (default 0.15) — the weight given to the newest sample in the exponential moving average. A low alpha (0.1–0.2) damps alarm-storm spikes, requiring a fault to persist before the score climbs; a high alpha makes scoring reactive but jittery. Tune it against your sampling cadence so a genuine outage still crosses its tier within one MTTA budget.
• hysteresis_cycles (default 3) — the number of consecutive evaluation cycles a score must hold above a tier boundary before the tier state changes. This is the primary defense against threshold thrashing and aligns with the state-persistence philosophy of ITU-T Recommendation X.733 alarm reporting, which emphasizes sustained state over instantaneous spikes.

Scores map onto dispatch tiers with fixed bands, so routing remains deterministic and auditable:

Threshold bands must also account for maintenance windows, known vendor firmware bugs, and seasonal traffic patterns; the closed-loop recalibration of these bands is the subject of Threshold Tuning Methods.

Debugging Workflow and Observability #

Treat the scorer as a deterministic, observable function rather than a black box. The following checklist isolates the most common scoring defects in production:

1. Emit the full breakdown. Log the returned dictionary — base_raw, modulated, final, tier, decay_applied — alongside event_id and the correlation trace_id. Comparing modulated against final instantly reveals whether an unexpected tier came from topology modulation or from EMA lag.
2. Assert idempotency with synthetic payloads. Maintain a fixture suite of normalized events representing known scenarios (optical link degradation, BGP peer flap, power-supply failure). Assert that compute returns identical final and tier across repeated invocations; any divergence points to leaked state.
3. Track score distribution. Export a histogram metric severity_score_distribution and a score_recalculation_latency summary. A p99 recalculation latency above ~5 ms per event usually means a blocking call crept into the hot path.
4. Alert on variance drift. Alert when the variance of final for identical event signatures exceeds ±0.15. Drift at a stable input indicates configuration skew between workers or an upstream normalization regression.
5. Shadow-test new weight matrices. Route ~10% of live traffic through a shadow scorer carrying the candidate ScoringConfig. Compare its tier outcomes against the production baseline and require that the change reduces the false-positive routing rate (target steady-state below ~2%) without suppressing legitimate Critical events.

Failure Modes and Mitigation #

• Malformed enrichment. A validation failure (missing field, out-of-range base_severity, contradictory topology flags) must isolate the event to a dead-letter queue rather than crash the worker or emit a default-critical. The DLQ preserves the raw payload for re-enrichment and replay once the upstream defect is fixed.
• Topology service unavailable. If the flags that drive modulation cannot be resolved, fall back to a conservative path: score with neither discount nor escalation and tag the result degraded=true so downstream routing can apply a wider safety margin. Never assume SPOF on missing data.
• Score-source storms. Under a flood, the EMA already damps per-signature spikes, but the batch fan-out should sit behind a bounded queue. When the queue saturates, shed the lowest base-severity events first (graceful degradation), since a Watch-tier event delayed by seconds costs nothing against SLA.
• Configuration drift between workers. Load ScoringConfig from a single versioned source and stamp the config version into every breakdown. A circuit breaker should halt dispatch and page operations if two workers report different config versions for the same evaluation window, preventing inconsistent tiering from reaching customers.

By keeping severity scoring deterministic, validated, and instrumented, platform teams hold SLA compliance steady while scaling correlation across multi-vendor, multi-domain telecom infrastructures.

Frequently Asked Questions #

Why use an exponential moving average instead of a fixed threshold count? A fixed count treats every breach equally, while an EMA weights recent severity against sustained history. This damps single-cycle alarm-storm spikes yet still lets a genuine, persistent outage climb into its dispatch tier within one MTTA budget — without the score thrashing that a raw threshold produces.

Should the scorer ever create tickets or suppress alarms? No. The scorer is deliberately side-effect free: it emits a score, a tier, and a breakdown. Ticket creation, symptom suppression, and topology resolution are owned by adjacent stages, which keeps the scorer stateless and safely re-runnable mid-incident.

How do redundancy discounts avoid hiding real outages? The discount applies only when the topology layer has confirmed a protected path that absorbed the fault. If that protection is itself impaired, the path is no longer redundant and the single-point-of-failure escalation applies instead, so a double fault surfaces at full severity rather than being discounted away.

What keeps two workers from scoring the same event differently? Configuration is loaded from one versioned source and stamped into every breakdown. A circuit breaker halts dispatch if two workers report different config versions for the same window, so inconsistent tiering can never reach customers.

Related #

• Fault Correlation & Rule Engines — the parent reference for the correlation domain this scoring stage belongs to.
• Implementing Weighted Severity Scoring — weight-matrix calibration and the deterministic scoring vectors that feed this engine.
• Topology-Aware Correlation — resolves the redundancy and single-point-of-failure flags that modulate every score.
• Threshold Tuning Methods — closed-loop recalibration of the alert boundaries that precede scoring.
• Defining Severity Levels for Telecom Faults — the taxonomy anchor that standardizes the tiers a score maps onto.