Ticket Deduplication: Idempotent Incident Creation Across Retries and Failover #

A correlated fault does not fire once. A transport ring going down emits the same root incident every time a collector reconnects, every time a retry loop wakes, and every time an active-active pair of pipeline replicas both decide the incident is theirs to raise. Without a deduplication guarantee, each of those re-fires becomes a separate ITSM ticket, and a single outage that should produce one actionable work item instead spawns forty — burying the on-call engineer, corrupting MTTR statistics, and training the NOC to ignore its own ticket queue. Ticket deduplication is the stage inside Ticket Routing & ITSM Automation that guarantees incident identity is stable and singular: no matter how many times the same correlated fault arrives, across how many retries, replays, or failovers, exactly one ticket is ever opened for it.

This stage sits between correlation and ticket creation. It consumes an enriched, correlated incident and emits a decision — create, suppress, or update — along with the deterministic key that the downstream creation and escalation stages use as their identity. It is the load-bearing guarantee that lets everything downstream retry aggressively without fear: because identity is fixed here, a retried ServiceNow POST or a re-fired PagerDuty trigger converges on the one ticket rather than multiplying it.

Operational Intent and Boundary #

What enters this stage is a correlated incident carrying its constituent fault identity: the affected element, the fault class, the correlation window, and the causal root the rule engine selected. What exits is that same incident stamped with a deterministic dedup_key and one of three decisions — CREATE for a genuinely new incident, SUPPRESS for a re-fire of a live one, or UPDATE for a re-fire that carries new material state such as a severity change. What is explicitly excluded is severity scoring and root-cause selection, which belong to the correlation tier, and the construction of the vendor ticket payload, which belongs to the sibling ITSM Ticket Creation stage. This stage also does not decide who is paged or when — that is the job of Escalation Routing, which trusts the dedup_key this stage produces as the incident’s permanent identity.

Keeping the boundary sharp is what makes the guarantee hold under failover. Because deduplication reconciles against a shared state store rather than per-replica memory, two pipeline replicas that both receive the same correlated fault resolve to the same key and the same decision — one creates, the other observes the create and suppresses. If identity lived in process memory, an active-active deployment would double every ticket the moment both replicas were healthy. The detailed construction of the keys and TTL windows this stage depends on is covered in Idempotency Keys for Retry-Safe Ticket Creation.

Pipeline Architecture #

The internal flow is a fixed sequence: deterministic key construction → shared-state lookup → decision (create / suppress / update) → idempotent commit → TTL bookkeeping. A correlated incident first has its dedup_key derived deterministically from its stable identity fields. That key is looked up in the shared state store. On a miss, the incident is new: the stage attempts an atomic first-writer-wins claim on the key, and only the winner emits CREATE. On a hit, the stage compares the incoming material state against the stored record — an unchanged re-fire emits SUPPRESS, while a re-fire carrying a new severity or a cleared flag emits UPDATE. Every path refreshes the key’s TTL so a long-lived incident stays deduplicated for as long as it keeps re-firing, and a genuinely resolved incident ages out cleanly.

Production-Ready Async Implementation #

The deduplicator below reconciles every correlated incident against a shared, low-latency key-value store (a Redis-style store), using an atomic first-writer-wins claim so that concurrent replicas cannot both create. The hot path is fully asynchronous; the only atomicity primitive is the store’s own conditional write (SET ... NX), which does the cross-replica coordination that in-process locks cannot. State is a compact record — the decision fingerprint plus a TTL — so the memory cost per live incident stays small even during a storm.

import asyncio
import hashlib
import json
import logging
import time
from enum import Enum

import redis.asyncio as aioredis
from pydantic import BaseModel, Field, field_validator, ConfigDict

# asyncio + hashlib references:
# https://docs.python.org/3/library/asyncio.html
# https://docs.python.org/3/library/hashlib.html
logger = logging.getLogger("ticket_dedup")


class Decision(str, Enum):
    CREATE = "CREATE"
    SUPPRESS = "SUPPRESS"
    UPDATE = "UPDATE"


class CorrelatedIncident(BaseModel):
    """Enriched incident handed in from the correlation tier. The identity
    fields below are the ONLY inputs to the dedup key — volatile fields such
    as event_time are deliberately excluded so a re-fire hashes identically."""
    model_config = ConfigDict(strict=True, extra="forbid")

    element_id: str = Field(min_length=1)     # affected network element
    fault_class: str = Field(min_length=1)    # canonical fault code
    root_cause_id: str = Field(min_length=1)  # causal root the engine selected
    severity: str
    cleared: bool = False

    @field_validator("severity")
    @classmethod
    def known_severity(cls, v: str) -> str:
        if v not in {"CRITICAL", "MAJOR", "MINOR", "INFO"}:
            raise ValueError(f"unknown severity: {v}")
        return v

    def dedup_key(self) -> str:
        # Deterministic across processes and restarts: same identity -> same
        # key on every replica, so failover cannot fork the incident.
        identity = f"{self.element_id}|{self.fault_class}|{self.root_cause_id}"
        digest = hashlib.sha256(identity.encode("utf-8")).hexdigest()[:32]
        return f"dedup:{digest}"

    def material_fingerprint(self) -> str:
        # Only the state worth updating a ticket for. A re-fire with the same
        # fingerprint is pure noise; a different one is a real UPDATE.
        return f"{self.severity}|{int(self.cleared)}"


class TicketDeduplicator:
    def __init__(self, redis_url: str, ttl_seconds: int = 3600):
        self._redis = aioredis.from_url(redis_url, decode_responses=True)
        self._ttl = ttl_seconds

    async def decide(self, incident: CorrelatedIncident) -> tuple[Decision, str]:
        key = incident.dedup_key()
        fingerprint = incident.material_fingerprint()
        record = json.dumps({"fp": fingerprint, "ts": time.time()})

        # First-writer-wins claim. NX makes the SET succeed only if the key is
        # absent, so exactly one replica among any racing set wins the CREATE.
        won = await self._redis.set(key, record, nx=True, ex=self._ttl)
        if won:
            logger.info("CREATE %s", key)
            return Decision.CREATE, key

        # Key already exists: this is a re-fire. Refresh the TTL so a live,
        # re-firing incident never ages out from under us.
        stored = await self._redis.get(key)
        await self._redis.expire(key, self._ttl)
        if stored is None:
            # Rare race: key expired between SET NX and GET. Re-claim it.
            await self._redis.set(key, record, ex=self._ttl)
            logger.info("CREATE %s (post-expiry re-claim)", key)
            return Decision.CREATE, key

        prior = json.loads(stored)
        if prior.get("fp") == fingerprint:
            logger.debug("SUPPRESS %s (unchanged re-fire)", key)
            return Decision.SUPPRESS, key

        # Material state changed (e.g. MAJOR -> CRITICAL, or cleared). Persist
        # the new fingerprint and signal an UPDATE to the existing ticket.
        await self._redis.set(key, record, ex=self._ttl)
        logger.info("UPDATE %s (%s -> %s)", key, prior.get("fp"), fingerprint)
        return Decision.UPDATE, key

    async def close(self, key: str) -> None:
        # Called when an incident resolves: drop the key so a genuinely new
        # future fault on the same element opens a fresh ticket.
        await self._redis.delete(key)

The design leans entirely on the store’s atomic SET ... NX for cross-replica safety — that single conditional write is what makes create idempotent under active-active failover, where two replicas race to claim the same key and exactly one wins. The decision this stage returns is then consumed by ticket creation: a CREATE becomes a new incident payload, a SUPPRESS is dropped, and an UPDATE becomes a patch against the existing record. The key-construction rules — which identity fields to include and, critically, which volatile fields to exclude — are developed fully in Idempotency Keys for Retry-Safe Ticket Creation.

Schema and Identity Validation #

Deduplication is only correct if the identity it hashes is trustworthy, so the incoming incident is a strict-mode Pydantic V2 model that rejects a missing element_id, an empty fault_class, or an unknown severity at the boundary. The most dangerous failure here is a volatile field leaking into the key: if event_time or a per-message sequence number were included in the identity hash, every re-fire would produce a different key and deduplication would silently collapse into no-op, opening a fresh ticket for every arrival. The dedup_key method therefore hashes only the three stable identity fields — element, fault class, and causal root — and the material_fingerprint deliberately carries the small set of fields worth an update, nothing more.

The second validation concern is fingerprint honesty. If the material fingerprint omitted a field that operators care about, a real severity escalation would be suppressed as noise; if it included a noisy field, every re-fire would masquerade as an update and re-touch the ticket. Keeping the fingerprint to severity and cleared-state is the calibrated middle: those are exactly the transitions that change what a responder should do, and nothing else re-touches the ticket. This is the same boundary discipline the ingestion tier applies when it quarantines an event that cannot be mapped to a known element rather than admitting it unbounded.

Configuration and Tuning Parameters #

The behaviour of this stage is governed by a handful of parameters, and the TTL window is the most consequential of them. The table gives starting values with the reasoning that should drive your own numbers.

Three tuning rules matter most. First, set the dedup TTL longer than the longest gap between re-fires of the same incident — if a collector reconnect cycle can be twenty minutes, a ten-minute window will let the incident age out and re-open a duplicate. Second, shorten the window for flap-prone tiers, where the same element legitimately produces a new fault hours apart and a long window would wrongly suppress it. Third, never widen the fingerprint to chase completeness; every field you add is a field whose churn re-touches the ticket, and the point of this stage is to reduce ticket noise, not manufacture it. Reload TTLs by swapping the configuration reference atomically so an in-flight decision never reads a half-updated value.

Debugging Workflow and Observability #

Deduplication failures are quiet until they are catastrophic — either a flood of duplicate tickets or, worse, a suppressed incident that should have paged. Work this checklist when dedup behaviour regresses:

1. Decision-mix monitoring — emit dedup.decision counters tagged CREATE, SUPPRESS, UPDATE. A healthy storm shows one CREATE per genuine incident and a large SUPPRESS tail; a rising CREATE rate during a known single outage means keys are diverging and a volatile field has leaked into the identity hash.
2. Key-cardinality tracking — gauge the number of distinct live dedup keys against the number of open tickets. They should track one-to-one; a growing gap means the store and the ITSM backend have drifted, usually because a close was missed on resolution.
3. Claim-race auditing — count SET NX failures that resolve to SUPPRESS from a lost race versus from an unchanged re-fire. A spike in lost-race suppressions is normal during active-active failover and confirms the atomic claim is doing its job; its absence during failover means both replicas may be writing to different stores.
4. TTL-expiry sampling — log when a key expires while its incident is believed open. Frequent premature expiries mean the TTL is shorter than the re-fire gap and duplicates are imminent; extend the window for that element class.
5. Fingerprint-churn analysis — track how often UPDATE fires per incident. An incident updating dozens of times is a sign the fingerprint includes a noisy field; pull the offending field out of material_fingerprint.

Keep all instrumentation off the hot path — atomic counters and sampling rather than a lock inside decide. The operating envelope to defend is a p99 decision under 3 ms and a state-store round trip under 2 ms, so the deduplicator never becomes the bottleneck between correlation and ticket creation.

Failure Modes and Mitigation #

The deduplicator must fail safe, and “safe” here has a specific meaning: under uncertainty, prefer a single duplicate ticket over a silently suppressed real incident. A duplicate is annoying; a suppressed outage is a missed SLA.

When the shared store is unreachable, the stage fails open rather than closed: it emits a provisional CREATE and records the key locally so that, on recovery, duplicates raised during the outage can be reconciled and merged. This preserves the inviolable guarantee that a real incident is never silently dropped, at the acceptable cost of a small, reconcilable duplicate rate during a store outage. By hashing only stable identity, coordinating creates through an atomic claim, and bounding every key with a refreshed TTL, deduplication becomes the guarantee that lets ITSM Ticket Creation and Escalation Routing retry and fail over freely — one fault, one key, one ticket.

Frequently Asked Questions #

Why hash only identity fields into the dedup key instead of the whole incident? #

Because a dedup key must be identical across every re-fire of the same incident, and volatile fields like the event timestamp or a per-message sequence number change on every arrival. If any of those leaked into the key, each re-fire would hash differently, deduplication would silently become a no-op, and a single outage would open a fresh ticket for every message. Hashing only the stable element, fault class, and causal root guarantees the same fault always resolves to the same key.

How does the stage stop two active-active replicas from both creating a ticket? #

It uses the shared store’s atomic first-writer-wins conditional write, a SET that succeeds only if the key is absent. When two replicas race to claim the same key, exactly one write succeeds and emits CREATE; the other sees the key already present and emits SUPPRESS. Because the coordination happens in the shared store rather than in per-replica memory, failover and active-active operation cannot fork the incident into duplicate tickets.

What happens when the shared state store is unreachable? #

The stage fails open rather than closed. It emits a provisional CREATE and records the key locally, so a real incident is never silently suppressed during a store outage. When the store recovers, the provisional creates are reconciled and any duplicates raised during the gap are merged. The design deliberately prefers a small, reconcilable duplicate rate over the risk of dropping a genuine outage.

How does the TTL window decide when a recurring fault gets a new ticket? #

The dedup key carries a TTL that is refreshed on every re-fire, forming a sliding window, so a continuously re-firing incident stays deduplicated for as long as it keeps arriving. Once the fault is resolved and stops re-firing, the key ages out after the TTL, releasing the identity. A genuinely new fault on the same element that occurs after expiry therefore hashes to the same key but finds it absent, and correctly opens a fresh ticket.

Related #

• Up to the parent reference: Ticket Routing & ITSM Automation
• Key construction and TTL detail: Idempotency Keys for Retry-Safe Ticket Creation
• Sibling stage that consumes the decision: ITSM Ticket Creation
• Sibling stage that trusts the dedup key as identity: Escalation Routing
• Upstream root-cause selection this stage trusts: Topology-Aware Correlation