Idempotency Keys for Retry-Safe Ticket Creation #

A ticket-creation call is a network request, and network requests fail halfway. The ITSM backend accepts the POST, creates the incident, and then the acknowledgement is lost to a timeout — so the automation retries, and a second identical ticket is born. Multiply that by a retry loop, a message replay after a consumer restart, and an active-active pair of pipeline replicas both processing the same correlated fault, and one outage produces a drift of near-duplicate tickets that no responder can reconcile in the middle of an incident. The idempotency key is the single design element that makes creation retry-safe: a deterministic token derived from the fault’s stable identity, presented on every attempt, so the backend and the shared state store both recognise the second, third, and fortieth attempt as the same operation and converge on one ticket. Get the key right and the entire downstream pipeline can retry as aggressively as it needs to; get it wrong and every safeguard above it leaks duplicates.

Schema Alignment and Taxonomy Anchor #

This page is the key-construction detail of Ticket Deduplication, the stage that owns the create/suppress/update decision and the shared-state reconciliation the key drives. That stage supplies the machinery — the atomic claim, the TTL bookkeeping, the failover semantics — and this reference specifies how the key itself is built: which fields of the correlated incident carry stable identity, which must be excluded because they are volatile, and how the TTL window is chosen so a re-firing incident stays deduplicated while a genuinely recurring one is allowed a fresh ticket. The incident the key is built from is the same strict-mode correlated object the rule engine produces, so the identity fields it hashes carry the guarantees established when the causal root was selected during Topology-Aware Correlation.

The core principle is a clean split between two kinds of fields. Identity fields answer “which fault is this?” and must be identical across every re-fire — element, fault class, causal root. Volatile fields answer “when and how did this particular message arrive?” — timestamp, sequence number, collector hostname — and must never touch the key, because they change on every arrival and would fork the identity.

Production Code: Building and Claiming the Key #

The builder below constructs a deterministic key from a whitelisted set of identity fields — a whitelist rather than a blacklist, so a new volatile field added upstream can never silently leak into the key. It then claims the key against a shared store with an atomic conditional write and returns whether this attempt is the creator or a convergent retry. Everything is async and non-blocking; the store’s SET ... NX is the sole coordination primitive.

import hashlib
import json
import logging
import time
from enum import Enum

import redis.asyncio as aioredis
from pydantic import BaseModel, Field, ConfigDict

# hashlib for deterministic digests: https://docs.python.org/3/library/hashlib.html
logger = logging.getLogger("idempotency_key")

# WHITELIST, not blacklist: only these fields may ever enter the key. Anything
# not listed here is volatile by default and excluded, so a new upstream field
# cannot accidentally fork incident identity.
IDENTITY_FIELDS = ("element_id", "fault_class", "root_cause_id")


class ClaimResult(str, Enum):
    CREATED = "CREATED"       # this attempt owns the ticket
    CONVERGED = "CONVERGED"   # a prior attempt already owns it


class Incident(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    element_id: str = Field(min_length=1)
    fault_class: str = Field(min_length=1)
    root_cause_id: str = Field(min_length=1)
    # Volatile fields are still modelled, just never used for the key.
    event_time: float
    sequence_no: int
    collector_host: str


def build_idempotency_key(incident: Incident) -> str:
    """Deterministic across processes, restarts, and replicas. Uses a sorted,
    delimited canonical form so field order can never change the digest."""
    parts = [f"{name}={getattr(incident, name)}" for name in IDENTITY_FIELDS]
    canonical = "|".join(sorted(parts))
    digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:32]
    return f"idem:{digest}"


class IdempotentClaimer:
    def __init__(self, redis_url: str, ttl_seconds: int = 3600):
        self._redis = aioredis.from_url(redis_url, decode_responses=True)
        self._ttl = ttl_seconds

    async def claim(self, incident: Incident) -> tuple[ClaimResult, str]:
        key = build_idempotency_key(incident)
        token = json.dumps({"claimed_at": time.time(),
                            "seq": incident.sequence_no})
        # Atomic first-writer-wins. Only the first attempt among any set of
        # retries, replays, or failover races gets True back.
        won = await self._redis.set(key, token, nx=True, ex=self._ttl)
        if won:
            logger.info("CREATED %s (seq=%s)", key, incident.sequence_no)
            return ClaimResult.CREATED, key
        # Any later attempt lands here and refreshes the sliding TTL so a live,
        # re-firing incident is never allowed to age out mid-outage.
        await self._redis.expire(key, self._ttl)
        logger.debug("CONVERGED %s (seq=%s)", key, incident.sequence_no)
        return ClaimResult.CONVERGED, key

The whitelist is the safety mechanism that matters most. Because IDENTITY_FIELDS is an explicit allow-list, adding collector_host or a new sequence_no variant upstream cannot change the key — the builder simply never reads them. The sorted canonical form removes the last source of nondeterminism, so the same fault produces byte-identical input to the hash on every replica.

Async Processing Hook #

The claimer sits directly in front of the ticket-creation call. Only a CREATED result proceeds to actually POST a ticket; a CONVERGED result short-circuits, because some earlier attempt already owns the ticket. Critically, the same idempotency key is also passed to the ITSM backend as its idempotency token, so even if two attempts both believed they were the creator during a store partition, the backend itself de-duplicates on the key.

import asyncio


async def create_ticket_idempotent(incident: Incident, claimer: IdempotentClaimer,
                                    backend) -> str:
    """Front every creation with a claim. Retry-safe: the key is stable, so
    every retry converges rather than duplicating."""
    result, key = await claimer.claim(incident)
    if result is ClaimResult.CONVERGED:
        # Someone already created it; nothing to do, no duplicate POST.
        return key

    # Pass the SAME key to the backend as its idempotency token, so the backend
    # is a second, independent line of defence against a partition-time race.
    for attempt in range(1, 5):
        try:
            await backend.create_incident(idempotency_key=key,
                                          incident=incident.model_dump(),
                                          timeout=5.0)
            return key
        except (asyncio.TimeoutError, ConnectionError) as exc:
            # The retry reuses the SAME key, so a lost acknowledgement after a
            # successful create cannot spawn a duplicate on the next attempt.
            logger.warning("create retry %d for %s: %s", attempt, key, exc)
            await asyncio.sleep(min(0.5 * 2 ** (attempt - 1), 8.0))
    # Exhausted retries: hand to the dead-letter path, key intact for replay.
    logger.error("create exhausted for %s; dead-lettering", key)
    await backend.dead_letter(key, incident.model_dump())
    return key

The two-layer defence is deliberate. The shared-state claim stops the common case — retries and replicas racing — cheaply and early, while the backend idempotency token catches the rare case where the state store itself is partitioned and two attempts both claim. Because every retry reuses the same key, the classic failure of “the create succeeded but the acknowledgement was lost” resolves to a convergent no-op instead of a duplicate.

Mitigation and Hardening #

1. Whitelist enforcement. Build keys from an explicit identity allow-list, never by hashing the whole incident. Add a test that feeds the builder two incidents differing only in volatile fields and asserts an identical key; that test is the tripwire that catches a field leak before it reaches production.
2. TTL sizing against re-fire gaps. Set the key TTL longer than the longest expected interval between re-fires of the same incident, and refresh it on every convergent attempt. A window shorter than the re-fire gap lets the key expire mid-incident, and the next arrival opens a duplicate.
3. Backend token as a second line. Always pass the same key to the ITSM backend as its idempotency token. The shared-state claim can be defeated by a store partition; the backend token cannot, so the two together close the window that either leaves open alone.
4. Deterministic canonicalization. Sort and delimit identity fields before hashing so field ordering or serialization differences between replicas cannot produce divergent keys for the same fault. A non-canonical concatenation is a subtle way to fork identity across a heterogeneous fleet.

Operational Hardening Notes #

The cost of a claim is one round trip to the shared store, so the tuning target is keeping that round trip flat under storm load: reuse a pooled async connection, and pipeline the SET NX with the TTL refresh so a convergent attempt is a single round trip rather than two. The hash itself is negligible — a truncated SHA-256 over a short canonical string — so resist the urge to cache keys in process memory; recomputing is cheaper than a cache lookup and avoids a staleness class of bug entirely. Choose the truncation length with collision headroom: thirty-two hex characters is one hundred twenty-eight bits, far beyond the birthday bound for any realistic incident volume, so accidental key collisions between distinct faults are not a practical concern. Finally, keep the TTL a per-element-class parameter rather than a global constant — flap-prone access elements need a shorter window so a legitimately recurring fault earns a fresh ticket, while stable core elements can hold a longer window to absorb long reconnect cycles. Tuned this way, idempotent creation adds well under three milliseconds to the p99 creation path while eliminating the duplicate-ticket class of failure entirely.

Frequently Asked Questions #

What makes an idempotency key deterministic across retries and replicas? #

It is built only from stable identity fields, combined in a sorted canonical form, and hashed. Because the input never includes volatile fields like the timestamp or sequence number, and because sorting removes any dependence on field ordering, the same fault produces byte-identical input to the hash on every attempt and on every replica. That determinism is what lets a retry, a replay, or a failover attempt present the exact same key and converge on one ticket.

Why use a field whitelist instead of just excluding known volatile fields? #

Because a blacklist fails open. If the key is built by excluding named volatile fields, a new field added upstream is included by default and silently leaks into the key, forking incident identity the moment it appears. A whitelist fails closed: only explicitly listed identity fields ever enter the key, so any new upstream field is excluded automatically and cannot corrupt deduplication until someone deliberately adds it to the identity set.

Why pass the same key to the ITSM backend when the shared store already claims it? #

Because the two defences cover different failure modes. The shared-state claim handles the common case cheaply, stopping racing retries and replicas before they POST. But if the state store itself is partitioned, two attempts can both believe they are the creator; passing the same key to the backend as its idempotency token means the backend independently de-duplicates and still opens only one ticket. Together they close the window that either layer leaves open alone.

How does the TTL window decide when a recurring fault earns a new ticket? #

The key carries a TTL that is refreshed on every convergent attempt, so a continuously re-firing incident stays claimed for as long as it keeps arriving. Once the fault resolves and stops re-firing, the key ages out after the TTL and the identity is released. A genuinely new fault on the same element after expiry rebuilds the same key, finds it absent, and correctly claims a fresh ticket, so the window must be set longer than the re-fire gap but shorter than the interval between genuinely distinct faults.

Related #

• Up to the parent stage: Ticket Deduplication
• The stage that consumes the key to create a record: ITSM Ticket Creation
• The stage that trusts the key as incident identity: Escalation Routing
• Where the causal root that anchors the key is selected: Topology-Aware Correlation