Fault Correlation Ticket Routing Automation

BMC Remedy Ticket Integration for Telecom Fault Routing

Carriers with a long-standing BMC Remedy deployment face an integration problem that neither a modern REST-first platform nor a green-field tool imposes: the incident interface form is strict about entry-form field names and enumerated values, its create call is comparatively slow, and it offers no native idempotency token. Feed it a correlated fault stream naively and three things go wrong at once. A mistyped enumeration value — an Impact string the form does not recognize — is rejected, dropping the incident. A create that commits server-side but times out on the client re-fires on retry and opens a duplicate. And because the form ingests one entry at a time under a bounded worker pool, a storm of correlated faults saturates the interface, backs up the queue, and pushes a P1 transport outage past its acknowledgement target. This reference shows how to map a correlated fault onto the Remedy incident entry form in async Python, dedup with a reconciliation identifier queried before create, and dispatch under backpressure so the interface is never overrun.

Schema alignment and taxonomy anchor

This is the Remedy-specific write path within ITSM Ticket Creation, the serialization stage of the Ticket Routing & ITSM Automation reference pipeline. That parent stage owns the general fault-to-payload contract; this page owns the narrower job of mapping a correlated incident onto the Remedy incident interface entry form and creating it exactly once. The input is the canonical incident object emitted by the correlation engine, carrying the dedup_key, severity, and provenance defined in Event Schema Design, so the Remedy path re-serializes an existing decision rather than recomputing one.

Sharing that input contract makes the Remedy write a sibling of the ServiceNow incident payload path and the Jira issue-creation path. All three consume the same correlated fault and differ only in field naming, transport, and how idempotency is enforced. Remedy sits between the other two: it has no correlation-id auto-dedup like ServiceNow and no label search like Jira, so idempotency is enforced by storing the dedup_key as a reconciliation identifier in a dedicated form field and querying it before every create.

Correlated fault to Remedy entry-form field mappingFour correlated-fault fields map onto four Remedy incident entry-form fields. The summary maps to Description, the impact tier maps to the Impact weight, the urgency tier maps to the Urgency weight, and the stable dedup key maps to a Reconciliation ID field. The Reconciliation ID mapping is emphasized because it is queried before every create to prevent duplicate tickets.Correlated faultRemedy entry formsummaryDescriptionimpact_tierImpact (weight)urgency_tierUrgency (weight)dedup_keyReconciliation IDqueried before create — prevents duplicate tickets

The Pydantic V2 entry-form model

The model maps a correlated fault onto the Remedy incident interface Values object. Impact and urgency are constrained to the form’s enumerated weights, the reconciliation identifier carries the stable dedup_key, and extra="forbid" catches any stray field before it reaches the form, where an unknown key is a hard reject.

from enum import Enum
from typing import Any
from pydantic import BaseModel, Field, ConfigDict, field_validator

class RemedyWeight(str, Enum):
    # Remedy stores impact and urgency as enumerated weight strings.
    W1_EXTENSIVE = "1-Extensive/Widespread"
    W2_SIGNIFICANT = "2-Significant/Large"
    W3_MODERATE = "3-Moderate/Limited"
    W4_MINOR = "4-Minor/Localized"

_TIER_TO_WEIGHT = {
    1: RemedyWeight.W1_EXTENSIVE,
    2: RemedyWeight.W2_SIGNIFICANT,
    3: RemedyWeight.W3_MODERATE,
}

class RemedyIncident(BaseModel):
    # Strict mode + forbid: the entry form rejects unknown/miscast fields,
    # so catch them here rather than after a slow round trip.
    model_config = ConfigDict(strict=True, extra="forbid")

    Description: str = Field(min_length=1, max_length=100)
    Detailed_Decription: str = Field(min_length=1)   # form's own spelling
    Impact: RemedyWeight
    Urgency: RemedyWeight
    Status: str = "New"
    Reported_Source: str = "Systems Management"
    Service_Type: str = "Infrastructure Event"
    # Reconciliation_ID stores the stable dedup key; it is the field we
    # query before creating so a retry never opens a second ticket.
    Reconciliation_ID: str = Field(min_length=1, max_length=128)
    CI_Name: str = Field(min_length=1)

    @field_validator("Reconciliation_ID")
    @classmethod
    def opaque_token(cls, v: str) -> str:
        if v != v.strip() or " " in v:
            raise ValueError("Reconciliation_ID must be a single opaque token")
        return v

    def to_values(self) -> dict[str, Any]:
        """Render the entry-form Values object the interface expects."""
        return {"values": self.model_dump(mode="json")}

def build_remedy_incident(fault: dict[str, Any]) -> RemedyIncident:
    return RemedyIncident(
        Description=fault["summary"][:100],
        Detailed_Decription=fault["detail"],
        Impact=_TIER_TO_WEIGHT[fault["impact_tier"]],
        Urgency=_TIER_TO_WEIGHT[fault["urgency_tier"]],
        Reconciliation_ID=fault["dedup_key"],   # stable across retries
        CI_Name=fault["ci_name"],
    )

Mapping the numeric tier onto the exact enumerated weight string through _TIER_TO_WEIGHT is what keeps the form from rejecting the record — Remedy will not coerce a bare integer into a weight. The Reconciliation_ID is the idempotency anchor, derived only from the stable dedup_key, the same deterministic construction described in Idempotency Keys for Retry-Safe Ticket Creation.

Backpressure-safe async dispatch with reconciliation dedup

The Remedy create call is slow relative to a modern REST endpoint, so the dispatcher must both guard against duplicates and cap how many creates are in flight. The code below queries the reconciliation identifier first, then creates only on a miss, with a bounded semaphore that applies backpressure so a storm never saturates the interface.

import asyncio
import logging

logger = logging.getLogger("remedy_dispatch")
_RETRYABLE = {429, 500, 502, 503, 504}

# Bounded concurrency: the entry form is the scarce resource. A semaphore
# here is the backpressure valve that stops a storm overrunning the interface.
_dispatch_gate = asyncio.Semaphore(8)

async def find_by_reconciliation(client, recon_id: str) -> str | None:
    # Query the incident interface for an existing entry carrying this id.
    q = f"'Reconciliation_ID' = \"{recon_id}\""
    resp = await client.get("/api/arsys/v1/entry/HPD:IncidentInterface",
                            params={"q": q, "fields": "values(Incident_Number)"})
    resp.raise_for_status()
    entries = resp.json().get("entries", [])
    return entries[0]["values"]["Incident_Number"] if entries else None

async def create_remedy_incident(client, incident: RemedyIncident,
                                 dlq: asyncio.Queue,
                                 max_attempts: int = 5) -> str | None:
    # Dedup first: a retry after an ambiguous timeout resolves to the
    # existing ticket instead of opening a duplicate.
    existing = await find_by_reconciliation(client, incident.Reconciliation_ID)
    if existing:
        logger.info("reconciliation hit, skipping", extra={"num": existing})
        return existing

    body = incident.to_values()
    async with _dispatch_gate:          # backpressure: bounded creates in flight
        for attempt in range(1, max_attempts + 1):
            try:
                resp = await client.post(
                    "/api/arsys/v1/entry/HPD:IncidentInterface_Create",
                    json=body)
                if resp.status_code < 300:
                    return resp.headers.get("Location", "").rsplit("/", 1)[-1]
                if resp.status_code not in _RETRYABLE:
                    await dlq.put({"body": body, "status": resp.status_code,
                                   "detail": resp.text})
                    return None
            except (asyncio.TimeoutError, ConnectionError) as exc:
                logger.info("transient remedy failure", extra={"err": str(exc)})
            await asyncio.sleep(min(2 ** attempt * 0.15, 6.0))   # non-blocking
    await dlq.put({"body": body, "status": "exhausted"})
    return None

The dispatcher hooks into the async pipeline as a bounded consumer, but the semaphore is what makes it Remedy-safe: it holds concurrent creates to a fixed ceiling so the interface degrades gracefully under a storm instead of collapsing, while the upstream bounded queue absorbs the overflow. That valve reuses the shed-versus-block reasoning from Backpressure with Bounded asyncio Queues, and the reconciliation query is the Remedy analogue of the deduplication posture across Ticket Deduplication.

Mitigation and hardening

  1. Reconciliation-first deduplication. The Reconciliation_ID is queried before every create, so a retry after an ambiguous timeout resolves to the existing incident number rather than opening a twin. The identifier is derived only from the stable dedup_key, never a send-time value.
  2. Enumeration rejects. A create that fails because Impact or Urgency is not a recognized weight is a mapping defect, not a transient fault. Route it to the dead-letter queue with the response body, alert when the reject rate exceeds 0.5 percent over five minutes, and correct the tier-to-weight table rather than replaying a doomed request.
  3. Interface saturation. Because the entry form is the scarce resource, cap concurrent creates with the bounded semaphore and let the upstream queue hold the burst. Shedding must be severity-aware, so a P1 transport fault is never dropped to make room for low-priority noise.
  4. Slow-commit ambiguity. A create that commits but times out is reconciled on the next attempt by the identifier query, so no duplicate reaches an engineer even when the interface is slow enough to make timeouts routine.

Operational hardening notes

Reuse a single authenticated async session so the reconciliation query and the create share a warm connection and the token is refreshed once rather than per request — session setup against the interface is expensive. Size the semaphore to the interface’s measured safe concurrency rather than guessing high; Remedy throughput is the constraint, and over-driving it raises latency for every incident including the P1s. Keep the per-request timeout generous enough for a slow commit but bounded, and cap max_attempts so a degraded interface drains to the dead-letter queue instead of pinning workers. Held this way, correlated faults reach Remedy within a predictable creation budget, the reconciliation query keeps the duplicate rate effectively zero, and the semaphore keeps a storm from turning a slow interface into a stalled one.

Frequently Asked Questions

Why store the dedup key as a reconciliation identifier instead of using a native token?

The Remedy incident interface offers no native idempotency token like a ServiceNow correlation id. Storing the stable dedup key in a dedicated reconciliation field and querying it before every create gives the same guarantee client-side: a retry after an ambiguous timeout finds the existing incident number and returns it instead of opening a duplicate ticket.

How do I stop a storm from overwhelming the Remedy interface?

The entry form is the scarce resource, so a bounded semaphore caps how many creates are in flight and the upstream bounded queue absorbs the overflow. Shedding is severity-aware, so a P1 transport fault is never dropped to make room for low-priority noise, and the interface degrades gracefully instead of collapsing.

Why map numeric tiers to weight strings before sending?

Remedy stores impact and urgency as enumerated weight strings and will not coerce a bare integer into a valid weight. Mapping each numeric tier to the exact weight string through a lookup table, and validating it with a strict Pydantic enum, means an unrecognized value is caught locally rather than after a slow round trip that ends in a reject.

What happens when a create commits but the client times out?

The next attempt runs the reconciliation query first, finds the incident that did commit, and returns its number instead of creating a second one. Because Remedy commits can be slow enough to make timeouts routine, this reconciliation-first ordering is what keeps duplicates from ever reaching an engineer.