Fault Correlation Ticket Routing Automation

Parsing NETCONF YANG-Push Diff Notifications

The operational failure this page addresses is the config-churn false positive. NETCONF YANG-push lets a device stream a push-change-update every time a subscribed subtree of its datastore changes — an interface goes admin-down, a routing policy is edited, a BGP neighbor is removed. That is a rich fault signal, but it is also noisy: a single maintenance edit can emit a burst of diffs as the device settles, and a naive parser that raises one fault per diff turns one planned change into a dozen spurious tickets. Every spurious ticket is triage time an on-call engineer will never get back, and it erodes trust in the feed until real faults get ignored alongside the noise. Parsing each diff correctly — and debouncing the change-set so a settling burst collapses into one decision — is what keeps this feed contributing genuine configuration-fault signal instead of alarm fatigue, and it is a direct lever on both mean-time-to-acknowledge and the false-positive rate.

Schema alignment and taxonomy anchor

This page is the NETCONF specialization of Logparser Integration, the deterministic normalization stage within the Ingestion & Parsing Workflows pipeline. That parent stage owns the rule cascade and the one-in-one-out contract; this page owns the narrower job of turning a push-change-update notification — a set of YANG-modeled edits, each a path plus an operation — into the canonical fields defined by Event Schema Design. It sits alongside its sibling procedures Parsing RFC 5424 Syslog Structured Data and Mapping SNMP Trap OIDs to Canonical Fault Codes, each normalizing a different source format onto the same contract. Whether a resulting fault is a symptom of a wider event is decided later by Topology-Aware Correlation; this stage only produces the typed event.

Anatomy of a YANG-push diff

A push-change-update carries a subscription ID and a datastore-changes payload. Each edit inside it is a YANG patch: an operation (create, delete, replace, merge) and a target path in the modeled tree, such as /interfaces/interface[name='ge-0/0/1']/config/enabled. The path plus the operation, not the message text, is what identifies the fault: a replace that sets enabled to false on an interface is an admin-down, a delete on a BGP neighbor is a session teardown. Mapping is a path-pattern-to-fault-code resolve, and debouncing groups the edits arriving for the same target within a short window so a settling burst is one decision.

Diagram: YANG-push edits are keyed by target path and coalesced in a debounce window before a single path-to-fault decision, so a settling burst produces one event.

YANG-push debounce and path-to-fault mappingOn the left, four edit notifications arrive along a time axis; three of them target the same interface path in quick succession and one targets a different path. A debounce window box groups the three same-path edits together. When the window closes, the net settled state of that target is passed to a path-to-fault mapping table in the center, which resolves the target path and operation to a canonical fault code and emits one canonical fault event on the right. A separate lower path shows a target whose edits cancel out to a no-op being routed to a suppressed outcome instead of producing a ticket, illustrating how config churn is collapsed.push-change-update editsreplace enabled=falsemerge mtu=1500replace enabled=falsecreate + delete (net 0)Debouncewindowcoalesce by pathPath-to-fault mappath + op → codeOne fault eventADMIN_DOWNSuppressednet no-op · no ticket

Parsing and debouncing the change-set

The implementation below parses a push-change-update into per-target edits, applies last-writer-wins within a debounce window, and maps the settled state through a path-pattern table to a canonical fault code. The debounce is time-bounded and keyed on the target path, so edits that arrive close together for the same target coalesce, and a net no-op is suppressed rather than ticketed. Validation is Pydantic V2 in strict mode.

import asyncio
import hashlib
import re
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Optional

from pydantic import BaseModel, ConfigDict, Field, field_validator


class FaultCode(str, Enum):
    ADMIN_DOWN = "ADMIN_DOWN"
    BGP_NEIGHBOR_REMOVED = "BGP_NEIGHBOR_REMOVED"
    ROUTING_POLICY_CHANGED = "ROUTING_POLICY_CHANGED"


# Path-pattern + operation -> (fault code, baseline severity). Compiled once.
PATH_FAULT_RULES: list[tuple[re.Pattern[str], str, tuple[FaultCode, int]]] = [
    (re.compile(r"/interfaces/interface\[[^\]]+\]/config/enabled$"),
     "replace", (FaultCode.ADMIN_DOWN, 3)),
    (re.compile(r"/network-instances/.*/bgp/neighbors/neighbor\[[^\]]+\]$"),
     "delete", (FaultCode.BGP_NEIGHBOR_REMOVED, 2)),
    (re.compile(r"/routing-policy/policy-definitions/.*$"),
     "merge", (FaultCode.ROUTING_POLICY_CHANGED, 4)),
]


class ConfigFaultEvent(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid", frozen=True)

    device_id: str
    fault_code: FaultCode
    severity: int = Field(ge=0, le=7)
    target_path: str
    event_time: datetime
    raw_payload_hash: str = Field(min_length=64, max_length=64)

    @field_validator("event_time")
    @classmethod
    def require_utc(cls, value: datetime) -> datetime:
        if value.tzinfo is None:
            raise ValueError("event_time must be timezone-aware UTC")
        return value.astimezone(timezone.utc)


def parse_edits(notification: dict[str, Any]) -> list[dict[str, Any]]:
    """Flatten a push-change-update into per-edit {path, op, value} dicts."""
    edits = []
    for change in notification.get("datastore_changes", []):
        edits.append({
            "path": change["target"],             # YANG instance-identifier
            "op": change["operation"],            # create/delete/replace/merge
            "value": change.get("value"),
        })
    return edits


def settle(edits: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
    """Last-writer-wins per target path; drop targets whose net change cancels."""
    by_path: dict[str, dict[str, Any]] = {}
    for edit in edits:
        path = edit["path"]
        if edit["op"] == "delete" and path in by_path and by_path[path]["op"] == "create":
            del by_path[path]                     # create then delete = net no-op
        else:
            by_path[path] = edit                  # later edit supersedes earlier
    return by_path


def map_edit(device_id: str, path: str, edit: dict[str, Any]) -> Optional[ConfigFaultEvent]:
    """Resolve one settled edit to a fault event, or None if no rule matches."""
    for pattern, op, (code, severity) in PATH_FAULT_RULES:
        if edit["op"] == op and pattern.search(path):
            return ConfigFaultEvent(
                device_id=device_id,
                fault_code=code,
                severity=severity,
                target_path=path,
                event_time=datetime.now(timezone.utc),
                raw_payload_hash=hashlib.sha256(
                    f"{device_id}{path}{edit['op']}{edit.get('value')}".encode()
                ).hexdigest(),
            )
    return None                                    # unmapped path: caller records it

Async ingestion hook

Because a burst of edits must be coalesced before a decision, the debounce is the one place this stage holds state — a short-lived, per-device buffer flushed by a timer, never a cross-event history. The hook below buffers incoming notifications per device, and a background flusher settles and maps each buffer when its debounce window closes, emitting at most one event per target path. Nothing blocks the receiving loop.

import logging
from collections import defaultdict

logger = logging.getLogger("yang_push")

DEBOUNCE_S = 2.0                                  # coalesce a settling burst


async def debounce_and_publish(
    device_id: str,
    buffer: list[dict[str, Any]],
    publish,
    dlq: asyncio.Queue[dict[str, Any]],
) -> None:
    """Wait out the debounce window, then settle, map, and emit one per target."""
    await asyncio.sleep(DEBOUNCE_S)               # let the burst settle
    settled = settle(buffer)
    for path, edit in settled.items():
        try:
            event = map_edit(device_id, path, edit)
            if event is None:                     # no rule: record coverage gap
                await dlq.put({"device_id": device_id, "path": path,
                               "reason": "unmapped_path"})
                continue
            await publish(event)
        except (ValueError, KeyError) as exc:
            await dlq.put({"device_id": device_id, "path": path,
                           "reason": "schema_reject", "detail": str(exc)})


async def ingest(notifications, publish, dlq: asyncio.Queue) -> None:
    """Buffer per-device edits and schedule a debounced flush per burst."""
    buffers: dict[str, list[dict[str, Any]]] = defaultdict(list)
    pending: dict[str, asyncio.Task] = {}
    async for notification in notifications:      # async iterator of push-change-updates
        device_id = notification["device_id"]
        buffers[device_id].extend(parse_edits(notification))
        if device_id not in pending or pending[device_id].done():
            # One flush task per settling burst; extends the same buffer.
            pending[device_id] = asyncio.create_task(
                debounce_and_publish(device_id, buffers[device_id], publish, dlq)
            )

Mitigation and hardening

When a change-set is noisy or unmapped, degrade deterministically. These are the concrete failure paths a production deployment should implement.

  1. Debounce, do not deduplicate blindly. The debounce window collapses a settling burst for one target into a single decision using last-writer-wins, so a maintenance edit that flaps enabled several times before settling yields one event, not several. Keep the window short — one to three seconds — so genuine rapid faults are not masked; too long a window hides real recurrence, exactly the risk the parent Logparser Integration tuning notes flag for any dedup key.
  2. Net-no-op suppression. A create followed by a delete on the same target within the window is a net no-op — config churn that changed nothing. Suppress it rather than emitting paired create and delete faults, which would otherwise open and immediately resolve a spurious ticket.
  3. Unmapped-path recording. An edit whose path matches no rule is not dropped and not guessed; it is dead-lettered as unmapped_path so the rule table can be extended. Losing a genuine new configuration fault is worse than recording a path for review.
  4. Bounded per-device buffers. The debounce buffer is per-device and time-bounded; a device that never stops emitting edits must not grow its buffer without limit. Cap it and flush early on overflow, echoing the backpressure posture of Async Batch Processing.

Operational hardening notes

Keep the per-notification cost flat and the debounce state small. The path-pattern rules are compiled once at load, and each edit is tested against them with anchored patterns so a pathological path cannot trigger catastrophic backtracking — the same regex discipline the parent stage enforces. The debounce buffer is the only state this stage holds, and it is short-lived and per-device, so the stage stays effectively stateless and horizontally scalable: a device is owned by one consumer, and its buffer never outlives its window. Tune the debounce window against the observed settling time of your maintenance workflows; measure the false-positive rate before and after and expect a well-tuned window to cut config-churn tickets by well over half while adding at most the window’s duration to detection latency for a genuine change. Held this way, this parser turns a noisy datastore-change stream into one deterministic fault event per real configuration change, keeping the feed trustworthy instead of a source of alarm fatigue. The async buffering and timer patterns follow the non-blocking model in the official Python asyncio documentation.

Frequently Asked Questions

Why debounce YANG-push diffs instead of raising a fault per edit?

Because a single maintenance change often emits a burst of edits as the device settles, and raising one fault per edit turns one planned change into many spurious tickets. Debouncing coalesces the edits for one target within a short window and applies last-writer-wins, so the settling burst produces a single decision. That is what keeps config churn from generating alarm fatigue and eroding trust in the feed.

What identifies the fault in a YANG-push notification?

The target path and the operation, not the message text. A replace that sets enabled to false on an interface path is an admin-down; a delete on a BGP neighbor path is a session teardown. Mapping resolves the combination of path pattern and operation to a canonical fault code, which is why the parser keys on the modeled datastore path rather than any human-readable description.

How does the parser avoid ticketing a change that cancels itself out?

By settling the change-set before mapping. When a create and a matching delete arrive for the same target inside the debounce window, their net effect is nothing, so the target is suppressed rather than emitting paired create and delete faults. This stops a churny edit sequence from opening and immediately resolving a spurious ticket.

What happens to an edit whose path matches no rule?

It is recorded to the dead-letter queue as an unmapped path rather than dropped or guessed into a fault code. Losing a genuine new configuration fault is worse than holding an unrecognized path for review, and the recorded paths are the signal for extending the mapping rules. A rising unmapped-path rate usually means a new YANG model or a firmware release the rule table has not caught up to.