Fault Correlation Ticket Routing Automation

Parsing RFC 5424 Syslog Structured Data

The operational failure this page addresses is the silent loss of the most useful part of a modern syslog line. RFC 5424 replaced the free-text soup of the old BSD syslog format with a machine-readable STRUCTURED-DATA section — key-value parameters grouped under named SD-IDs — that carries exactly the fields a correlation engine wants: interface names, alarm codes, enterprise identifiers, sequence numbers. A parser that treats a 5424 line as a blob and regexes the message text throws that structure away, reintroduces the ambiguity 5424 was designed to remove, and forces the correlation tier to re-derive fields that arrived already typed. Worse, a parser that assumes structured data is always present will trip over the nil value - that 5424 mandates when it is absent, and a single malformed line can stall a batch. Parsing the STRUCTURED-DATA section correctly — and quarantining what does not parse — is what keeps this feed contributing typed, deduplicable events instead of noise, and it is a direct lever on mean-time-to-acknowledge for any fault that arrives over syslog.

Schema alignment and taxonomy anchor

This page is the structured-data specialization of Logparser Integration, the deterministic normalization stage within the Ingestion & Parsing Workflows pipeline. That parent stage owns the rule cascade and the one-in-one-out contract; this page owns the narrower job of decoding a single RFC 5424 line — its PRI, its header, and especially its STRUCTURED-DATA — into the canonical fields defined by Event Schema Design. Header-level field mapping for vendor-prefixed lines is covered in How to Map Cisco Syslog to RFC 5424; this page picks up where the header ends and the SD-ID parameters begin. It sits alongside its sibling procedures Mapping SNMP Trap OIDs to Canonical Fault Codes and Parsing NETCONF YANG-Push Diff Notifications, each normalizing a different source format onto the same contract.

Anatomy of an RFC 5424 line

An RFC 5424 line has three regions before the free-text message. The PRI, in angle brackets, packs the facility and severity into one integer: facility = PRI // 8, severity = PRI % 8. The header carries the version, an ISO 8601 timestamp, the hostname, the app-name, the process ID, and the message ID. Then comes STRUCTURED-DATA: either a literal - for nil, or one or more elements of the form [SD-ID param="value" param="value"]. The SD-ID names the parameter group; enterprise-specific IDs carry an @ and an IANA private enterprise number, such as [meta@32473 sequenceId="14"].

Diagram: an RFC 5424 line decomposed into PRI, header, and one or more STRUCTURED-DATA elements, each mapping onto canonical event fields.

RFC 5424 line structure and field mappingA single RFC 5424 syslog line is drawn as three adjacent segments. The first segment is the PRI in angle brackets; an arrow below it shows PRI divided by eight yields the facility and PRI modulo eight yields the severity. The middle segment is the header, listing version, timestamp, hostname, app-name, process ID and message ID. The third segment is the STRUCTURED-DATA region containing a bracketed SD-ID element with an enterprise number and quoted parameters. Arrows from the severity value and from the SD-ID parameters point down to a canonical event box holding severity, device_id, vendor_alarm_code, event_time and sequence fields, showing how each parsed region populates the schema.PRI<190>HEADERver · ts · host · app · procid · msgidSTRUCTURED-DATA[meta@32473 sequenceId="14" alarm="LINK_DOWN"]facility = PRI // 8severity = PRI % 8SD-ID + params extractedenterprise id @32473Canonical fault eventseverity · device_id · vendor_alarm_code · event_time · sequence

Parsing STRUCTURED-DATA

The parser below decodes a full 5424 line: it splits the PRI, header, and STRUCTURED-DATA regions, tokenizes each SD element into its SD-ID and parameters, and constructs a validated event. Structured-data tokenizing is done with a small state machine rather than a single greedy regex, because parameter values may contain spaces, escaped quotes, and brackets — precisely the cases a naive \[.*\] pattern gets wrong. Validation uses Pydantic V2 in strict mode so a line that yields the wrong shape becomes a clean quarantine entry.

import hashlib
from datetime import datetime
from typing import Optional

from pydantic import BaseModel, ConfigDict, Field, field_validator


class SyslogEvent(BaseModel):
    """Canonical fault event derived from one RFC 5424 line."""
    model_config = ConfigDict(strict=True, extra="forbid", frozen=True)

    device_id: str
    facility: int = Field(ge=0, le=23)
    severity: int = Field(ge=0, le=7)            # 0 emerg .. 7 debug
    event_time: datetime
    vendor_alarm_code: str
    sequence_id: Optional[int] = Field(default=None, ge=0)
    raw_payload_hash: str = Field(min_length=64, max_length=64)

    @field_validator("event_time")
    @classmethod
    def require_utc(cls, value: datetime) -> datetime:
        if value.tzinfo is None:                 # 5424 timestamps are offset-aware
            raise ValueError("event_time must be timezone-aware")
        return value


def decode_pri(token: str) -> tuple[int, int]:
    """Split the <PRI> token into (facility, severity)."""
    if not (token.startswith("<") and ">" in token):
        raise ValueError("missing PRI")
    pri = int(token[1:token.index(">")])
    if not 0 <= pri <= 191:
        raise ValueError(f"PRI out of range: {pri}")
    return pri // 8, pri % 8                      # facility, severity


def parse_structured_data(sd: str) -> dict[str, dict[str, str]]:
    """Tokenize STRUCTURED-DATA into {sd_id: {param: value}}.

    Handles the nil value '-', multiple elements, quoted values containing
    spaces, and backslash-escaped quotes and brackets per RFC 5424 section 6.3.
    """
    if sd == "-":                                # explicit nil; not an error
        return {}
    elements: dict[str, dict[str, str]] = {}
    i, n = 0, len(sd)
    while i < n:
        if sd[i] != "[":
            i += 1
            continue
        j = i + 1
        # SD-ID runs up to the first space or the closing bracket.
        while j < n and sd[j] not in " ]":
            j += 1
        sd_id = sd[i + 1:j]
        params: dict[str, str] = {}
        while j < n and sd[j] != "]":
            if sd[j] == " ":
                j += 1
                continue
            k = j
            while k < n and sd[k] != "=":        # param name
                k += 1
            name = sd[j:k]
            k += 1                                # skip '='
            if k < n and sd[k] == '"':           # quoted value with escapes
                k += 1
                buf = []
                while k < n and sd[k] != '"':
                    if sd[k] == "\\" and k + 1 < n:
                        k += 1
                    buf.append(sd[k])
                    k += 1
                params[name] = "".join(buf)
                k += 1                            # skip closing quote
            j = k
        elements[sd_id] = params
        i = j + 1
    return elements


def parse_5424(line: str) -> SyslogEvent:
    """Decode one RFC 5424 line into a canonical event or raise ValueError."""
    # HEADER fields are space-delimited; the message (if any) follows the SD.
    pri_tok, rest = line.split(" ", 1) if " " in line else (line, "")
    # PRI and VERSION are joined: <190>1 -> pull the leading <..> token.
    gt = pri_tok.index(">")
    facility, severity = decode_pri(pri_tok[: gt + 1])
    header = rest.split(" ", 6)                   # ver ts host app procid msgid sd+msg
    if len(header) < 7:
        raise ValueError("truncated 5424 header")
    _ver, ts, host, app, _procid, _msgid, sd_and_msg = header
    # STRUCTURED-DATA is either '-' or a run of bracketed elements; the message
    # begins after the last closing bracket (or after '-').
    if sd_and_msg.startswith("-"):
        sd_str, msg = "-", sd_and_msg[1:].strip()
    else:
        end = sd_and_msg.rindex("]") + 1
        sd_str, msg = sd_and_msg[:end], sd_and_msg[end:].strip()
    sd = parse_structured_data(sd_str)
    meta = next(iter(sd.values()), {})            # first SD element's params
    return SyslogEvent(
        device_id=host,
        facility=facility,
        severity=severity,
        event_time=datetime.fromisoformat(ts),
        vendor_alarm_code=meta.get("alarm", app or msg[:32] or "UNCLASSIFIED"),
        sequence_id=int(meta["sequenceId"]) if "sequenceId" in meta else None,
        raw_payload_hash=hashlib.sha256(line.encode()).hexdigest(),
    )

Async ingestion hook

Syslog arrives over UDP and TCP at high rate, so parsing must never block the event loop or let one malformed line stall a batch. The hook below drains a batch, parses each line in isolation, and quarantines failures to a dead-letter queue without stopping the consumer. Parsing is pure CPU and bounded per line, so it stays inline; the only awaited calls are the downstream handoff and the dead-letter put.

import asyncio
import logging
from typing import Any

logger = logging.getLogger("syslog_5424")


async def process_syslog_batch(
    lines: list[str],
    publish,
    dlq: asyncio.Queue[dict[str, Any]],
) -> None:
    """Parse a batch of RFC 5424 lines, isolating each failure."""
    events = []
    for idx, line in enumerate(lines):
        try:
            events.append(parse_5424(line))
        except (ValueError, KeyError) as exc:
            # One bad line never stalls the batch; quarantine and continue.
            await dlq.put({
                "index": idx,
                "raw": line,
                "reason": type(exc).__name__,
                "detail": str(exc),
            })
            logger.warning("5424 parse failure", extra={"reason": str(exc)})
    if events:
        await publish(events)                     # single non-blocking bus write
    await asyncio.sleep(0)                         # yield to the event loop

Mitigation and hardening

When a line does not parse, degrade gracefully rather than halt. These are the concrete failure paths a production deployment should implement.

  1. Malformed-line quarantine. A truncated header, an out-of-range PRI, or unbalanced SD brackets raises during parse; the line is serialized with its exact reason code and pushed to the dead-letter queue, never dropped and never forwarded half-formed. Alert when the quarantine rate exceeds 0.5% over five minutes — a spike usually means a firmware change altered the SD-ID layout on one device class.
  2. Nil structured data is not an error. RFC 5424 mandates a literal - when structured data is absent. The parser must treat - as an empty map, not a failure; conflating the two floods the dead-letter queue with legitimate lines and masks real drift.
  3. PRI bounds enforcement. A PRI above 191 or a non-numeric PRI is a corrupt frame or a spoofed source. Rejecting it at the boundary — the same posture as Security Boundary Mapping — keeps a malformed severity from skewing correlation.
  4. Enterprise-ID awareness. Enterprise SD-IDs carry an @ and a private enterprise number registered in the IANA Private Enterprise Numbers registry. Preserving that number lets downstream mapping distinguish a Cisco meta@9 from an unknown vendor without guessing, and an unrecognized enterprise number is a coverage gap to record, not a line to drop.

Operational hardening notes

Keep the per-line cost flat under storm load. The state-machine tokenizer is O(n) in line length with no backtracking, which is the whole reason to prefer it over a greedy regex that can degrade catastrophically on a pathological value — the same discipline the parent Logparser Integration stage enforces for its rule matrix. Parse at the batch level, 500 to 1000 lines per consumer poll, so the loop yields between batches rather than between lines. Warm the facility and severity lookups at load rather than resolving per line. Held to these bounds, this parser holds p99 parse latency under 50ms per line and keeps the quarantine rate under 0.5%, which is the practical signal that vendor SD-ID coverage is healthy and that this syslog feed is contributing typed, deduplicable events to the correlation tier rather than raw text.

Frequently Asked Questions

Why parse STRUCTURED-DATA instead of regexing the message text?

Because the structured-data section already carries the fields a correlation engine needs — interface names, alarm codes, sequence numbers — as typed key-value parameters under named SD-IDs. Regexing the free-text message throws that structure away and reintroduces the ambiguity RFC 5424 was designed to remove, forcing the correlation tier to re-derive fields that arrived already parsed. Reading the SD parameters directly is both faster and more reliable.

How should the parser treat a structured-data value of dash?

Treat it as explicit nil, meaning no structured data is present, and return an empty parameter map rather than raising an error. RFC 5424 mandates the dash when structured data is absent, so conflating it with a parse failure floods the dead-letter queue with legitimate lines and hides the real schema drift you actually want to alert on.

What do facility and severity come from in an RFC 5424 line?

They are both encoded in the PRI, the integer in angle brackets at the start of the line. Facility is the PRI divided by eight and severity is the PRI modulo eight, where severity runs from zero for emergency to seven for debug. A PRI above 191 or a non-numeric PRI is a corrupt or spoofed frame and should be rejected at the boundary rather than parsed.

What happens to a line the parser cannot decode?

It is quarantined to a dead-letter queue carrying the raw line, a reason code, and the failing field, rather than dropped or forwarded half-formed. One malformed line never stalls the batch. A quarantine rate above 0.5 percent over five minutes is alertable and usually means a firmware change altered the SD-ID layout on one device class, which is traceable from the reason codes.