Deduplicating error storms: one root cause, one ticket

A single failure rarely announces itself once. A bad config, an expired certificate, or a dependency outage produces events from every service that touches it, and each service describes the failure in its own vocabulary. Downstream, that becomes a wall of alerts and — in most setups — a wall of tickets.

The cost is not storage. It is that a team facing forty near-identical items learns to skim, and skimming is precisely how the genuinely novel incident underneath gets missed.

The obvious approach is to group events with identical messages. It fails immediately, because almost nothing about a real error message is stable: identifiers, timestamps, user references, hostnames, and request paths are all interpolated into the text. Two events from one cause are rarely byte-identical.

The next approach — fingerprinting on the exception type plus the top stack frame — is better and still brittle. One cause surfaces as different exception types in different services: a database failure appears as a connection error in one place, a timeout in another, and a null reference in a third where the code did not handle the empty response. Fingerprinting treats those as three problems.

Semantic matching compares what an event means rather than how it is spelled: the described failure, the subsystem involved, the timing, and the relationship to other recent signals. That is what lets a connection error, a timeout, and a null reference arriving in the same window from services sharing a dependency collapse into a single incident.

It also spans the human and machine sides of the queue. Three people reporting "checkout is broken" in different words, plus the error-tracker events behind it, are one incident — and a system that deduplicates alerts among themselves but not against human reports still leaves the team working the same problem twice.

Collapsing must not discard information. The occurrence count is what distinguishes a curiosity from an emergency, and the first and latest timestamps are what tell you whether the thing is accelerating. A deduplication system that hides the scale of a problem has traded one failure mode for a worse one.

Equally, deduplication that is too aggressive merges genuinely distinct failures that happen to be adjacent, and the second problem then disappears inside the first one’s ticket. The practical safeguard is that merges are visible and reversible: the linked sources stay on the ticket, and a wrongly merged item can be split back out.

Order matters. If deduplication happens after notification, the pager still fires forty times and the collapse only tidies the queue afterwards. Doing it at ingestion means one root cause produces one page, one owner, and one timeline — which is the property that keeps the pager credible.

This is the machinery FlowTux runs on error-tracker and CI intake: signals collapse semantically at ingestion, repeat occurrences increment the count on the existing ticket rather than creating new ones, human reports of the same incident join it, and only severity-gated conditions page. The postmortem then starts from a single timeline instead of forty fragments.