Denoising production alerts to make on-call easier
How investigation evidence helps us connect related alerts, preserve separate failures, and guide the next step.
During an incident, alerts can arrive faster than an engineer can investigate them. Hundreds of alerts can point to only a handful of underlying problems. With each alert, we have to decide whether we’re seeing another symptom of the failure already under investigation or a separate problem that needs its own fix.
Different alerts can point to the same failure, and identical alerts can come from different causes.
At Antimetal, we use investigation evidence to decide which alerts belong together. The goal is a clear set of problems for engineers to act on, with less repeated investigation for agents.
Investigating every alert separately can mean repeating work and spending time, attention, and compute on a failure we’re already investigating. But grouping too aggressively can hide a second failure. To automate this decision, we need a learned system to determine what evidence is enough to connect two alerts and when we need to collect more.
Antimetal agents investigate production failures, bringing related signals, logs, metrics, and code from multiple monitoring tools into an Issue. As an investigation progresses, we use the available evidence to decide whether to associate an alert with an existing Issue or create a new one. This gives engineers one place to follow our understanding of an incident as more evidence arrives.
That context also gives the investigation agent an opportunity to recognize work already underway, carry useful findings forward, and redirect the investigation when a new alert reveals something the current explanation misses.
#Alert similarity is only a starting point
An alert tells us what a monitor observed, but often leaves us without the details we need to distinguish one failure from another. With a workflow warning, we know something failed, but we may still have to find out which step failed and why. If we group alerts by that warning alone, we’re assuming the same symptom means the same root cause.
In our own system, for example, the same GitHub workflow warning led to two different investigations. One found that a database lookup searched too few records and missed the record it needed. The other investigation claimed that the workflow had found the record but failed because the record had no link to an Issue.
The alerts came from the same monitor and had the same title, yet needed different fixes. We needed to expand the lookup for the first and handle records without an Issue association for the other. Matching the warnings would have combined two problems that needed separate work.
We can make the opposite mistake when alert messages differ, separating symptoms that share a root cause. Consider an API that starts timing out while a worker queue grows.
The alerts describe different symptoms, but traces show both services waiting for connections to the same database, and database metrics show that the connection limit has been reached. Those findings give us a reason to connect the two alerts.
Alert similarity may help us find related incidents, but before putting them in the same Issue, we need to compare what we’ve learned about the failures behind them.
#Correlating alerts with investigation evidence
When an alert reaches us, we may have only a symptom or an affected service, along with a time window. As our agents investigate, they add evidence about where the failure occurred in the code, along with metrics, events, logs, and traces from the affected service at the time of failure. Together, these findings give us a working hypothesis for root cause analysis.
To decide whether an alert belongs to an existing Issue, we work in two stages. We start with semantic search to find potentially related Issues, using embeddings to compare the meaning of the alert and any available investigation findings with what we already know about each Issue. Appropriate time windows help us limit the number of candidates.
Once we have those candidates, we compare the evidence. The embedding representations of our alerts and Issues capture semantic information relevant to the questions we need to answer.
Does the candidate Issue explain the incoming alert? Do the affected components and failure mechanism fit? Is there evidence of a separate problem that needs its own fix?
Alert correlation is the process of finding candidate Issues and deciding whether the evidence supports an association. When we’re unsure of the relationship, further investigation gives us more to compare.
#How new evidence can redirect an investigation
Grouping alerts gives engineers a more coherent view of a failure. We also want that shared context to help the agent decide what to investigate next.
In the database example, an agent investigating API timeouts might initially focus on the API’s request handling. A worker alert adds evidence that a separate service is also waiting for connections to the same database. Once we establish that relationship, the agent’s next step would be to investigate connection use across both services.
The agent should carry forward what it has learned about the API, incorporate the worker’s evidence as it arrives through correlation decisions, and test an explanation that accounts for both services. If we find that the connection limit has been reached, we’ve identified a shared constraint that may support a root cause analysis explaining both symptoms.
This can also change a proposed fix. If the agent was preparing to adjust the API’s timeout behavior, the worker evidence gives it a reason to ask whether that change would address the failure. Bringing the findings together helps it work toward a more complete explanation and fix.
When we associate an alert before a full investigation, we have an opportunity to reuse existing work. We need to decide when the evidence is strong enough to do that reliably and when the new alert needs further investigation. How well we make that decision affects the cost of running our agents, as we balance wasted work against the risk of missing information.
#An Issue engineers can act on
Engineers need a clear problem to solve, with enough context to act. An Issue brings together our current explanation, supporting evidence, and remaining work. As more signals arrive, we update that understanding.
In a customer environment, keeping related signals and their investigation findings in one place reduces their cognitive burden. As an incident unfolds, we want new alerts to contribute to that picture while independent failures remain visible.
That shared context also gives our agents work they can build on. Through alert correlation, they have an opportunity to reuse earlier findings and avoid investigating the same failure from scratch. When new evidence challenges our current explanation, it should change what the agent investigates next and prompt it to reassess whether the proposed fix is still sufficient.
Our goal is to give engineers a clear set of problems to act on, reduce repeated investigation, and keep separate failures visible. Each incoming signal should add to an existing investigation or bring a new problem into focus.
Thibaud Roy
Shreyas Iyer