Compressing Production Telemetry for Agents

At Antimetal, our agents debug production incidents by searching through petabytes of telemetry each day, including logs, traces, events, and metrics. In a large distributed system, a single trace can contain millions of tokens of raw JSON, more than even the largest context windows can hold.

This mismatch was a limiting factor in how effectively our agents could investigate several classes of incidents. But one of our early insights was that much of this volume comes from repeated structure. We mine that structure into templates: the fixed shape of each log, span, and event is stored once, and every occurrence keeps only its changing values, its timestamp, and a pointer to the raw record. This fits 10 to 100 times more telemetry into the same context window, which improves root cause analysis, lowers the cost per incident, and lets agents reason over weeks of behavior instead of a narrow incident window.

Scaling Context Doesn't Scale Understanding

Context windows have grown from 4K to more than a million tokens, but as a consequence of how compute and memory scale in transformers, the cost of using them grows with them. Longer inputs require more attention compute and a larger KV cache. Comprehension degrades as input grows too.

Scaling the context window through multi-agent architectures does not solve it either. Splitting context makes relationships across the data, such as cause and effect, harder to recover. Consider an incident split across three agents: the cause may land in one agent’s shard and its effect in another. If each agent exchanges only a compacted summary, the cause-and-effect relationship can be lost at the shard boundary.

Lossy by Design

Production telemetry is optimized for dashboards and alerts, where sampling and aggregation help human operators decide where to look. Agents face a different problem: they need enough coverage to identify patterns across millions of events while preserving the individual details that explain an incident. The common ways of controlling this volume create four distinct limitations for agents.

Sampling reduces coverage. Passing only a sample of the available telemetry to the agent controls context size, but can exclude tail events before the agent knows whether they matter. Every investigation therefore begins with an incomplete view.

Aggregation guides an investigation but cannot explain the incident on its own. Counts, percentiles, and top values help direct the agent, and we use them for exactly that purpose. However, every aggregation is limited to dimensions chosen in advance. A p99 value can identify an outlier, but it cannot recover the event, what happened around it, or answer a question that was not anticipated when the grouping was defined.

Model-based compaction loses semantic detail. A summarizer decides which details to retain before it knows which questions the investigation will raise. It may therefore omit the status code or other detail that ultimately explains the incident.

Raw data preserves detail but remains difficult to inspect at scale. Raw files and search tools keep the underlying evidence accessible, but each read covers only a slice. The agent must decide what to inspect before it has enough context to know which parts of the telemetry matter.

To avoid these tradeoffs, we needed a more effective way to represent telemetry, one that could fit more data into the context window without losing the details an investigation might depend on.

Telemetry Is Templates

Production telemetry repeats structure at several levels. A log statement preserves the same event structure while its runtime values change. The fixed text records what happened. The variables record where, when, and with what magnitude. Across many occurrences, their distributions show how the system's behavior changed.

Traces follow the same pattern. Operation names and call paths recur while timings and attributes vary. Code is a compressed description of system behavior. Telemetry is its high entropy expansion.

We took inspiration from log template miners such as Drain, which incrementally clusters incoming log lines into templates, replacing the parts that vary with typed parameters while keeping the fixed text. We then generalized the approach beyond log messages to structured telemetry, including traces and events. We designed the resulting representation for agents to query. Each occurrence retains its changing values, timestamp, and reference to the raw record. The representation also preserves how those values are distributed across the window.

Repeated stack traces are stored once. Recurring spans become templates, with parent links preserving the call tree and timings and attributes retained for each occurrence.

The agent can adjust the grouping parameters and time range to move between broad and detailed views. Each view combines templates and occurrence data with latency distributions, service edges, trace rollups, and outliers linked to their trace IDs.

The raw telemetry remains available for inspection. Once the agent finds a relevant template, value, or outlier, it can return to the original records.

Replaying Production Incidents

Across our internal incident replays, the compressed view changed the shape of investigations.

Accuracy improved most when the root cause depended on evidence across time ranges, services, log indexes, or traces. These incidents included memory leaks developing over hours, retry behavior spread across services, dependency failures followed through several log indexes, and latency introduced deep in a trace.

Focused application bugs saw less benefit. When a single error or stack trace contained the answer, both approaches usually reached the same diagnosis, and the compressed view could add modest overhead.

In one replay, the raw agent made 38 tool calls and inspected only 5 percent of the logs. It described several short error bursts as sustained activity and missed an additional connection pooler shutdown that produced 28 errors in under 300 milliseconds.

The agent with the compressed view pulled the full log window into a single view, then spent a few more calls drilling into the raw records behind the outliers. It reconstructed the timeline correctly and found the shutdown.

With this representation, our agents’ context windows can hold weeks of telemetry and system behavior without degrading diagnostic quality.