What is Observability?

#How observability works

Observability comes from instrumentation that emits enough detail, with enough dimensions, to support questions nobody wrote down in advance. The critical property is high cardinality: telemetry tagged with user ID, request ID, region, version, and customer tier can be sliced arbitrarily after the fact.

A system that emits only pre aggregated counters can answer only the questions those counters were built for. A system that emits richly tagged events can answer questions invented during an incident, which is when the important questions actually arrive.

#Why observability matters

Distributed systems fail in ways nobody predicted, and predicting them is not a workable strategy.

• Novel failure investigation: new questions can be answered against existing data.
• Narrow slice analysis: problems affecting one customer or one region become visible.
• Faster diagnosis: less guessing and less shipping code to add logging mid incident.
• Emergent behavior: interactions between correct components become visible.

#Limits and costs

Observability is expensive. High cardinality, high dimensionality telemetry is costly to store and query, and observability spend regularly rivals the infrastructure it observes. Sampling controls cost and risks discarding exactly the rare requests worth examining.

Data is also not understanding. A team with excellent telemetry and no idea how to query it during an incident has paid for observability without receiving it, and the skill to interrogate the data is as much of the investment as the pipeline.

#Observability vs monitoring

Monitoring watches for conditions you defined in advance: thresholds, health checks, known failure modes. It answers whether the system is behaving as expected, and it is essential and inherently limited to anticipated problems.

Observability supports investigation of the unanticipated. Monitoring tells you something is wrong, observability lets you find out what. Neither replaces the other: monitoring without observability produces alerts nobody can diagnose, and observability without monitoring produces excellent data that nobody looks at until a customer complains.

#Key takeaways

• The test of observability is answering an unanticipated question without deploying new code.
• High cardinality telemetry is what enables arbitrary slicing after the fact.
• Cost is substantial, and sampling trades expense against the risk of discarding rare significant requests.
• Monitoring covers anticipated conditions and observability covers investigation, and both are required.

#Frequently asked

Is observability just three tools bundled together?

No. Metrics, logs, and traces are inputs. Observability is whether they can answer questions nobody planned for, which depends on cardinality and correlation rather than tool count.

What is high cardinality and why does it matter?

A field with many distinct values, such as user ID. It is what allows narrowing an investigation from all traffic to the specific requests that failed.

How do you control observability costs?

Sample routine high volume data while keeping errors and slow requests unsampled, and set retention by data type rather than uniformly.