What is Observability?
Observability is the property of a system that lets you understand its internal state from its external outputs. The practical test is whether you can answer a question you did not anticipate, without deploying new code to gather the data.
How observability works
Observability comes from instrumentation that emits enough detail, with enough dimensions, to support questions nobody wrote down in advance. The critical property is high cardinality: telemetry tagged with user ID, request ID, region, version, and customer tier can be sliced arbitrarily after the fact.
A system that emits only pre aggregated counters can answer only the questions those counters were built for. A system that emits richly tagged events can answer questions invented during an incident, which is when the important questions actually arrive.
Why observability matters
Distributed systems fail in ways nobody predicted, and predicting them is not a workable strategy.
- Novel failure investigation: new questions can be answered against existing data.
- Narrow slice analysis: problems affecting one customer or one region become visible.
- Faster diagnosis: less guessing and less shipping code to add logging mid incident.
- Emergent behavior: interactions between correct components become visible.
Limits and costs
Observability is expensive. High cardinality, high dimensionality telemetry is costly to store and query, and observability spend regularly rivals the infrastructure it observes. Sampling controls cost and risks discarding exactly the rare requests worth examining.
Data is also not understanding. A team with excellent telemetry and no idea how to query it during an incident has paid for observability without receiving it, and the skill to interrogate the data is as much of the investment as the pipeline.
Observability vs monitoring
Monitoring watches for conditions you defined in advance: thresholds, health checks, known failure modes. It answers whether the system is behaving as expected, and it is essential and inherently limited to anticipated problems.
Observability supports investigation of the unanticipated. Monitoring tells you something is wrong, observability lets you find out what. Neither replaces the other: monitoring without observability produces alerts nobody can diagnose, and observability without monitoring produces excellent data that nobody looks at until a customer complains.
Key takeaways
- The test of observability is answering an unanticipated question without deploying new code.
- High cardinality telemetry is what enables arbitrary slicing after the fact.
- Cost is substantial, and sampling trades expense against the risk of discarding rare significant requests.
- Monitoring covers anticipated conditions and observability covers investigation, and both are required.