SRE concepts 2 min

What is Chaos Engineering?

Chaos engineering is the practice of deliberately injecting failures into a system to discover weaknesses before they surface on their own. It treats resilience as something to be verified experimentally rather than assumed from design.

01 Mechanics

How chaos experiments work

A chaos experiment starts with a hypothesis about steady state: under normal conditions, error rate stays below a threshold and latency stays within a band. The experiment then introduces a specific failure such as terminating an instance, adding network latency, or making a dependency return errors.

If steady state holds, the system tolerates that failure and the hypothesis is confirmed. If it does not, you have found a weakness under controlled conditions with an abort switch available, which is a dramatically better setting than discovering it at 3am.

02 Value

Why chaos engineering works

Resilience mechanisms that have never been exercised usually do not work, and there is no way to know which ones without testing them.

  • Verification: timeouts, retries, and failovers are confirmed to behave as designed.
  • Hidden coupling: dependencies nobody documented surface immediately.
  • Response rehearsal: the team practices incident response on a known problem.
  • Controlled discovery: weaknesses are found with an abort switch rather than during an outage.
03 Limits

Limits and prerequisites

Chaos engineering requires a baseline of maturity. A team without solid monitoring cannot tell whether an experiment caused a problem, and a team already firefighting daily does not need synthetic failures to find weaknesses.

Experiments also need real blast radius control: a defined scope, a monitored steady state, and a fast abort. Chaos without those properties is just breaking production, and it costs the practice its credibility inside the organization for a long time afterward.

04 Comparison

Chaos engineering vs load testing

Load testing answers a capacity question: how much traffic can the system handle and where does it saturate. The variable is volume.

Chaos engineering answers a resilience question: what happens when a component fails. The variable is availability of dependencies. They find different problems. Load testing finds the point where the system breaks under demand. Chaos finds the places where one component's failure propagates further than the design intended. Systems need both, and passing one says nothing about the other.

Key takeaways

  • An experiment defines a steady state hypothesis, injects a specific failure, and checks whether steady state holds.
  • Untested resilience mechanisms usually do not work, and testing is the only way to find out which.
  • Prerequisites are solid monitoring, defined blast radius, and a fast abort path.
  • Load testing finds saturation points while chaos finds propagation paths, and neither substitutes for the other.

Frequently asked

Product

  • Agentic Production Engineering

Compliance

All systems normalBuilt in NYC

The autonomous system for production.
SOC 2, GDPR, and HIPAA compliant.