SRE concepts 2 min

What is Toil?

In SRE, toil is work tied to running a production service that tends to be manual, repetitive, automatable, tactical, lacking enduring value, and linearly scalable with service growth. A task does not need to have every characteristic to qualify as toil.

01 Mechanics

What qualifies as toil

Toil exists on a spectrum. Common characteristics include being manual, repetitive, automatable, tactical, lacking enduring value, and increasing with service size or usage. The more closely a task matches these characteristics, the more likely it is to be toil.

The scaling characteristic is the one with the most operational consequence. Work that grows in step with the system can eventually consume a large share of a team's capacity, because infrastructure often grows faster than headcount.

02 Value

Why toil is managed explicitly

Toil does not announce itself. It accumulates through individually reasonable decisions to handle something manually rather than automate it.

  • Growth ceiling: unmanaged toil can cap how large a system a team is able to operate.
  • Displaced engineering: time spent on toil is time not spent reducing it.
  • Morale cost: repetitive manual work is a recognized contributor to attrition on operations teams.
  • Error rate: manual repetitive work performed under time pressure invites mistakes.
03 Limits

How toil is reduced

Measurement precedes reduction. Tracking what operational work is performed and how often identifies the highest-volume tasks, which are often the automation candidates that pay back fastest.

Google caps operational work, including toil and non-toil operational work, for its SRE teams at 50% so that at least half of their time remains available for engineering. Other organizations should set a limit appropriate to their staffing and operating model rather than treating 50% as a universal standard.

04 Comparison

Toil vs operational work

Not all operations work is toil. Incident response requiring judgment, capacity planning, architectural review, and postmortem analysis are all operational, and they involve thought and produce lasting value.

Whether a machine could perform the task is one useful test, but it is not the only test. Repetition, tactical rather than strategic value, lack of enduring improvement, and linear scaling also matter. A task that requires deciding what to do based on circumstances is engineering work even when it happens during an incident.

Key takeaways

  • Toil tends to be manual, repetitive, automatable, tactical, without enduring value, and linearly scaling; not every toil task has all of these.
  • Linear scaling with the system is the characteristic with the most operational consequence.
  • Measurement identifies the highest-volume tasks, which are often the fastest-paying automation candidates.
  • Google caps operational work for SRE teams at 50%; other organizations should choose a limit that fits their staffing model.

Frequently asked

Sources

Product

  • Agentic Production Engineering

Compliance

All systems normalBuilt in NYC

The autonomous system for production.
SOC 2, GDPR, and HIPAA compliant.