What is Toil?
In SRE, toil is work tied to running a production service that tends to be manual, repetitive, automatable, tactical, lacking enduring value, and linearly scalable with service growth. A task does not need to have every characteristic to qualify as toil.
What qualifies as toil
Toil exists on a spectrum. Common characteristics include being manual, repetitive, automatable, tactical, lacking enduring value, and increasing with service size or usage. The more closely a task matches these characteristics, the more likely it is to be toil.
The scaling characteristic is the one with the most operational consequence. Work that grows in step with the system can eventually consume a large share of a team's capacity, because infrastructure often grows faster than headcount.
Why toil is managed explicitly
Toil does not announce itself. It accumulates through individually reasonable decisions to handle something manually rather than automate it.
- Growth ceiling: unmanaged toil can cap how large a system a team is able to operate.
- Displaced engineering: time spent on toil is time not spent reducing it.
- Morale cost: repetitive manual work is a recognized contributor to attrition on operations teams.
- Error rate: manual repetitive work performed under time pressure invites mistakes.
How toil is reduced
Measurement precedes reduction. Tracking what operational work is performed and how often identifies the highest-volume tasks, which are often the automation candidates that pay back fastest.
Google caps operational work, including toil and non-toil operational work, for its SRE teams at 50% so that at least half of their time remains available for engineering. Other organizations should set a limit appropriate to their staffing and operating model rather than treating 50% as a universal standard.
Toil vs operational work
Not all operations work is toil. Incident response requiring judgment, capacity planning, architectural review, and postmortem analysis are all operational, and they involve thought and produce lasting value.
Whether a machine could perform the task is one useful test, but it is not the only test. Repetition, tactical rather than strategic value, lack of enduring improvement, and linear scaling also matter. A task that requires deciding what to do based on circumstances is engineering work even when it happens during an incident.
Key takeaways
- Toil tends to be manual, repetitive, automatable, tactical, without enduring value, and linearly scaling; not every toil task has all of these.
- Linear scaling with the system is the characteristic with the most operational consequence.
- Measurement identifies the highest-volume tasks, which are often the fastest-paying automation candidates.
- Google caps operational work for SRE teams at 50%; other organizations should choose a limit that fits their staffing model.
Frequently asked
Sources
- Google SRE: eliminating toilsre.google