What is Toil?

#What qualifies as toil

Toil exists on a spectrum. Common characteristics include being manual, repetitive, automatable, tactical, lacking enduring value, and increasing with service size or usage. The more closely a task matches these characteristics, the more likely it is to be toil.

The scaling characteristic is the one with the most operational consequence. Work that grows in step with the system can eventually consume a large share of a team's capacity, because infrastructure often grows faster than headcount.

#Why toil is managed explicitly

Toil does not announce itself. It accumulates through individually reasonable decisions to handle something manually rather than automate it.

• Growth ceiling: unmanaged toil can cap how large a system a team is able to operate.
• Displaced engineering: time spent on toil is time not spent reducing it.
• Morale cost: repetitive manual work is a recognized contributor to attrition on operations teams.
• Error rate: manual repetitive work performed under time pressure invites mistakes.

#How toil is reduced

Measurement precedes reduction. Tracking what operational work is performed and how often identifies the highest-volume tasks, which are often the automation candidates that pay back fastest.

Google caps operational work, including toil and non-toil operational work, for its SRE teams at 50% so that at least half of their time remains available for engineering. Other organizations should set a limit appropriate to their staffing and operating model rather than treating 50% as a universal standard.

#Toil vs operational work

Not all operations work is toil. Incident response requiring judgment, capacity planning, architectural review, and postmortem analysis are all operational, and they involve thought and produce lasting value.

Whether a machine could perform the task is one useful test, but it is not the only test. Repetition, tactical rather than strategic value, lack of enduring improvement, and linear scaling also matter. A task that requires deciding what to do based on circumstances is engineering work even when it happens during an incident.

#Key takeaways

• Toil tends to be manual, repetitive, automatable, tactical, without enduring value, and linearly scaling; not every toil task has all of these.
• Linear scaling with the system is the characteristic with the most operational consequence.
• Measurement identifies the highest-volume tasks, which are often the fastest-paying automation candidates.
• Google caps operational work for SRE teams at 50%; other organizations should choose a limit that fits their staffing model.

#Frequently asked

Is all manual work toil?

No. Work requiring judgment, such as incident diagnosis or capacity planning, is engineering work even when performed manually.

What is a reasonable toil target?

Google uses a 50% cap on operational work for its SRE teams. Treat that as a reference point and set a limit appropriate to your own staffing and operating model.

How do you start reducing toil?

Track operational tasks and their frequency for a few weeks. The highest-volume repetitive tasks are usually the first automation candidates.

#Sources

• Google SRE: eliminating toil