---

What Is 'The Irony of Automation'?

The Irony of Automation is that the more reliable the system, the less human operators have to do, so the less Attention they pay to the system while it's operating.

Reliable systems tend to make it hard for operators to notice when something's wrong. If an error is not noticed, it can eventually become the "new normal." The best way to avoid Automation errors is rigorous Sampling and Testing.

Focus on keeping your operators engaged, and they will be better suited to notice when something's wrong.

Josh Kaufman Explains 'The Irony of Automation'

Efficient Automated systems make skilled human intervention critically important to prevent the amplification of errors, so it's best to keep skilled operators on hand at all times, right?

That's easier said than done: businesses are typically loath to pay highly skilled employees to sit around in a room all day doing nothing, which is what they'll be doing if the system appears to be reliable.

And even if this policy is implemented, the operators will spend most of their time being bored out of their skulls.

Here's the Irony of Automation: the more reliable the system, the less human operators have to do, so the less they pay Attention to the system while it's in operation.

Remember the Mackworth Clock and the vigilance studies conducted on British radar operators in World War II in our discussion of Novelty? Humans get bored extremely quickly if things stay the same, and the more reliable the system, the more things stay the same.

Ironically, reliable systems tend to dull the operator's senses, making it much harder for them to notice when things go wrong-the moment when the operator's attention is most sorely needed. As a result, the more reliable the system, the lower the likelihood that human operators will notice when something goes wrong-particularly if the error is small.

Left long enough, small errors can become the "new normal," which is how a company like Toyota ends up with a billion-dollar recall. (By the way, don't make the Attribution Error and think that Toyota employees were stupid or careless-if you use automated systems, it could easily happen to you.)

The best approach to avoid major automation errors is rigorous, ongoing Testing.

Remember, Normal Accidents can and will occur. If you assume that errors will be made and plan a battery of tests to find the most important ones, you can keep your system operators engaged and increase the probability of finding critical errors quickly.

Keep your automated system's operators mentally engaged, and they'll be far more likely to notice when errors inevitably occur.

Questions About 'The Irony of Automation'

• How can you keep the operators of your automated system engaged enough to notice when things go wrong?
• What means of testing do you have at your disposal to quickly identify errors in automation?

---

"There will always be a set of circumstances that was not expected, that the automation either was not designed to handle or other things that just cannot be predicted. As system reliability increases, the more difficult it is to detect the error and recover from it."

Dr. Raja Parasuraman, professor of psychology at George Mason University

---

---

Share this concept:

https://personalmba.com/irony-of-automation/

---

---

---

---