Risks of neglecting human factors in increasingly automated systems

As software development becomes increasingly automated through processes such as CI and DevOps, what will happen to risks, particularly those associated with human factors? Will risks reduce, increase, or transform in ways that we can neither anticipate nor easily comprehend?

We examine the tragic events of flight AF477 to illustrate that, as systems become increasingly automated, they also become vulnerable to failures from an entirely different set of human factors. We show that whilst increased automation of cockpit tasks has led to a significant reduction in numerous small errors, this reduction has been accompanied by an increased opportunity for creating a catastrophic error.

This increased opportunity for catastrophic error typically comes via two routes. Firstly, there is an increase in operator load extremities, with a decreased load during normal conditions, but a significantly increased load at times of crisis. Secondly, there is a de-skilling of roles, which leads to three consequences:

1. A reduction in calibre of people required, and hence selected or attracted, into the role
2. Reduced opportunities to practise techniques, as practise itself becomes more dangerous
3. The role becomes a dulled task, with employees filling time with non-work activities, such as Internet and mobile surfing, rather than honing task skills.

We use lessons and parallels from increased automation in aviation to explore the benefits, risks, and limitations of increased automation within software development. We also investigate potential mitigation strategies that would allow us to enjoy the benefits of increased automation whilst minimising the risk of catastrophic error.