The importance and origin of reasoning skills used within software testing

We use our reasoning skills every day. Mostly they work well, but sometimes they let us down. Unfortunately, one place they let us down is when we think about testing.

One of the most important parts of software testing is to devise tests that reveal defects through breaking the software. This statement may appear self-evident, but it is surprisingly difficult to devise such critical tests. In this session we focus on exploring one question related to this problem:

Why do we find it easy to devise critical tests for some systems, but struggle for others?

To answer this question, we need to understand the reasoning processes we use to devise tests, where those processes originate from, and the original purpose from which they have been adapted.

Devising tests can be likened to understanding rules that govern a system’s expected behaviour, then devising ways to reveal system shortcomings that could be exploited to circumvent or break those rules.

Where in our evolutionary past could our ancestors have faced a similar situation, where an ability to either circumvent rules or spot others attempting to do so would have given us an advantage?

One such situation is within group and social behaviour. To survive and prosper as a group, we have developed a series of rules to govern an individual’s behaviour. An ability to conform to these rules, spot when others are not conforming, and to transgress those rules without getting caught has been an important part of successful group living. Importantly, such ability requires advanced reasoning skills.

Is it possible that the reasoning skills we use to devise critical software tests originate from the same set of skills we use to live successfully within social groups? If so, is this why we can reason and develop good tests for some systems, but not others? What other implications could this hold for testing?

In this interactive session, we use a series of puzzles and problems to demonstrate how our reasoning ability within testing is crucially dependent upon our ability to translate a testing problem into a context that is relevant to resolving a problem within social cheating.

We present two problems, with identical logic. One problem is framed in a way that triggers us to reason about the problem using mechanisms that we have developed for detecting social cheating. The second problem is framed in a way that does not trigger these mechanisms.

We discover that we can solve the first problem quickly, easily, and correctly, but we have difficulty solving the second problem and only manage to do so with errors.

Through these and other puzzles, we demonstrate that the suite of software tests devised by a test team is consistent with the extent to which the tester’s mind is triggered to reason about the problem from a social cheating point of view.

We use further puzzles to explore the implications for a test team and project in terms of; translating testing problems into a social context, the real reason why concrete examples work, enumeration versus elimination, hypothesis testing, diagnostic and non-diagnostic tests, and positive and negative testing.