Training TEST DESIGN PRINCIPLES8. Prioritization: how should we get the biggest "bang for the buck" in a limited number of software tests?

8. Prioritization: how should we get the biggest "bang for the buck" in a limited number of software tests?

Pairwise testing methods (and related Hexawise test design approaches) help testers make smart prioritization decisions by taking advantage of the information in this chart:

Pairwise testing methods (and related Hexawise test design approaches) help testers make smart prioritization decisions by taking advantage of the information in this chart:

Sources: averages from the following four studies, each of which measured how many test inputs needed to be included together in a single test case in order to trigger defects in production in mature, well-tested applications?

Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.


Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA, Goddard SFC 4-6 December, 2002. 

NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.

Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006. 

This presentation summarizes what pairwise testing is and the theory behind why it works so well.

This fun presentation explains what pairwise testing (and closely-related test design approaches) are all about and why they work. The presentation also explains some of the more thorough combinatorial testing approaches (e.g, 3-way, 4-way, 5-way, and 6-way testing). If you're looking for a photo-rich introduction to these concepts, this is it.

Video Tutorial: What is pairwise testing?

This video, an introductory tutorial on "what is pairwise testing?" walks through the process of creating a set of allpairs tests using a banking example with almost 20,000 possible tests.

Want fewer pictures and more text? This 3-page article, written by 3 PhDs and the CEO of Hexawise, is a concise introduction to efficient and effective test design strategies.

Want fewer pictures and more text?  This 3-page article, written by 3 PhDs and the CEO of Hexawise, is a concise introduction to efficient and effective test design strategies.

"Combinatorial Software Testing" IEEE, August 2009.