8. Prioritization: how should we get the biggest "bang for the buck" in a limited number of software tests?
Pairwise testing methods (and related Hexawise test design approaches) help testers make smart prioritization decisions by taking advantage of the information presented in this chart:
The above chart is based on averages presented in the following four studies, each of which measured how many test inputs needed to be included together in a single test case in order to trigger defects in production in mature, well-tested applications.
Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.
Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA, Goddard SFC 4-6 December, 2002.
NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.
Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.
Summary of Pairwise Testing and the Theory Behind Why It Works So Well
This fun presentation explains what pairwise testing (and closely-related test design approaches) are all about and why they work. The presentation also explains some of the more-thorough combinatorial testing approaches (e.g, 3-way, 4-way, 5-way, and 6-way testing). If you're looking for a photo-rich introduction to these concepts, this is it!
Video Tutorial: What is Pairwise Testing?
This video, an introductory tutorial on pairwise testing? walks through the process of creating a set of all pairs tests using a banking example with almost 20,000 possible tests.
Prefer text? This article, written by three PhDs and the CEO of Hexawise, is a concise introduction to efficient and effective test design strategies.
"Combinatorial Software Testing" IEEE, August 2009.