Training COMMON CHALLENGES AND TIPS FOR SUCCESS13. Tip: Use "Mixed-Strength" tests more often than "regular" higher-strength tests

13. Tip: Use "Mixed-Strength" tests more often than "regular" higher-strength tests

This lesson explains why testers will usually find that Mixed-Stength tests provide a better balance of "additional thoroughness" vs. "additional time required" than regular 3-way, 4-way, 5-way, or 6-way tests.

Many testers new to this kind of testing are too quick to use sets of 3-way tests when they are seeking more thorough coverage than pairwise sets can provide.

Instead of executing an entire set of higher strength tests, it is usually more effiient and effective to execute set of Mixed-Strength tests instead.

Mixed-strength tests have the ability to include more thoroughness in selected areas (where you want the extra thoroughness). Sets of Mixed-strength tests will usually be fewer in number than "regular" higher strength plans.

There are only two possible reasons that a set of 2-way tests could fail to trigger a software defect:

  1. There was a missing test "idea." (e.g., the only way the defect would be found would be if the application were tested using a specific Operating System and that specific Operating System was not included as one of the inputs in the set of tests).
  2. All of the test ideas and test conditions were included as test inputs, but the defect could only be triggered by a test script that included three or more of those existing test conditions all together.

In our experience in working with hundreds of software teams, the first reason (not thinking to include a particular test idea) is responsible for more defects slipping by testing than the second reason (specific combinations of 3 or more already-included ideas).

There are two possible reasons that a set of 2-way tests could fail to trigger a software defect:

  1. There was a missing test "idea." (e.g., the only way the defect would be found would be if the application were tested using a specific Operating System and that specific Operating System was not included as one of the inputs in the set of tests).
  2. All of the test ideas and test conditions were included as test inputs, but the defect could only be triggered by a test script that included three or more of those existing test conditions all together.

In our experience in working with hundreds of software teams, the first reason (not thinking to include a particular test idea) is responsible for more defects slipping by testing than the second reason (specific combinations of 3 or more already-included ideas).

Accordingly, this is what we recommend for testers who are planning to manually execute sets of 3-way tests:

  • Don't.*
  • At least not until you first experiment a bit with executing well-thought through sets of Mixed-Strength tests.
  • Why? 90% of the extra thoroughness you might be looking to achive can probably be achieved by a well-thought out set of mixed-strength tests. And the mixed-strength tests might be half as many in number as the full "regular" higher-strength test set.
  • With the extra time that the team saves by not executing all those additional tests, they should use it to add more wrinkles into their testing / more testing ideas to their testing.  

*This advice is for manual testing projects (where the costs of executing extra tests is relatively high), not for automated test execution projects (where the costs of executing extra tests is low).

Examples of good sources for additional testing ideas include these: