Training COMMON CHALLENGES AND TIPS FOR SUCCESS24. TIP: How to Deal with "Select All That Apply" Scenarios

24. TIP: How to Deal with "Select All That Apply" Scenarios

The Problem:

This explanation provides a solution to a difficult test design concept often faced by Hexawise users: how to handle "Select All That Apply" scenarios. Such scenarios occur frequently in Web Applications, and are typically mishandled by test designers. Let's look at the following example for some context: 

The “Select All that Apply” pattern is very common in many types of applications. This pizza ordering example, shown above, is a clear illustration of a "Select All That Apply” situation. 

As you can see in the image above, you have many topping choices available. You can choose as many or as few as you would like, and in this system there are no guidelines surrounding the maximum number of toppings you can add (though it may get a bit pricey). When facing a situation like this, the first test design strategy that occurs to many test designers is to create a single parameter called "Pizza Toppings” in Hexawise and list each of the 24 available toppings as Values, as shown below.

Avoid this Common Mistake - Do not list all of the “Select all that apply” values in single parameter

While this approach seems natural, this test design strategy would have two major disadvantages. The first problem is that every generated test scenario using this test design approach would have one and only one pizza topping. The system in its current design would only have test scenarios for pizzas with pepperoni OR sausage OR bell peppers, when in reality a user might actually order a pizza with pepperoni AND sausage AND bell peppers. The second problem with this test design strategy is that the number of 2-way tests generated would be unnecessarily large. This is because each of the 24 toppings would need to appear together with every other value in the plan. This would result in at least 98 test scenarios. There would be 24 test scenarios involving Small-sized pizzas (one with each pizza topping), 24 more tests with Medium-sized single-topping pizzas, 24 more tests with Large-sized single-topping pizzas, and 24 more tests with Extra Large single-topping pizzas. 

Alright, so we need to account for pizzas that contain more than one topping in our tests. And we’d like to reduce the number of tests in our 2-way test set solution. How do we accomplish these goals? While there are multiple strategies you could employ to ensure that pizzas with multiple toppings to appear in your tests, let’s start with the most basic and widely-applicable solution. 

First Recommended Test Design Strategy: Use Multiple Parameters (and Weight them)

We want to start by including each of the available toppings as parameters. This will ensure that each and every topping can appear on a single pizza together. But here's where it gets a bit tricky.

If we only include values of "Yes" and "No" for each of the toppings, the pizza ordered in each test will end up with an average of 12 toppings. As there are very few real world scenarios where a user would actually order a pizza with 12 toppings, we do not want so many of our tests to include this many toppings. But don't worry, this can be fixed with a simple weighting strategy inside each parameter.

Rather than having one "Yes" and one "No" as values for each pizza topping parameter, we can force the tests to include less toppings by including a larger number of "No" values than "Yes" values. Let's say we want an average of about 6 pizza toppings per pizza. We can accomplish this by weighting the values at a 3:1 ratio of "No" to "Yes".  The resulting inputs look like this.

In order to provide the tester with easy-to-follow instructions, let's change "Yes" to "add {Parameter Name}" and "No" to "do not add {Parameter Name}". 

 Let's consider what this plan design accomplishes:

  1. Multiple pizza toppings can appear on the same pizza
  2. The parameters are weighted so that a relatively realistic number of toppings (an average of about 6) appear in each test
  3. It only takes 38 tests to test every 2-way interaction in the system and achieve full pairwise coverage. Obviously we would want to include parameters like Size, Crust Type, and Sauce Type in the final version of this plan, but these additions would only minimally, if at all, increase the number of tests. 

If you are okay with the 3:1 weighting of the parameters, then great! Go execute your tests and be merry. But if you are trying to ensure that each test contains an even more realistic average number of toppings and are comfortable with increasing the number tests, you can further adjust the weighting ratio.

Increasing the ratio to 7:1 of "No" to "Yes" increases the number of tests from 38 to 143, but leads to an average of less than 3 toppings per pizza. You can continue to adjust the ratio of "No" to "Yes" to fit your exact needs.

In a similar vein of thinking, you can adjust the weighting of each individual topping based on the frequency it will actually appear on a pizza. Obviously customers will choose Pepperoni as a topping more than they would order Hot Sauce, so it might makes sense to weight these toppings differently. For instance, you could apply a 2:1 ratio of "No" to "Yes" for Pepperoni, while increasing the ratio to 4:1 of "No" to "Yes" for Hot Sauce. This would result in the most repeatedly ordered toppings appearing  more frequently in your tests without drastically increasing the total number of tests.

Second Recommended Test Design Strategy: Model a Maximum Number of Selections (Together with 1-Way/Mixed Strength Coverage)

If having an average of 6 “select all that apply” items per test scenario seems excessive, this test strategy may be preferable.

For this example, let's say we want to model a maximum of 4 pizza toppings per pizza. Your parameters should be Pizza Topping 1, Pizza Topping 2, Pizza Topping 3, and Pizza Topping 4, as you might expect. Where this gets interesting is deciding on the parameter values. You could simply list out all the pizza toppings for each  parameter and be done, but more than likely there is a greater chance of the 1st pizza topping being added to a pizza than a 4th topping. We want to account for this difference in our test plan. 

We can accomplish this by weighting the parameters to fit our exact needs, as shown below.

By adding increasing numbers of "No __ Topping" as values, we can ensure that a 4th topping appears less than a 3rd topping, a 3rd topping less than a 2nd topping, and a 2nd topping less than the first topping. We can also adjust the ratio of "No Topping" to "Topping" based on the frequency with which we want multiple toppings to appear in our tests.

This may seem like the perfect strategy, but there is one problem: using this strategy will result in a very large number of tests unless we incorporate one or more of the test-reduction-focused test design strategies described below. Even just the 4 parameters in the above example would require 677 tests in order to achieve 100 percent pairwise coverage.   

One way to avoid this problem is to use mixed-strength test coverage, utilizing 1-way coverage for each of the pizza toppings while keeping the 2-way coverage for any additional parameters. Mixed strength test plans allow you to select different levels of coverage for different parameters. This allows you to control the balance between increased test thoroughness with the workload created by additional tests. If we simply adjust the coverage level of each of the 4 pizza toppings to 1-way coverage while keeping the rest of the parameters at 2-way coverage, we are left with only 25 tests

While there are risks to using mixed-strength test coverage, this is an optimal test design concept to ensure that every pizza topping is tested at least once in this "maximum number of selections" type of model.

Third Recommended Test Design Strategy: Model a Maximum Number of Selections (Together with the Value Expansion Feature)

As a way of keeping the number of tests you generate small, the Value Expansion feature often works very well with the "Modeling a Maximum Number of Selections" strategy. When you’re using this “Modeling a Maximum Number of Selections” strategy, using Value Expansions is often a strong alternative to the strategy described immediately above of including the 1-way Mixed-Strength approach. 

Particularly when your underlying business rules treat certain values (or in this situation, toppings) in the same way, you can assemble those values into groups. For instance, if the pizza application makes no internal distinction between pepperoni, ham, and bacon and only cares that each of those toppings are meats, you can easily account for this inside your Hexawise plan. To do so, create a parameter value called "Meat" and include each type of meat inside a value expansion for that value.

An example of a test plan following this pattern is shown below. This plan generates only 46 tests to achieve 100% 2-way coverage, significantly limiting the total number of tests without sacrificing test coverage.

Please note that any time you use the Value Expansion feature, there will not be a guarantee that pairwise coverage will be achieved for all of the items in your Value Expansion lists. Therefore, whenever you use the Value Expansion feature, you should make sure to check the Test Plan Scorecard for the plan to confirm whether every Value Expansion item does, in fact, appear at least once in your test set. 

Conclusion

The “select all that apply” pattern occurs frequently. Knowing how to weigh various tradeoffs involved is important. Having a solid understanding of the pro’s and con’s of different test design approaches that you can use in this situation will allow you to make good decisions. This article has referred to 6 strategies and suggested that 3 are usually bad and 3 are usually worth considering.  

Option
Guidance
Pros
Cons
1 - Single parameter with all "Select all that apply" options
Not Recommended
N/A
  1. Every test will include only one item
  2.  Too many tests will be generated  
2 - Use multiple parameters (without weighting them)
Not Recommended
  1. Relatively few 2-way tests would be generated
  1. Too many options would be included in each test (e.g. 10 pizza toppings in each test)
3 - Use multiple parameters (and weight them)
Worth Considering
  1. Can achieve appropriate balance between number of tests and options included in each test 
  1. Aggressive weighting can lead to large amounts of tests
4 - Model a maximum number of selections (without other test-reduction strategies)
Not Recommended
N/A
  1. Often leads to large number of tests without test-reduction strategy
5 - Model a maximum number of selections (with 1-way/mixed strength coverage)
Worth Considering
  1. Relatively few tests needed
  2. Pairwise coverage achieved for all "standard" inputs
  1. No guaranteed that pairwise coverage will be achieved for "1-way" columns
6 - Model a maximum number of selections (with value expansions)
Worth Considering
  1. Relatively few tests needed
  2. Pairwise coverage achieved for all "standard" inputs
  1. No guarantee that pairwise coverage will be achieved for all items in value expansion lists