Between-Subjects vs. Within-Subjects studies

Setting up experiments for testing the usability of multiple user interfaces and conducting user research requires some planning. One thing you need to think about is whether to go for a between-subjects study or take a within-subjects study approach.

In this article, we’ll take a look at the differences between both approaches in the context of testing multiple user interfaces for usability. We’ll also highlight the key advantages of both methods to help you decide which one is right for your study.

Conducting quantitative usability studies

When you’re testing the usability of multiple user interfaces in a single study (with the same test participants), you need to find a way to assign test participants to the different conditions for which you’re testing.

There are two possible ways of doing it:

• Between-subjects study experiment: This study design involves assigning different user interface to different test participant. This way, each test participant interacts with one user interface.
• Within-subjects study experiment: This study design involves exposing each test participant to all of the user interfaces you’re testing. This way, each test participant will test all of the conditions.

For example, when comparing the user interfaces of two e-commerce websites X and Y to see how easy it is for users to use filters and add products to their shopping cart, you could go about it one of two ways:

1. Between-subjects study experiment: You assign each test participant a different website. They would then complete the task (use filters and add products to their site) only on the site to which they’re assigned.
2. Within-subjects study experiment: Each test participant is instructed to complete the task on both websites X and Y.

So, when your goal is to test the usability of multiple conditions (for example, multiple user interfaces) in a single study, you need to decide whether you’ll take the between-subjects approach or the within-subjects approach for your research.

Experimental design in quantitative studies

In the context of usability testing, the primary goal of quantitative studies can be to compare:

• Different iterations of the same user interface.
• One site to its competitor site.
• How users with different levels of expertise interact with the interface.

Quantitative usability studies typically involve independent variables and dependent variables. Independent variables are those that researchers (in this case, you) manipulate whereas dependent variables are those that are measured against changing independent variables.

Following our example, the two different e-commerce websites X and Y would be independent variables and the time it takes for the user to complete the task (efficiency of use) or how easy it is for the user to complete the task (ease of learning) would be the dependent variables.

In this case, the goal of the usability study might be to determine whether the dependent variables change or remain the same when we manipulate the independent variables. Meaning is it easier (or faster) for the test participant to complete the task on website Y than it is on website X or not. If so, we would conclude that site Y is better than site X.

Part of the experimental design of the quantitative usability study is to decide whether the study should be between-subjects or within-subjects.

Between-Subjects vs. Within-Subjects: Which experiment design is better?

Both approaches have their own key advantages. The experiment design you decide to go with for your usability study can depend on many factors including your independent variables, test participants, and the conditions for which you’re testing.

Here, we’ll lay out the advantages of both between-subjects studies and within-subjects studies to help you make an informed decision.

Advantages of Between-Subjects studies

Here’s what you need to keep in mind if you decide to take the between-subjects approach in your experiment design:

There’s no transfer of knowledge

Let’s consider a scenario: test participants who complete tasks on e-commerce website X will have gained some level of knowledge regarding e-commerce websites and the tasks they’re assigned before they begin testing the usability of website Y.

For example, they will develop a better understanding of how to use product filters, whether to expect a page refresh or an AJAX-based site or where the shopping cart icon will likely be placed after they’ve completed the task scenarios on website X.

So, even if the website Y is completely different than website X, the test participant still goes into the test armed with knowledge (and expertise) they didn’t have when they were testing the usability of website X.

With between-subjects studies, there’s no chance to run into a transfer of knowledge issue since each test participant is assigned a different user interface to test.

The sessions are shorter

Compared to within-subjects studies, between-subjects studies have shorter testing sessions. Test participants who are assigned one website to test will be able to complete the usability test faster than those who need to test two (or more) websites, making the between-studies approach ideal for remote usability tests.

Sessions are easier to set up

In the case of within-subjects studies, you need to randomize the testing order of the independent variables to minimize the risk of order effects on the usability study. What this means is that each test participant shouldn’t test website X first and then website Y. Half of the test participants should instead test website Y first and website X after.

Structuring it this way might sound simple for when you’re testing two user interfaces but as you increase the number of independent variables, randomizing the testing order of the independent variables will become more and more difficult.

However, with between-subjects studies, you don’t have to worry about randomizing testing orders since each test participant is responsible for testing the usability of a single user interface.

Advantages of Within-Subjects studies

However, if you decide to take the within-subjects approach in your experiment design, there are two advantages you should know about:

You’ll need fewer test participants

To gain valuable insight from the usability study, you’ll need at least 30 data points. As compared to a between-subjects study, a within-subjects study requires fewer test participants to get the same number of data points.  Why? Because a single test participant is assigned multiple user interfaces to test.

Following our example, if we were to go with a within-subjects study with 15 test participants, we’d be able to collect 30 data points in total. On the other hand, with a between-subjects study, we’d only manage to gather 15 data points which aren’t nearly enough to detect a difference between the usability of both sites.

It’s also worth noting that the more test participants you recruit, the more it’ll cost you. So, within-subject studies are also more cost-effective than between-subjects studies.

You’ll be able to minimize random noise

When you recruit test participants for usability studies, there are certain factors you have some control over and others that you have practically no control over.

For example, you might be able to recruit test participants that fall under a specific age range or have a particular level of expertise on the subject matter. However, you will not be able to control factors like the test participant’s mood or stress levels.

The benefit of a within-subjects study is that each test participant interacts with each independent variable. Following our example, this means each test participant will complete the task assigned to them on both websites X and Y.

However, if you were to go for a between-subjects study, your usability test would go something like this: a test participant who’s stressed out before the test will only interact with website X and a test participant who isn’t stressed out will only interact with website Y. Your usability study test results will likely have significant variance and would be more prone to random noise.

In this way, it’s easier to minimize random noise for within-subjects studies than it is for between-subjects studies.

Conclusion

You can conduct usability testing studies using either the between-subjects study approach or the within-subjects approach depending upon your goals and resources. Let’s quickly recap the key advantages of both methods:

• If you’re short on time, you might want to go with a between-subjects study.
• However, if your budget is limited, you might consider going with the within-subjects approach and hire fewer test participants to keep costs low.
• If you’d like to minimize the transfer of knowledge across different testing conditions, the between-subjects study would be the right option for you.
• And if you’d like to keep random noise as low as possible, we’d recommend going with a within-subjects study.

Between-subjects studies or within-subjects studies – which approach do you prefer for user research? Share your thoughts in the comments section below.