Moderated vs. Unmoderated User Testing

Moderated vs. Unmoderated User Testing:
Which is Right for You?

Rumzz Bajwa | Reading time: about 8 min

Both moderated and unmoderated usability tests prove to be effective solutions to getting accurate feedback on an ongoing project. Which should you use?

When it comes to usability testing, there are two main methods to choose from: moderated and unmoderated. But what is usability testing, and which of the two design techniques should you use?

Comparing moderated vs unmoderated usability testing is important when you’re planning the customer’s entire journey on your website or application. You need to make sure that your interface is intuitive, your buttons are in the right place, and even your copywriting is on point. 

The end goal for any usability test is to help customers see your company across all platforms, leading them to make a purchase of your goods or sign-up to schedule a meeting. If your design makes it difficult for the customer to find what they’re looking for, you might lose your chance to make a sale.

So before we dive right into the differences between moderated and unmoderated usability testing, let’s identify what a usability test is and why you need it.

What Is a Usability Test?

For designers, usability testing means evaluating a product or service based on how a user assesses it.  

During a test, people who will be participating in the test will look into every aspect of the product before they are coded. An outsider (usually called a facilitator or moderator) takes notes of the comments and suggestions, allowing development teams to fix any usability issues before the product goes out to the public.

The primary goal for any usability test is to analyze the performance of the website or the application and to identify any issues that may affect the user’s experience. Both quantitative and qualitative data will be used to determine how effective the product is and which aspects warrant minor changes or a complete overhaul.

To validate the effectiveness of a product, there are two ways to do it: through moderated and unmoderated usability testing.

Moderated Usability Testing

Moderated usability testing simply means that there is active participation between a facilitator and the participants. 

The facilitator guides the participants on how or what to test, and records their behaviors, comments, and provides answers to their questions. All of these are happening in real-time, meaning that both the moderator and the participant go through the usability test together.

Moderated usability tests can either be done remotely or inside a lab. If the usability test will be performed in person, or virtually, both parties don’t necessarily have to be in a lab to go through the process. Conducting the moderated test in quiet room, or via a virtual conference tool such as Zoom or Teams will suffice, as long as you have all the tools you need, from different gadgets to recording devices to monitor the test. 

Unmoderated Usability Testing

Unmoderated usability testing, in contrast with moderated usability, allows the facilitator to provide the tasks to the participants beforehand and lets them go through them at their own pace and in their native environment. Each party can work on their tasks on their own time and at separate locations.

You might be wondering how unmoderated usability testing becomes successful when the entire process is unsupervised, but most participants prefer to work on their tests and provide feedback when they are not under pressure to do so.

Moderated vs Unmoderated: What’s the Difference?

Although both moderated and unmoderated usability tests prove to be effective solutions to getting accurate feedback on an ongoing project, say a website, their differences boil down to three C’s: convenience, costs, and control.

Convenience

In terms of convenience, most developers would opt for unmoderated usability testing simply because they don’t have to bring together all respondents in one location and finalize a schedule that works for everybody.

Facilitators simply have to provide detailed instruction on what the participants need to work on and how they should go about it. Once that information is provided, facilitators can work on other tasks while waiting for feedback.

For the participants, there’s added convenience, too: they get to do their tests at their own pace, on their own time, and in their preferred location. Unmoderated testing leverages participants the time and focus they need to accomplish the task and submit their input on a specific deadline.

On the other hand, moderated usability testing requires that all participants meet at a designated time and location designated location to go through the tests. It’s convenient for facilitators who need to get urgent feedback, but it can be challenging when trying to bring everyone together.

Costs

In a recent study conducted by the Nielsen Norman Group, an unmoderated usability study with five participants may be 20 to 40 percent cheaper compared to its moderated counterpart. 

In addition to costs, there are savings in terms of time, too: in fact, the study found that facilitators may save around 20 hours of time when they do an unmoderated test.

Despite that conducting an unmoderated usability test is more affordable than doing it otherwise, you have to take note that there may be other costs involved, too. 

For example, if you’re running a remote unmoderated usability test, you are more prone to getting results from “professional” participants. These are the people who regularly participate in multiple studies to earn from them. You are also more prone to getting responses that don’t provide any value to your research. In some instances, you might even encounter cheaters who lift their answers from across the web or may have coordinated with other participants to use their notes and lazily do the test just to get it over with.

In moderated usability testing, there definitely are costs involved when you conduct the research. However, the benefits are getting accurate results from participants who really take their time to assess your product and work with you throughout the process.

Before you perform any usability testing, it’s important that you factor in the costs involved. Figure out everything you’ll need for the experiment, such as the resources and number of respondents, then see if your budget works around it. There are low-cost usability testing tools like Crazy Egg and Qualaroo that you can use to support the study.

Control

Lastly, comparing moderated vs unmoderated usability tests will always touch on control. As we discussed how the two processes work, you now have a general idea of how they work.

When you conduct a moderated usability test, the facilitator or moderator has control over the process. If that’s you, you have the power to dictate how the entire test should flow and what questions you need to have answered. You get to ask the questions that are not on your list, and you get straight answers from your participants to maximize the time you have with them.

Conducting a moderated test, whether in-person or remotely, also allows you to guide your participants so they won’t lose their train of thought. You also give them the advantage to ask questions as they go through the test.

You also get to understand where your participants are coming from on a much deeper level, just by getting their reactions and initial feedback. You get accurate results from what they instantly thought about the product or any aspect of it.

On the other hand, an unmoderated test won’t allow you to ask participants any follow-up questions in real-time. Sure, you can always shoot them an e-mail when necessary, but you won’t get instant responses, therefore delaying the outcome.

Which Is Right for You?

Usability testing, whether moderated or unmoderated, is crucial to the entire UX process. You discover any loopholes that your developers may have missed, giving you the chance to improve them before the public sees them.

When deciding which of the two tests to go for, we recommend that you assess your available resources and see which makes more sense for you. Do you have enough time to look for participants and get their schedules? Do you have the budget to conduct the test in a lab? Do you want to take control of the entire process and guide your participants through it? 

If the answer to all the questions above is yes, then you definitely should go for moderated usability testing. Otherwise, if your time and budget are limited and you want to get results in your timeframe, then you can opt for unmoderated testing.

No matter what usability test you’ll go for, just make sure that you’re able to cascade the instructions very clearly to your participants. Make sure that they know why you’re conducting the test in the first place, and value their responses no matter how different they are from your expectations.

Article originally published December 2021 in the DZone Performance Zone Blog