How to Conduct Usability Testing During a Pandemic?

Before releasing a product, or even after its release, it is a good practice for businesses to evaluate their products’ usability and effectiveness of use through usability testing. Usability testing allows us to identify usability issues as it’s crucial because it equally impacts all users, regardless of their ability.

During the COVID-19 pandemic, the Idea Theorem™ worked with several companies to conduct remote usability testing to ensure that their product is usable and aligned with their expectations.

A remote moderated test is when a moderator provides the tester with tasks of testing a product, while in the comfort of their environment. In other words, the moderator and the testing are in the same virtual space, at the same time. As the tester works through the task, the moderator will observe and ask questions to understand their thought processes, and frustrations to gather more data.

In this article, we share how we conduct usability testing sessions during a pandemic to help you understand that you can also take the initiative to help your users thoroughly enjoy and be satisfied with your product.

Phase 1: Planning 

Define Goals and Objectives

Before gathering insight from the usability tests, the design team conducted several team meetings with the client to understand their business goals and the indicators of success that can be achieved from the usability tests. Often clients would specify specific flows or features in the product that needs to be tested. If not, the design teams will work collaboratively to help clients identify areas of concern that should be addressed through usability testing.

Writing the Test Plan 

After discussing with the client, we focus on creating the usability test plan. The usability test plan is like a blueprint that contains information such as:

• The Purpose of the Test: An overview of the purpose of the test, to inform both the team and future readings about the test that was conducted, who conducted and additional information.

• The Methodology: A brief introduction to how the test was conducted.

• User Profile:  An overview of the user profiles, that will be referred to help recruit the right testers.

• Moderator Script: A script that a moderator consists of instructions, topics, and questions that will be utilized for all test sessions, to ensure that all tests are consistent.

• Task Cases: Task cases are written based on what typical users might go through while using the product.  These tasks focus more on a user goal that has an endpoint, in which users are expected to complete.

• Questions (Pre-Test, Post-Task, and Past Test Questions): An outline of open-ended questions that the moderator should refer to when asking questions to the testers.

Define Resources 

Finally, the design teams identify the resources and tools to utilize for usability testing. As mentioned previously, the design team used remote moderated testing during the COVID-19 pandemic. To conduct a successful remote test, we used the following tools:

1. Lookback: A third-party virtual space that allows us to talk to users, see what they are doing and record their interactions with the product. Using these types of tools will enable us to interview testers anywhere and anytime.
2. Zoom: A video conference call tool used to start and join video conferencing from any device (desktop and mobile). Participants can join a meeting from an invite, share their screen and mute themselves. Zoom calls provide participants with a waiting space if the moderator has not joined the meeting. Moderators can also record meetings.

1. Gotomeeting: Similar to Zoom, this software allows participants to join video conferences on both desktop and mobile. Moderators can also record meetings.
2. Google Meets: Google Meets is handy if you do not want your participants to download any software on their desktop, simply open the invitation link and join the video call. Although, Google Meets may not support video recording.

Gathering Participants 

After the clients provide details about their users’ profiles, we find participants using the following methods:

1. The client provides a list of testers that can take part in the tests. Our team then directly contacts the testers on behalf of the client to set an appointment.
2. The design team sets up ads (for example, Linkedin) to recruit participants.
3. Gather testers from well-established multicultural and general population databases. These databases help us target and recruit highly specific testers.

Phase 2: Facilitation 

Usability Testing Sessions

In this phase, the design team conducts usability testing sessions. The moderator and the tester will communicate with each other throughout the test. The moderator will guide the testers through the purpose of the test, the task cases, and the questions. Remote and in-person usability testing differ in the following areas:

• In-person, Moderated Testing: Tests where the moderator and the testers are in the same physical environment, face-to-face interaction. 
• Remote, Moderated Testing: Tests where the moderator and the testers are in the same virtual environment, video call interaction. 

The benefit of remote moderated tests is that we can invite many observers to the tests to watch. Still, the tester will be unaware of the observers, making them feel less intimidated.

Analyzing Data

After the completion of the test sessions, the design team analyzes the results from the tests and draws conclusions (why do the issues exist?). For the task cases, we focus on the following:

1. Effectiveness: How effectively the user successfully achieves each task.
2. Efficiency: The average time it takes for participants to complete each task.
3. Satisfaction: How satisfied the users were with the product.

We organize the task case data based on issues, actions, and comments per task. The usability issues are prioritized by ranking them based on impact (if solved) and the severity of the issue (critical, serious, or minor).

We show the highest priority issues first, including the evidence and presenting the solutions. In addition, we provided insights about common usability issues and a UX analysis from a UX perspective.

UX recommendations are created to solve the identified usability issues. We also provided UX best practices to leverage to enhance the user experience. Apart from the usability issues, we also present the positive findings from the test sessions.

Want to drive better results from your product?
Download the Product Manager’s Guide To UX >>

Phase 3: Final Report 

After the analysis of the data, a final report is created to present the results of the usability testing sessions. When writing the final report, we focus on writing the report in such a way that if it is shared with a person that does not know about the project, they can understand it quickly.

The usability report provides details about the entire usability testing process such as:

• The Purpose of the Test: An overview of the purpose of the test, to inform both the team and future readings about the test that was conducted, who conducted and additional information. 

• The Methodology: A brief introduction to how the test was conducted. 

• Recruited Testers: The names, occupations, ages, and contact information of all the recruited participants.

• Overview of Task Cases: A list of all the task cases presented to the testers.

• Task Case Results: Results of the task cases. This includes the number of errors, successes, or failures of the task per participant.

• Tester Feedback: Feedback from the testers from the survey that was sent after the test session.

• Identified Behaviour Issues: A prioritized list of behaviour issues based on severity.

• Recommendations: A list of recommendations to solve usability issues.

• Observations: A detailed overview of the observations from the test sessions. We keep this section detailed so the reader can refer to it whenever they review the results of the task cases.

What Went Well 

Working with a team of designers helps speed up time in preparing the reports and it’s beneficial when assisting in the testing sessions.

Sometimes as moderators we may miss key information to identify, but another designer can help keep you on track to ensure all testing sessions are consistent.

Using a third-party platform that specializes in remote testing is more effective than using standard virtual meeting spaces since the recording tools and webcam are implemented into the platform.

In addition, these are more professional to use as it creates an introduction and a waiting space for the tester while the moderator sets up.

Things to Keep in Mind

Learning Curves

Leave some dedicated time to learn about the resources that you will be utilizing for the usability test. Talk with a customer representative to conduct a product demo and ask questions. You should be well aware of the constraints, shortcuts, and features of the resources to help you conduct test sessions with ease.

Help Testers Setup

Spend some time prior to the test session to help your testers set up the virtual space. Ideally, you do not want to use your test session time with the setup process. Otherwise, you can create a video tutorial on how to set up the virtual space and share it with your testers.

Product Demo 

Have your client walk you through the entire platform and spend some time yourself using the product itself. You want to know all the ins and outs of the product, nothing should surprise you as your testers interact with the product. Testers may ask about the product, and you should be ready to answer any questions.

Conduct a Mock Test

Conduct a mock test internally with your team, to ensure that the usability runs smoothly. Be aware of inconsistencies within the test, and make sure that the third-party platforms work well with your test.

Conclusion 

Overall, the remote moderated usability sessions were successful. Moving forward, we would continue to use remote moderated usability testing as part of our design process, along with in-person moderated usability testing.

It is beneficial, especially when conducting usability tests outside of Canada.

You can also download the whitepaper here.

—

What’s Next

Idea Theorem is an award-winning design & development agency based in North America. Through our empathy-driven approach, we have crafted digital products that have positively impacted over 10 million users. Our mission is to shape the digital future by delivering exceptional experiences. Contact Us if you have any questions; we will gladly help you.