How to plan a successful remote mobile usability test (and avoid nightmares)

Remote mobile usability testing had a moment quite a few years ago. A trend was born when tools started coming out to enable testing prototypes or concepts on a mobile device. Suddenly, I went from being stuck to testing prototypes with those near me to a whole wide world of usability testing. This news was fantastic because we had users across the pond in Europe, and, as much as I loved traveling, it wasn’t feasible to do for each project and idea. 

The first remote mobile test I did was with a prototype on Invision. It was with a client we had run several usability tests with, so I felt comfortable with them being a guinea pig. But, let me tell you, it was horrible. Everything that could have gone wrong did:

1. The client hadn’t done remote mobile testing, so they didn’t read the part of my email asking them to download the Invision app onto a mobile device

2. The participant had no idea how to sign into Zoom on their mobile device

3. Sharing screens on mobile, once they finally got on to Zoom and downloaded Invision (at this point, we’re 15 minutes into our 45-minute session), proved to be impossible to explain

4. The hot spots on Invision were not great for the mobile experience at the time, and it was tough to get through the prototype meaningfully

After that complete blunder of a test, I decided to do remote mobile testing on a desktop. So, the participant would open the Invision link, which would show as a phone, and navigate through it on their desktop. Despite being less than ideal, I pushed through with this idea for some time. I had no idea how to set up a proper remote mobile test, especially for less experienced or new participants.

In a nutshell, I was terrified of remote mobile usability testing.

Luckily, many tools and guides have come out since then, making remote mobile testing less of a nightmare. However, it is still one of the more difficult methodologies to do well. It takes a lot of planning, a bit of luck, and a sense of “fake it ’til you make it” from the researcher. After running many (too many!) remote usability tests, I have finally found a rhythm.

When to use a remote mobile test

There are times when remote mobile usability tests are the best approach for a project, but not always. So before you go down the remote mobile testing rabbit hole, ensure it is the best method for your goals. These are scenarios when I choose to use remote mobile testing:

1. When I need to reach a diverse participant sample, especially in inaccessible and varied locations 

2. If we need to do research quickly across several different segments or personas

3. When I need more participants and need to lower the cost of the test to include more people (remote testing tends to cost less than in-person)

4. If I have a group of stakeholders interested in observing each of the sessions

5. If we need the participant to be in a particular context/environment, such as at work or home

6. Oh, and of course, if there is a global pandemic

Aside from these scenarios, it is also essential to look at the benefits and drawbacks of remote usability testing to see if it is the best fit for your project.

The pros of remote usability testing

• Allows you to test prototypes, ideas, or concepts with a broader audience and more diverse participant sample

• You can test with specific markets or locations without being near them

• The environment in which you’re testing (e.g. usually a participant’s home) is more realistic than a usability lab or them coming into an office

• At times, it can decrease the cost of testing due to less travel or necessary equipment 

The cons of remote usability testing

• Participants typically have to download software to run on their mobile devices (or multiple types of software)

• With many tools, you can’t see the participant’s hand gestures, body language, or exactly where they click—you can just see the screen changing

• Remembering to share permissions so participants can open a prototype

• Having to do a lot of “in-the-session” work, such as sharing the link with the participant (so they don’t see it ahead of time), sharing screens, opening the right software, etc.

• With specific tools, you cannot see the participants face during testing, and they cannot see you, reducing the rapport and potentially impacting the comfort of the participant 

• You have little control over the environment, including distractions or internet connectivity

I always consider the scenario and weigh the pros and cons before choosing this approach. That way, I know what to expect and can best plan for any obstacles or bumps along the road. 

How to run a remote mobile test

So you’ve decided a remote mobile test is the right path for you. Despite the hardships, they can be fun and extremely rewarding with fascinating insights from various participants. Here is how I set up and run my remote mobile usability tests.

Define your research question and goals

Before doing anything (for any study), make sure you have a well-defined research question and goals. These goals will help ensure you are indeed picking the correct method. 

Create the tasks

The core of usability testing is tasks. You ask participants to perform specific actions and essentially see:

• If the person can do this task (effectiveness)

• How difficult or easy it is for the person to complete the task (efficiency)

When it comes to any remote testing, you need to ensure your tasks are easy to understand and complete. Take the time to create your scenarios and tasks to ensure participants can properly navigate and execute them.

Decide on moderated or unmoderated testing

 I won’t go too deep on moderated versus unmoderated testing here, but it is a consideration. If this is your first remote mobile usability test, I recommend moderated testing because it will teach you how to write straightforward, actionable tasks. With unmoderated testing, you can get a lot of mediocre data, especially if your tasks are even remotely vague. When in doubt, start with moderated testing.

Choose a tool

You can do this step in parallel or even before the previous steps. It is impossible to conduct remote mobile testing without a tool. There are many tools to consider (check out this table and filter by remote research category). Some of the considerations I take into account when choosing a tool are:

1. Easy to use for participants. You can test this internally with your colleagues to determine how difficult it is to download and use the tool.

2. Compatibility with devices. Is the tool only compatible with Apple or Android? What devices will we need to test?

3. The participants’ privacy. Can you do things such as snoozing notifications while the participant’s screen is shared?

4. The necessary features. You should look for features such as screen sharing, recording, and observer capabilities 

5. Cost. How expensive is the tool, and how often will we use it?

6. Auxiliary features. Do you need features for recruitment or integrations with other tools you currently use?

My favorite tool for remote mobile testing? Zoom. 

Recruit participants

Ensure you are recruiting the correct participants with the necessary criteria. For example, focus on psychographics, not demographics, through a screener survey. Also, consider timezones when you are recruiting a global sample!

Do a dry run

Before going into the wild (or real-world), conduct a dry run or two with colleagues to ensure everything goes smoothly. Yes, this doesn’t mean everything will be perfect with the participant, but it will help you recognize any bumps or bugs before the actual test. In addition, a dry run will make you feel more confident and comfortable during real tests.

Conduct the test!

Finally, it is time to conduct the test after all the planning. Get set up about 10 minutes before the test and be prepared to refresh your email looking for “help” pleas from participants. Ensure all links work and you’ve turned the permissions on for the prototype. Whenever usability testing, make sure to observe what participants are doing in addition to the feedback they give.

And take a deep breath.

Lessons learned

As I mentioned, I have done this quite a few times, and here are a few lessons I learned along the way.

Plan ahead

Plan, test, and practice. Use the time before the test to try different scenarios where things may go awry. The best training I did with colleagues was fake bad scenarios. The colleague would have everything go wrong, and we would have to fix it during the session. 

Have a backup plan

I learned this the hard way—always have a backup plan. If you are conducting a 45-minute test and you’ve burned through 15 to 20 minutes trying to connect to the software, be ready to switch to plan B (or even plan C). If Zoom doesn’t work, have a Google Meet link ready; if the participant can’t share the screen, be prepared to share your screen and give remote access; if the software isn’t working, have the prototype ready to test on a desktop.

Think about your participants’ skills

I consulted for a company with quite a few older generation audiences who were critical to their revenue and market. Getting some participants to use a computer was a feat, but getting them to do remote mobile testing proved impossible. If you will waste time getting a participant base to use your software, consider a different method. It might be better to test in-person with fewer people than barely get any data from more. 

Record an onboarding video

Recording an onboarding video for people to watch before the session has been vital in my successful studies. I record a video (across devices) of how to download and open the software, share the screen, and what they can expect. When people watch this beforehand, the session always goes much more smoothly.