When user testing it’s often a good idea to define one or mere research questions to set a direction for the test.

Having one or more research questions also makes it easier to define and write the user tasks the test participants need to perform during the test.

Define Research Questions

Defining one or more research questions prior to the test, to set a direction for the test, can be quite beneficial. When working with quantitative tests it’s often an advantage to define success criteria as well.

For qualitative tests, a research question could be: ’How does the users navigate on our website and what would they like to change?’. The outcome from this, would normally be qualitative and consists of statements and maybe the different paths the participants have taken.

For quantitative tests you need to be more specific and add success criteria to the research questions. A huge benefit with quantitative tests is the opportunity of using the test results as KPIs and comparing design iterations over time. Examples of quantitative research questions including success criteria could be:

Example 1:
Research question: How satisfied are our users with our new signup flow on a scale from 1-5?
Success criteria: Rating 4.3 or above.

Example 2:
Research question: Are our users able to book a table on our site?
Success criteria: 75% of the test participants should be able to book a table within 1 min.

Example 3:
Research question: Are our new setup flow better than the old one?
Success criteria: How have you measured usability and UX on the old one – use these metrics as benchmarks.

It’s a good idea to discuss research questions, success criteria and in general what to test with the development team, developers and stakeholders.

Write User Tasks

There is a difference in how to write tasks for qualitative versus quantitative tests. With qualitative tasks you want to gain insights, hence qualitative tasks are often explorative and open-ended. With quantitative tasks you are testing to capture metrics, hence quantitative tasks are specific and focused.

General guidelines
When writing tasks, you want to give your participants tasks they would normally perform with your product. The tasks should be as realistic as possible to keep the test ecological. It’s also important that you do not provide clues in the tasks e.g. refer to the direct wording of a button or describe the exact steps needed to complete the task.

It is also preferable that you are as emotionally neutral as possible and you should not make assumptions about your participants’ life and relationships – if possible use a neutral language. And avoid tasks where your participants need to use external tools like calculators, accessing their email, etc.

Last, but not least, always pilot test the tasks. If you do not have the opportunity of conducting a pilot test with real participants, then use co-workers, friends, family, etc.

… and remember, it takes time to write good tasks.

Qualitative Tasks

Normally, qualitative tasks are open-ended and exploratory – you want to collect insights.

Set the scene
You should set the scene for your participants, but you do not need to go into details.

Example:
Introduction: One of your friends has sent you a link to a website.
Task: Open the link and take a look around. Then tell me what you think.

When working with qualitative tasks it is okay to change wording of tasks and whole tasks along the way. It’s okay that the participants do not perform the exact same tasks. This is not the case for quantitative tasks.

Quantitative Tasks

Since quantitative tasks give you information on your metrics, it’s important to make sure there is only one way to perform the task. The task should be very specific, if the task is too broad and open-ended, you risk that the internal validity of your test is contaminated, meaning you do not measure the right thing, and you will have trouble in defining, if your success criteria are fulfilled or not. Keep in mind to provide as many details as necessary to keep the task narrowed and focused.

Example:
Introduction: You are visiting our website for the first time. You want to book a room.
Task: Book a Junior Suite for two people at our Hotel CPH from March 14th to March 17th. Use this email: preely@preely.com to sign in..

Use dummy data and make sure tasks can stand alone
To minimize variability and not having participants sharing personal data provide them with fake credentials for any task where they need to enter personal information. The same goes for input fields in general – give the participants dummy data.
Make sure that tasks can stand alone. There are three reasons for this: 1. If the participant fails at the first task, you want them to be able to continue to the second task without any problems. 2. If the participant fails at one point, you’ll have obtained data up until then. 3. You might be able to randomize the order of the tasks.

Tasks should have one success criteria
Do not combine two tasks into one! Working with quantitative tests, you need one clear success criteria for the task to determine if the participants succeeded or not. If you have multiple success criteria in one task, you’ll not be able to determine the success if your participant fulfills one but not the other. Hence, split tasks up as much as possible.

Don’t change tasks when the test has begun
When working with quantitative tasks, it’s important that each participant is doing the exact same task. The conditions should be held constant. Therefore, you should never change tasks or wording of a task mid-way through a test. If you do so, you potentially contaminate your data. That’s why we at Preely lock your test, when your first participant has stated the test.

Remember – it takes time and practice to define and write both good research questions and tasks.
Happy testing!