Studies have shown that in software development the cost of a bug increases by a significant magnitude for each successive stage. This can be translated to, that if it costs a euro to fix before implementation, it costs €10 if it’s found in QA and €100 to fix when it’s in production.

To be honest, we don’t know if you can times the cost by 10 in each step, but we know there is a lot of money to save both in general cost and salary if you find the bugs as early as possible – this includes user errors as well.

Furthermore, testing with users from an early stage ensures that we are developing what our users want and need and that the solution is something that they understand using. In the end, this entails greater user- and customer satisfaction. The beauty of user testing is that you can start as soon as you have some tangible material.

In Preely you can upload pictures of your napkin sketches or whiteboard drawings and create an interactive prototype, or you can upload your prototype from your preferred prototyping tool, keeping the cost of user testing low. If you want to try it out, you can read more here.

User tests

We are speaking to a lot of UX researchers and -designers every day and we very often run into assumptions like user tests need to be conducted in a lab to have valid results, or it costs a fortune and takes a lot of time to conduct them. These assumptions can be right. However, as with everything else, this is very context-dependent and therefore not necessarily true. Often you do not need a dedicated usability lab or to spend thousands of euros on conducting moderated tests to have the insights and data you need. It can be done less expensively.

Let’s look into it!

In general, when considering user testing, we are facing three major hurdles:
– They are expensive
– They are time-consuming
– You have problems with access to test participants

Expensive

Conducting user tests in a lab can sometimes be the right approach, especially if it’s important to keep the test variables constant between sessions. But do user tests per default need to be conducted in a lab setting? No! Actually, you often have more valid results by conducting the tests ‘in the wild’, where the product is actually to be used. This type of testing brings a lot of valuable insights into the use context. Often you’ll not see the user use the product as intended while being in the lab and being surveilled by multiple researchers and notetakers. Another cost is the expense for test participants (see further down).

Time-consuming

User tests take time. Yes, especially moderated user tests. Often the salary of the moderators is not added to the budget when calculation the user testing cost. Let’s look into the actions needed to conduct a user test – and again this is very context-dependent! Overall, you need a prototype, plan the test, conduct the test, analyze the test, and convey the results.

Prototype
Creating a prototype can take from minutes to months.
We recommend creating your prototype as simple as possible, but still complex enough so you can answer your assumptions or hypotheses. Note, that often you do not need a whole flow, or a fully functional prototype – often it is enough to have a snapshot of a flow or specific feature.

How to work with prototypes in Preely: If you already have a prototype you can export it to Preely and set up your test. Or you can take pictures of a whiteboard session or sketches on a napkin and import them to Preely and create an interactive prototype. You can read more about this in our Preely Academy.

Planning
Before you start testing, we recommend that you define which assumptions or hypotheses you are testing and make sure that you are testing the right variables with your prototype. Furthermore, you’ll need to define different tasks and questions. Need help or inspiration, read more here.

When planning a moderated, in-person test, there are a lot of logistics that need to come together. You need a location, your participants should be booked individually (and they often only have time during the evening), and you might need a notetaker.

Planning a user test with Preely is quite simple and fast:
1. Define your tasks and questions
2. Share the test via a link or via your user panel
3. Lean back and let the platform facilitate the test while the results are automatically collected

No problems with logistics, timing, etc. Your participants conduct the test, when and where it suits them. This is a great benefit for your participants, read more here.

Moderating
When conducting a moderated test, you need to be there moderating each test. Let’s say that you have 10 participants and the test is one hour long. You need a bit of time in between sessions to prepare before the next session and capture observations, ideas, etc. from the previous session. Let’s say this takes 30 min. Furthermore, you need a notetaker. If you for example conduct ten moderated one-hour-long user tests you spend at least 30 hours. This ready add the user testing cost.

How moderation works in Preely: We call our tests unmoderated, but in fact, the platform facilitates and moderates all tests automatically. This means you have your hands free for other tasks that can mature the UX of your product, read more here.

Analyzing and conveying results
Last but not least we need to make the analysis of the sessions and share it. If you have moderated the tests, you will have a lot of notes with observations, quotes, etc. Analyzing this type of qualitative data can give a lot of valuable information, but it takes a lot of time to do. Furthermore, if you are more than one taking notes, you also need to consolidate your data. This can take from a couple of hours to days. When you have spent all this time and resources conducting a moderated test and making the analysis, you also want to convey as many details as possible to your peers, and maybe create a big report. Again sometimes this is a good idea, especially when working on new grounds. However, looking into more day-to-day user testing, where you e.g. are testing a new flow or a new feature, that approach can seem very extensive compared to the value you gain from it. Here you just need a one-pager containing an executive summary with key findings and recommendations.

Working with analytics and results in Preely: Preely collects the most common performance metrics, making it easy by a few clicks to access data and assess performance. Enrich this data with the self-reported data from the questions you have asked during the test and you have a great foundation for making informed decisions. And it’s easy to find key metrics to convey to your stakeholders. Read more about the different metrics you can collect with Preely.

Nearly two weeks of man-hours spent on moderated tests
We were part of a research study conducted by students from Aalborg University and a big finance organization, both Danish-based [1]. Here it was found that preparing, conducting, and analyzing a moderated, in-person, think-aloud test with five to seven participants took between 52 to 64 hours. This corresponds very well to other studies e.g. Expand the Room and NN/gr .

Time spent working in Preely: To learn how much time it would take to get similar results in Preely, the researchers created a very similar test. The test in Preely took nine hours in total. Hence, they saved between 43 to 55 hours. Hours that were freed up to do other tasks, without compromising with the original process or quality of the product.

Access to test participants

The third major huddle is access to test participants. Where can we find the right segments, without spending a fortune? It’s important to test with the proper segment(s) for your business or product, however for each test consider:

– Are you testing general usability?
– Are you testing target-specific user experiences?
– Are you targeting the general population or a specific group of individuals?

Testing general usability
If you are testing general usability, everyone can be your test participant. Hence the cost of participants is low.

Hired participants in Preely: When using Preely you have the opportunity to hire test participants directly on the platform for as low as €2.50 per participant, read more here.

Target specific user experiences
When you are testing target-specific user experiences it is very important to recruit within your target group and have a lot of focus on segmentation. Recruiting participants for these types of tests can often be tough and costly. When working with a very specific target segment, and if you wish for them to be physically present for a test, we have experienced pricing of up to €300 per participant.

Working with user panels in Preely: We recommend hosting your own user panel on the platform . This way you always have access to the right user segment(s). This is also a great way to build more engagement with your existing users and customers. Often they are happy to help to improve their own experiences and it is often exciting for them to get a glimpse into your roadmap. This way you do not need to pay an external firm to recruit participants and the only cost is the incentive you give the participants. A note here is that you should be aware of tax regulations in your country. In Denmark, the limit before having to pay tax is approximately €50 and it’s quite common to pay this to a participant for a one-hour test.

So what does a user test cost?

We have said it before and we will say it again. User testing is very context-dependent. How to test depends on the product you are working on, but it also depends on where you are in the development process, and which types of data are important to you at that stage. So, this is something you definitely need to consider before choosing an approach.

Cost overview

* average user researcher salary of $114,000/yr

No matter which approaches you choose, remember that a bug or user error found before implementation costs way less to solve than if it’s found in production. This is definitely something to take into account and maybe add to the overall budget and maybe use as an argument for having more funding for user tests. Furthermore, by user testing, you ensure that your product resonates with your users or customers and that they understand using it. In the end, leading to more satisfied end-users and customers.

——

Source
[1] Pedersen, J.H.H., Sørensen, M. (2020) Asynchronous Remote Usability Testing in Practice: Exploratory Interviews with IT Professionals. Department of Computer Science, Aalborg University, 2020.