User panels where to start? – create research plan

Once upon a time, it was enough to have the newest features in order to be market-leading. This is not the case anymore. No one cares about 1000 features, people just want the ones they need and they should work seamlessly. The best way to ensure this is by having user involvement throughout the development process.

There are different hurdles when it comes to user involvement, and why organizations often struggle to do it. There are of course a lot of different things going into this. However, overall it’s more often than not a matter of time and resources, and access to users.
Regarding the former, you can read more about how we believe it can be solved here. Then we have the latter.

Don Norman once stated:

‘User-centered design means
working with your users all
throughout the project’

To be honest, I have never experienced succeeding with this in any of the organizations I have worked with. Sometimes we have user involvement at the beginning of the development process to form user needs and user requirements, and maybe a usability evaluation at the end. Everything in the middle has been black box from a user perspective. As stated in the beginning there are many answers to why this is, but one of the major ones is access to participants. Hence a user panel can be a great idea.

Benefits of having a user panel

The benefits of having a user panel are many. Before you think – okay, they have a happy-go-lucky attitude to this, we acknowledge that there also are concerns, and we will go through them in a bit. But let’s start with some of the benefits:

Minimize recruitment time: Instead of having to start a recruitment process from scratch every time we need user involvement, we have them at hand, when we have a panel. This also allows us to work much more agile, as well as test as soon as an assumption arises.

Have the target group at hand: I have worked in organizations where we could not just ‘quickly’ go out and recruit people from the street due to very narrow user profiles. Therefore, it was worth gold having collected relevant people in advance.

Significantly cheaper than external recruitment: External recruitment can be really expensive. I have worked in places where a test participant cost between 2200 and 2500 DKK, as these were segments that were not in the common databases and profiles who were not interested in money for testing.

Can be used for many different purposes and methods: A user panel can be used for most things and different methodologies. Of course, you have to be aware of biases (which I will get into in a bit), as well as where you are in the development process and what you are testing – is it a new concept, is it usability, or is it the overall user experience?

Can be used to get your users closer to development and create more empathy: This is, to be honest, not due to the panel. But when it suddenly becomes easier to involve users, we fully see the benefits of user involvement – namely that we get users closer to development and thereby create much more empathy.

Hosting a user panel at Preely

Hosting a panel at Preely makes user access and user involvement faster, better, cheaper, and more dependable. Note that the platform is GDPR compliant, 100% EU-hosted, and we ensure disaster recovery.

Read more here

Concerns when using a user panel

There are things we should be aware of when using a panel. We have collected the ones we hear the most when speaking to designers and researchers. Let’s go through them.

Participants become expert users
We may risk that our panelists are biased toward the products we test. Have they been involved in testing the concepts from the beginning, is it then a good idea to test with them later in the process? Sometimes it’s not a problem, other times it is. This can be solved by:

• Having a research plan (we have created one and you can find it below)
• Acknowledging that there will be biases
• Evaluating hypothesis and findings with other team members
• Considering triangulation of both data collection types and recruitment channels

The advantage of using a platform like Preely is that you always have an overview of which panelists have participated in which tests. Hence, it’s fairly easy to apply bias control. This also gives the advantage that your participants feel seen, as you can always refer to other things they have participated in before, and you know which questions they have already answered in the past.

The panelists are not representative of the overall user group
We know that people who willingly sign up for a panel might be different from the people who don’t. This might of course screw the data and insights. Remember, that this can take different forms. From panelists being overly positive toward the product to the complete opposite. It can be handled similarly to when you have a non-response error during surveys (where people who don’t respond differ from the people who do). Here it is important to consider triangulation when it comes to where you recruit participants. However, you come a long way by being aware and taking these biases into account when analyzing.

We end up testing with the same participants again and again
This one is a bit related to the previous ones. What is the risk of testing with the same panelists again and again? Here we need to take a look at governance – because how often do we as an organization think it’s okay to use our panelists? And talking about governance, it is also important to talk about the GDPR space – how are your organization’s rules? Are you following the standard procedures, do you have something customized, etc.? At Preely, we take this into account and you have the opportunity of setting custom rules around your panel.

Participants only partake due to money
What is the participants’ motivation to be part of your user panel? We had an assumption that many were part of the panel to get discounts, make money, or otherwise ‘get something material’ out of it. One of our customers – youSee, has researched this and found that people are part of their panel due to the following reasons:

• Sharing their experiences with youSee and influence the products
• Getting involved in co-creation and actively contributing to product development
• Being first-movers, as well as being the first to try new solutions

We do not know what we want to know about the participants
We often meet the concern that you do not know what kind of questions you would like to be able to segment on when the panel is to be used. This often becomes a stopping block to getting started – because what is important to know? We often see that the recruitment questions are endless and the potential participants drown in questions and do not get through the signup flow, resulting in a large drop-off rate.

It is therefore important that the process of joining the panel is short and simple, we ask a handful of key questions, and then progressively get more and more information about the participants. This can e.g. be done continuously via various tests or you may have a strategy to send out follow-up questions e.g. every month – and thereby also ensure activation. So begin by asking yourself the question: What is the minimum information you need to know in order to use the panel? – then start there.

How to get started

It can be quite nerve-racking to set up your panel – I think all of our customers have been through various stages of panic when they started setting up their panel.

Start small – get your hands on it and get to know the platform. If necessary, start with an internal panel, e.g. a stakeholder panel. Then find out who you need in your panel – is it existing customers, or someone who does not know the organization? This also depends on what you intend to use the panel for. It can be beneficial to create a research plan on how to work with the panel and the panelists. We have created a list of things you can discuss within your team or organization before starting to recruit panelists to your panel (see below). When you have that figured out, you are ready to start recruiting.

Then the next question arises – where to recruit panelists?
We have customers who recruit via their website and via newsletters, and we have customers who recruit via pop-ups, directly through their own products (including youSee), as well as all forms of SoMe. I have experience recruiting customers from app stores. Let your imagination run wild.

The good thing about using our platform is that you can create your recruitment page directly in Preely and then share the link where you think your target segment is located.

Download the research plan 🎉

Having a research plan for how to use the panel is worth its weight in gold. We have collected areas that might be of value to you. What a research plan could include can be found here: User panel setup.