Preference testing is a test method that can be used when you want your participants to choose which variation they prefer. It can be between two or more versions. The simplest preference test is to ask the participant ‘which of these do you prefer?’. This will give you an idea of which design resonates most with your participants.
You can also assess other things, e.g. which design they feel is most trustworthy, which design looks the most modern, etc. Preference testing can also be used for testing copy, e.g. emails and web content. Here you get insights on which copy the participants feel convey a certain message or which text is easiest to understand. In both cases asking follow-up questions are important to understand why they prefer one over the other. Preference testing is normally used early in the design and prototype stages and can be conducted with simple wireframes of the design variations. When working agile, it can be used iteratively as a design evolves. For marketing purposes, it is used throughout the process.
Data and insights from a preference testing
When conducting a preference test you can collect both quantitative and qualitative data. Qualitatively you will have the number of participants who selected each design, which can be used for a simple design comparison. If your sample is big enough, you can calculate if there is statistical significance or not.
Qualitatively you will have insights from the participants on why they choose one over the other. Again, with a big enough sample size, you can e.g. make a thematic analysis of their responses. Read more about sample size here.
Preference vs. AB testing
Sometimes preference tests are confused with AB-tests, but they are actually quite different. As stated above preference testing is normally conducted early in the process, and never with a whole design or flow. AB-testing is normally conducted much later in the process and with more final designs. Very often the test includes different tasks the participants are to perform. You can read more on how to set up an AB- or multivariable test in Preely here.
Furthermore, the two test types focus on different aspects of a design. A preference test focuses on perceptions and opinions, whereas an AB-test focuses more on behaviors. Hence, preference tests often have a more qualitative nature, while AB-tests are more quantitative. Read more here about qualitative and quantitative data types.
Keep this in mind!
A preference test is a very simple test method. It is easy and fast to show participants something and ask which they prefer. However, you should be aware of what you expect of your participants. Asking a question like, ‘Which design is best?’ would give you an answer, however in this case you are asking them for a design opinion – and they may not be qualified to judge which one is best, but they can choose which they prefer. Remember a preference test is a desirability test, it does not give any context or any indication of how a design will work or how it will function. Also, note that it is relatively easy for the participants to choose which they prefer, often it is much harder to tell why that is the case.
We sometimes see designers solely rely on preference testing when making design decisions, which the method is not robust enough to handle. However, it can give an indicator of preference and desirability, and combined with other methods it can provide some good pointers, especially at the beginning of the design process.
Preely preference test
You can use Preely for preference testing, hence you have the opportunity of conducting the preference test remotely and unmoderated. The only things you need to prepare are a question, the design options or the copies, and the follow-up questions. Read more here about how to set up a preference test.
Don’t have a Preely account yet? Sign up right here.
