The One About Multivariate Testing

First of all, thank you to everyone who read my previous post about Persona Creation Tools. That post has been read by 1,000 of you and that is pretty exciting for a new blog such as this. That post started some great conversations which I am hoping to do here again. I want to talk about Multivariate Testing simply because I want to know more about it. Please feel free to comment here or wherever you see this posted so we can keep the conversation going. Now, let’s get to it.


Certain parts of the User Experience Design process are more well known than others. You’ve got your heavy hitters like Wireframing, Prototyping, etc… but what about Multivariate Testing?  What is Multivariate Testing or MVT?  Why and when would you need to use it?

Let’s start by attempting to define this form of testing . The fine people at Optimizely have defined it here as:

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

and over at SiteSpect they have an article here that similarly says:

…there are multiple elements being tested at the same time. For example, two alternate product images, plus two alternate headlines, plus two alternate product copy text, for a total of 27 possible combinations.

Multivariate Testing is simply the act of testing multiple design variables.  It’s A/B testing but with as many other letters of the alphabet as you’re willing to include!



Jakob Nielsen of the Nielsen Norman Group states that A/B Testing (including MVT) has 4 huge benefits including:

  1. It measures actual behavior.
  2. It can measure very small performances differences.
  3. It can resolve trade-offs between conflicting guidelines or qualitative usability findings.
  4. It’s cheap (although this becomes less true with every additional variant that you test if for no other reason than accounting for the time it takes to create each one.)

Keeping with the natural ebb and flow of life means that where there are positives there are negatives.  The two main limitations, according to Nielsen, are:

  1. A/B testing can only be used for projects that have one clear KPI (key performance indicator).
  2. This form of testing only works for fully implemented designs.

We’ve determined what MVT is, we’ve looked at the benefits and limitations but how do you perform a Multivariate test?  In this 2011 Smashing Magazine article called Multivariate Testing 101, author Paras Chopra informs us that there are actually multiple different types of MVTs.

  • Full Factorial Testing (the most common)
  • Partial or Fractional Factorial testing
  • Taguchi Testing

Conversion Rate Experts has a great list of all the different types of A/B & MV testing tools that are on the market here, some of which have been mentioned above.  In the list it tells you which type of testing each tool can accommodate.  It figures that there are multiple options to test multiple variables in multiple different ways.

This is where we get to the the ‘let’s make this a conversation’ part of the post.  I have a couple of questions about how and when to conduct Multivariate Testing for the greater UX community and I would love to hear from you.  Please feel free to comment here or wherever you see this posted.


  1. Do you or your organization actually perform Multivariate Testing?  If so, how often?
  2. What are the biggest factors that come in to play in determining if MVT is needed beyond A/B testing?
  3. What are you favorite MVT tools?


Please be sure to click on all links and read the articles mentioned above for many more details about Multivariate Testing.

Thanks for reading and keep checking back because there is more on the way!

Dan Richardson – – @uxorbust


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s