Multivariate testing: a quick primer

December 26th, 2007

Topics: Digital Marketing

Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!

Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.

Billy’s Multivariate Testing Primer

What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.

*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.

What happens:

Multivariate testing diagram

  • Visitors come to a page and are shown a random version of the page
  • Conversions are tracked based on which page they saw
  • Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate

Strengths:

  • If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
  • Allows for quicker testing of multiple elements than split or A/B testing
  • Analysis derived from live visitor data
    • Proves winning page elements to be better than others

Weaknesses:

  • Medium learning curve
    • Test results easily ruined by mistakes or poor methodology
  • Requires a minimum number of conversions
  • Code must be added to web page
  • Cookies and JavaScript must be enabled by visitors

Keep in mind:

  • Increasing the number of elements being tested increases the time needed for the test.
  • Number of conversions over time determines test size and length
  • The faster a page gets conversions, the shorter the test will be and/or the more things you can test
  • More conversions is better
  • A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
  • Don’t test things that are too similar, look for different segments or messaging to pursue

Other concepts:

  • Using it with split testing:
    • Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
    • Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
    • If you want to try new layouts later on, go back to split testing.
  • Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
    • Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
    • Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
    • Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.

There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.

  • http://www.matizmo.co.uk Jake Coventry

    Hi

    I wrote an article on this a while back: http://www.matizmo.co.uk/landing-pages-using-multivariate-testing-to-improve-your-conversion-rate – it is amazing how many A/B testing companies are springing up offering this kind of service. I particularly like the idea of tailoring content for the visitor – whether they have returned or first time, and using multivariate testing to maybe tell them about info they had not seen the last time they came. Systems are getting easier to use and more sophisticated at the same time.. it is an interesting space to watch.

    Cheers
    Jake

  • http://www.widemile.com Billy

    Hi Connie,

    I talk about template testing here:
    http://testingblog.widemile.com/2007/12/05/5-quick-tips-to-effective-ab-and-split-testing/

    If you’re looking for formulas and algorithms that’s much more complicated and falls within statistics. Unless you’re looking to build a tool from scratch, you should look towards existing testing tools such as Google Web Optimizer or what Widemile offers.

    I discuss fractional factorial test design here:
    http://testingblog.widemile.com/2008/01/28/why-google-optimizer-is-free-its-old-and-slow/

    The LPO Certification does not cover template testing or multivariate testing, it just covers LPO best practices and gives you solid ideas to test. I believe they also have started a class specifically about testing, but have not taken it myself.

    I hope this helps,
    Billy

  • http://consortemedia.com Connie

    Hi Billy,

    Thank you for providing great information on multivariate testing. I am searching for information on template tests as well as on fractional factorial multivariate tests. Can you reference and resource that provide extend info on these two topics, in terms of formulas, algorithms and etc. Also, you mention about the Landing Page Optimization Certification program with MarketingExperience, does this program cover multivariate testing (including template testing)?
    Thanks in advance for your response.
    Connie