Webtrends Optimization

If you missed it, see Part 1 (A/B Split Testing).  Update: Part 3 on Rules for a Successful Multivariate Test is here.

The technical and statistical aspects of multivariate testing can be complicated but in order to design successful tests you don’t need to know everything, just the basics of how it works and some guidelines.  I’m assuming you already have some understanding of multivariate testing, however I want to cover the basics and make sure we’re on the same level before going into how to design good multivariate tests.

Check out the wireframe below.  Pretty standard for a landing page, right?  To properly design a multivariate test, we have to look at the page in a certain way.  Using three key terms, factors, levels and experiments, we can break down a test and describe its framework.

Factor: An element of the Web page (headline, image, text) being tested.  The element can also be groups of content, e.g. left column, button and hero shot together, or all banner ads on the page.

Level: Content that is assigned to a specific factor to be tested.  For example, one variation of a hero shot.

Below are 4 factors from our example page (headline, hero shot, offer and button) and then each of those factors with 4 levels represented by the different colors.  Note that the levels of one factor do not have to relate in anyway to the levels of other factors.

The last term, experiments, makes use of both factors and levels.

Experiment: A unique combination of levels used during a test.

Here you can see 4 different experiments.  Each experiment is different and holds different combinations of levels.  Note that there actually are many more variations (4×4x4×4=256 combinations).

Essentially a multivariate test involves showing these experiments randomly to live traffic, while tracking how each experiment performs.  The one that performs the best wins.  Each experiment is shown to many people, but each person only sees one experiment.  (There is some complexity in this, if you are still confused or want to know more, go to my primer on full and fractional factorial testing.)

In my next post, I will use these terms to outline the rules to creating a great multivariate test.

Update: Check out Part 2 on Breaking Down Multivariate Testing and Part 3 on Rules for a Successful Multivariate Test.

Testing is not hard, but there are fundamentals that guarantee a successful optimization campaign.  To help get marketers up to speed with the basics, starting today, I will be writing about one topic per post and put together what I call Billy’s Optimization Guide.

The natural place to start is with a/b split tests, so let’s begin there.

First, a quick useful definition of an a/b split test: the competition of two distinct pages, where a portion of live traffic, usually 50%, is sent to one page and the rest to the other.  The winner is the page that provides the highest conversion rate, or whatever KPI is appropriate.

I want to emphasize that a good a/b split test requires distinct pages.  If that’s too vague, a simple rule that we follow at Widemile is:

You should be able to tell the difference between the 2 pages from 15 feet away.

Anything else isn’t a big enough change to be efficient in a split test and likely should go into a multivariate test.

With that definition in mind, here are three essential types of a/b split tests.  These are three of the tools in the testing toolbox that you should consider when putting together your optimization campaign.

1. Template test: Test the same general content (hero shot, copy, and button color) with a different layout and/or creative treatment.  The goal is to have a new template that better emphasizes the value proposition, improves readability and sets up well for a multivariate test.

   

   Use this when… you want to make sure you have a solid design, before or after testing messaging.  The majority of the time this should be your first test.

2. New concept test: Test a totally new approach.  Don’t let anything hold you back, test what you think will work best and see if it beats the original.  The approach here is to break out of the box and create a page that’s holistically designed around a new marketing concept.  Sometimes this involves introducing new functionality, animation, interactivity and other dramatic steps.  However it can also be on the smaller scale, such as introducing new messaging that requires a complete redesign.

   

   Use this when… your current page has already been tested many times and beating it has become difficult or you believe the way to really grab your visitors is through a big change.  This should only be done when the benefits of multivariate testing (knowing individual factor influences) are outweighed by the possible gains.

3. Funnel test: Send users to different multi-page experiences, e.g. no registration vs. requiring registration (below) and a one page form vs. a 3 page form. A funnel test can also be done with a multivariate but is simpler as an a/b split test.

   

   Use this when… you want to test content that extends past one page.  This should be done earlier in the testing process so that you don’t end up optimizing a page and then find out it’s a suboptimal experience.  It can be more technically demanding to do this sort of test though.

Every optimization campaign is different and so knowing what kinds of tests are available is one of the most important places to start.  For my next post, I will talk about the different ways to use a multivariate test.  Please post in the comments if you have any questions or contact me via Twitter @billysblog.