Webtrends Optimization

Use analytics? Update your website?  Then you have everything you need.

While optimization is a distinct process, it shares the same skill set as these common online marketing practices.

Similar to analytics, optimization requires implementation, data analysis and measurable marketing goals.  And as with updating a website, you need creative and design expertise, web development and copywriting.

The example optimization workflow below illustrates what and when the above skills and resources are needed in the process:

1. Planning: At the beginning, all you need are basic marketing skills: select a page, spell out the questions you have about the page and determine the KPI’s for success.
2. Design: Use copywriting and creative/design skills to create test ideas to answer your questions and drive performance based on the selected KPI’s.
3. Build out: Use web development resources to translate your ideas into code.  You may be able to do this step yourself, depending on the content of the test and your own technical abilities.
4. Reporting: After the test is live, you need to analyze the data.  You’re looking for answers to your questions  (Do testimonials  increase sales?) and new insights (We don’t have to use flash to grab attention.)

While some education is necessary, optimization utilizes skills familiar to online marketers.  Optimization isn’t more difficult than other online marketing, it’s just different.

I recommend starting testing ASAP, even if it is with a small portion of a web page and/or will have a small impact.  Going through the process will help make it a natural part of your marketing cycles.   After adjusting by doing a test or two, running a large scale optimization campaign quickly becomes not only feasible, but some of your most important work.

Photo Credit: http://www.flickr.com/photos/confused_andy/ / CC BY-SA 2.0

If you missed it, see Part 1 (A/B Split Testing) and Part 2 (Multivariate Test Basics).

With the basics of part 2 down, it’s time to start designing a multivariate test.  Every optimization project has different challenges and goals, luckily though, there are a few rules that apply to every multivariate test design.  These rules fit into two categories: technical rules and content rules.

Technical rules:

1. Choose the appropriate multivariate test type (full or fractional factorial)
2. Determine the number of factors and levels that can be tested based on estimated conversion traffic (choose a test array)
3. Stop the test when it has stabilized, not based on your earlier estimations

These rules ensure statistical significance by constraining the test to the appropriate size at the beginning and then letting the test gather the proper amount of data at the end.

Running a test full factorial, if your traffic supports it, may be a good choice if you’re testing content that you believe to have many interactions or if you only want to test 2 factors with 2 levels each.  (Note: the smallest fractional factorial test size is 3 factors with 2 levels each.)  Typically though, you’ll want to run a fractional factorial test to save time and expand the number of factors and levels you can test.

In order to find out how many factors and levels you can test, you need to have some idea of your predicted page views, conversions, as well as an estimate of lift.  The reason that lift matters, is that a large lift will get you more conversions and so your test will stabilize quicker.  Because of this, I would be conservative with lift estimates to ensure that the test is not designed too large.  At Widemile, we have a large list of arrays available to our tool and have calculated the approximate conversions needed to stabilize, allowing me to look at the three criteria I listed and find the arrays that are statistically viable for testing.  You should look for something similar with your tool of choice.

To figure out when a test is stabilized, I prefer to primarily look at level influence stabilization with experiment conversion rate stabilization for support.  Widemile Optimize shows this using graphs, so I simply look for horizontal trending of lines, meaning winning levels and experiments stay winners and their level of influence or conversion rates stay fairly constant (look horizontal) over 3-5 days.  If you don’t have graphs available,  the historical cumulative conversion rate for your experiments and see if there is a lot of variance between the latest few days of your test.

Content rules:

1. Every item you test should answer an important question
2. Test variety not quantity
3. Test opposites first then refine
4. Remember you can run more than one test

The content rules are closely tied together.  In effect, they ensure that the items selected for testing have purpose and that they don’t needlessly expand the size of your test, reducing its efficiency.  I begin designing tests by creating hypothesis regarding issues with the page and then choose factors and design levels to address those issues.

An example hypothesis is “Having a hero shot on the right side of the page causes users to ignore the important value proposition on the left side.”  To test this, I would choose hero shot position as a factor and then have “left side hero shot” as the baseline level and “right side hero shot” as the second level.  This example also illustrates that, other than headlines and images, testing layout is possible with creative use of CSS and sometimes JavaScript.  As long as you can revert from one to another and it matches the other factors and levels, you are at liberty to test anything.

Coming back to the rules, make sure that you are testing as few items as possible to find out what you need.  Before testing a collection of lifestyle hero shots, choose one and test it against an iconic hero shot.  This will save you the time of going down a path of testing something that may not work.

Lastly, you aren’t going to be able to get the best page on the first run or even second, third, etc.  If you knew what your audience liked 100% of the time then you wouldn’t need testing.  Remember to think of your overall test plan beyond just the first run, so that you can answer all the questions you need without having to force everything into one test.

In summary, determine what you’re trying to achieve, select the proper testing method to meet those goals and then make sure to be purposeful and efficient with the content you end up testing in front of your visitors.  Testing and optimization is not difficult, although it can be tough to start.  Follow these rules and you’ll be on your way to conquering conversion rates, bounce rates, funnel drop-offs and many other metrics.

Photo credit: Aranda\Lasch (CC)

If you missed it, see Part 1 (A/B Split Testing).  Update: Part 3 on Rules for a Successful Multivariate Test is here.

The technical and statistical aspects of multivariate testing can be complicated but in order to design successful tests you don’t need to know everything, just the basics of how it works and some guidelines.  I’m assuming you already have some understanding of multivariate testing, however I want to cover the basics and make sure we’re on the same level before going into how to design good multivariate tests.

Check out the wireframe below.  Pretty standard for a landing page, right?  To properly design a multivariate test, we have to look at the page in a certain way.  Using three key terms, factors, levels and experiments, we can break down a test and describe its framework.

Factor: An element of the Web page (headline, image, text) being tested.  The element can also be groups of content, e.g. left column, button and hero shot together, or all banner ads on the page.

Level: Content that is assigned to a specific factor to be tested.  For example, one variation of a hero shot.

Below are 4 factors from our example page (headline, hero shot, offer and button) and then each of those factors with 4 levels represented by the different colors.  Note that the levels of one factor do not have to relate in anyway to the levels of other factors.

The last term, experiments, makes use of both factors and levels.

Experiment: A unique combination of levels used during a test.

Here you can see 4 different experiments.  Each experiment is different and holds different combinations of levels.  Note that there actually are many more variations (4×4x4×4=256 combinations).

Essentially a multivariate test involves showing these experiments randomly to live traffic, while tracking how each experiment performs.  The one that performs the best wins.  Each experiment is shown to many people, but each person only sees one experiment.  (There is some complexity in this, if you are still confused or want to know more, go to my primer on full and fractional factorial testing.)

In my next post, I will use these terms to outline the rules to creating a great multivariate test.

A beginner’s mistake is to test every idea with every test. This is the most obvious way of being efficient. If I can test 50 things in a week, why not?

In my experience, efficiency has more to do with careful test design and doing things right the first time, than trying to test everything and rushing the process. By testing a few big ideas quickly and then designing the next test based on those results, you can do a set of small tests and get answers fast without having to risk your page to many bad ideas.

Every test should have specific questions its trying to answer. Not just “What’s the best performing page?” but questions that lead to that. A car salesman doesn’t blindly try every tactic in the book get you to buy a car, a real salesman probes you with a few questions and changes their technique accordingly.

That’s how you should design your tests.

Here’s an example test plan that works for most clients:

• Step 1 (Split Test) – Find an optimal template/design: What template and/or design effectively gets visitors to stick, click and convert? At this stage, you aren’t testing messaging yet, you’re merely re-skinning and moving elements around to find a good design. Some techniques to use are simplifying the page by de-emphasizing unimportant content (shrink company logo, move ads to the bottom of the page) and emphasizing core content (moving 3rd party validation near the call to action) and adding more whitespace to the page to enhance readability. These are in addition to a well done creative design. This test usually has the greatest impact, however it all depends on your current page and the audience. (Read more on template testing)
• Step 2 (Multivariate Test) – Find the biggest converting segment: This test focuses on finding the correct messaging by appealing to different segments that you know and hypothesize visit your page. If your product was Google Apps, you might test appealing to business users and freelancers. Or if you are selling a cell phone, you might test features against benefits.
• Step 3 (Multivariate Test) – Find the perfect way to communicate to the segment: Step 2 points you in the right direction, but this step helps you find the exact place you should be with your page. Use what you learned (freelance messaging won) and try variations on that winning theme to really grab your audience and give them what they want. Also, step 2 may have revealed 2 or more segments that are worth targeting. If you can segment them out, run multiple tests that are customized for each segment, and you’ll raise conversions even higher.

The alternative is to test 50 ideas of which many of the ideas overlap. Why test any ideas that are remotely similar until you know that they work in general? If I go to a dealership wanting a sports car and the dealer offers me 5 colors of minivans, I’m still not going to buy a minivan. Show me 4 types of cars, let me pick the one I like and then we might talk about color.

Let your visitors lead you!

This really is a simple process, but it drives results. Be methodical to be efficient. By course correcting in each test, you get closer and closer to what you need and don’t spend a lot of time testing losing elements. Follow a test plan like this and you’ll get results and learn a lot about your core converting visitors.

I just got news that Google Web Optimizer is out of beta. In addition, it doesn’t even require an AdWords account to use it anymore. This is great news for the testing industry and for all online marketers. Check it out here. In addition, there now is a dedicated Official Google Web Optimizer blog.

I’ll see if I can get some tests running just to see what the isolated tool looks like versus the integrated one. They also upgraded the setup of multivariate tests for all versions.

On another note, it’s good to see that Google saying things like “it’s hard to find a serious advertiser who doesn’t at least plan to do content testing this year.” They even mention some best practices that I’ve talked about at this blog:

• “don’t be shy: big changes generally yield big differences in performance”
• “We recommend letting your experiments run for at least two weeks, no matter how much traffic you get and how strong the results initially appear, just so the data has enough time to normalize.” – I recommended the same things in my Multivariate Testing Primer.

Also there’s a forum for Google Web Optimizer users, which isn’t new, but expect it to grow quickly with this latest announcement.

If you’re waiting for the last post in my 3 part series about difficult test results, I apologize. I’ve been sick all week and wanted to go over my last post with Vladimir Brayman, Widemile’s chief scientist, before I posted it for the world to see. It’s a very important topic and a challenging one too. I’ll try to get it out next week for sure.

I was going over my testing plans with my boss, Frans Keylard, today and he reminded me of a very powerful rule.

Test if something works before you try variations of it.

In this case, I was testing out two testimonials. They were quite different in the messaging, however, do I even know if testimonials are read or impact visitors at all? If I test a testimonial and no testimonial, I will immediately know if I should continue trying testimonials. If testimonials win or compare favorably against having no testimonial, then I know to test additional testimonials.

Not that I have never tested factors on/off or tried totally different factors, e.g. a testimonial against a product shot. I had a strong feeling testimonials were going to work, so I assumed they would, although I know I shouldn’t assume anything. An honest mistake, but a good reminder.

Ideally I would be testing variations, along with showing nothing, or “off”, as a variation, however in this case the page didn’t get much traffic so I was limiting my testing to the most important variations and factors.

There might be some fringe cases where this isn’t necessarily true, but in most cases you should just save extra variations for future runs and first find out if your factor has any impact on the page. Maybe I need to read some of my old posts more often.

AdAge asked everyone on their Power 150 blog list to contribute a short snippet about “what technology marketers should be paying most attention to in 2008.” I got included in the list and found it a good way to introduce someone to testing. If you have any marketing colleagues that still aren’t sure why testing is valuable, here is my full, unedited submission:

Site optimization is already becoming a focus of every marketer in 2008. Why? Because testing and tuning websites is a natural extension to the proliferation of web analytics. Marketers know how important analytics are to their campaigns, but even with all the valuable data gathering tools out there, there is not a straightforward next step to improve websites. Site optimization through multivariate and split testing helps turn analytics into action, allowing websites to improve and grow with their audience.

In addition, the rising costs of PPC makes each click even more important. Using site optimization, the large gains that come from optimizing SEM are now arriving at the post-click stage. These gains are not isolated to SEM either; site optimization improves ROI across the board. As the market matures, the need for site optimization will continue to grow, making it essential that every marketer considers it in 2008.

Having great testing technology puts a lot of power in your hands. You can test anything and everything you want. However, like any other tool, to use it effectively you have to use it right. There’s a lot of best practices and thought that goes into test design, but following these three rules can get you a good test in most situations.

1. Maximize your traffic: Pack as much as you can into a test for the amount of traffic you have to keep it a short test. Using Widemile’s platform that’s 2 weeks to be safe, with Google Optimizer you should do at least a month (explanation).
2. Test opposites: If you test stuff that’s similar, they’ll perform about the same. So find out the general theme you should be following first by testing opposites (B2B vs B2C, podcast vs ebook, descriptive vs benefits).
3. Learn from the previous test: Always make sure you line up your tests so that you learn something that can be used in the next one to either refine or to learn something new.

The goal of these three things are to maximize your time spent testing by testing as much as possible while also minimizing testing suboptimal content. For example, if I was selling iPods and I tested 2 images of people running with the iPod, one with a man and the other a woman, I might think that was a good test. However I could have totally missed out on an image that worked better, such as an iPod next to a PC. I could test that out after the initial test, but then I just wasted one test run. The right way would be to test one sport image versus one PC image and find out which direction to go. From there I could test against other opposing images or refine the PC image.

The only warning I’d throw in is that if you’re trying to test a lot of things at once, you might want to scale back. Pick a 2-4 themes depending on your test size and stick to testing them out. Don’t mix and match.

Follow these steps and you’re on your way to getting not quick tests, but efficient ones.

Multivariate testing is a buzz word these days, but the buzzword of multivariate testing seems to be Taguchi. However, that term is being abused. Do you know what Taguchi really means? I wasn’t even positive, so to get some background, I did some research and talked with Vladimir (Widemile’s Chief Scientist).

The name and method comes from Genichi Taguchi. His method, also known as Robust Design, attempted to improve product manufacturing quality. Therefore it falls into an area of engineering called Quality Engineering.

Does this sound aligned with website testing? Not really, and this is the problem of using the term Taguchi with web site testing. The goals of manufacturing and the goals of a website are not the same.

What most people are attempting to grasp when using the term Taguchi is fractional factorial test design. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of fractional factorial testing but is not the only or best fractional factorial method. In fact, even within manufacturing, the Taguchi method was the inspiration for many new techniques but many statisticians find it flawed.*

It is important to differentiate the Taguchi method from fractional factorial test design since one is a basis for manufacturing while the other is purely related to design of experiments. You need to ensure that the math and science behind your testing is based on methods that have the end goal of optimizing your website only. So if your testing tool uses the Taguchi method for testing, you better ask what that really means.

So does Widemile use Taguchi? We don’t use the Taguchi method, however do use fractional factorial test design. I like to say that our platform goes beyond Taguchi because it was specifically made for optimizing web content.

Don’t get sucked into the Taguchi method, it is just a buzzword used by your fellow marketers. Just because the technology doesn’t use Taguchi, doesn’t mean you should count it out.

*Read more after the jump for Vladimir’s explanation of the Taguchi method and its criticisms
Continue reading ‘What is Taguchi? How does it relate to testing?’

Quality PPC and LPO campaigns are key to great conversion rates. If either of them are optimized, you might get good results, but with both of them optimized, your gains are exponential. There are a few pitfalls in optimizing them both though, even with good intentions you may end up confusing your results rather than getting results.

Here are 3 methods to effectively optimize your PPC and landing pages:

1. Do one at a time: Test out your new PPC strategy, but wait until your landing page testing is done. Changing your PPC means you’re changing the audience, both in demographics and expectations. This will impact your landing page testing. Once you find a winning PPC campaign, test the same messaging on your landing page. This is the easiest way to optimize both, but the next two are better ways to go.
2. Do them simultaneously: If you are testing 2 PPC strategies, create 2 separate landing page tests to match the respective campaigns and drive traffic solely to the matching test. This avoids biasing the PPC that better matches your landing page.
3. Segment all the way through: For segments you know you’re going to have, make them go to different landing pages. Test your pages and separately track how each segment performs. Sometimes all your segments respond best to the same landing page, but often times your segments want something different and it’ll show in your testing results. Also, if you’re doing #2 and realize that the ROI is good enough for both campaigns, break it out and optimize them separately.

These are basic, but very effective methods to maximize testing both your PPC and landing pages. If you want to get actual and sustainable results, you have to control as many variables as possible. Only when you can trust your data, will it perform how you expect. Follow any of these methods and you’ll be on your way to higher conversions.