Webtrends Optimization

Update: Check out Part 2 on Breaking Down Multivariate Testing and Part 3 on Rules for a Successful Multivariate Test.

Testing is not hard, but there are fundamentals that guarantee a successful optimization campaign.  To help get marketers up to speed with the basics, starting today, I will be writing about one topic per post and put together what I call Billy’s Optimization Guide.

The natural place to start is with a/b split tests, so let’s begin there.

First, a quick useful definition of an a/b split test: the competition of two distinct pages, where a portion of live traffic, usually 50%, is sent to one page and the rest to the other.  The winner is the page that provides the highest conversion rate, or whatever KPI is appropriate.

I want to emphasize that a good a/b split test requires distinct pages.  If that’s too vague, a simple rule that we follow at Widemile is:

You should be able to tell the difference between the 2 pages from 15 feet away.

Anything else isn’t a big enough change to be efficient in a split test and likely should go into a multivariate test.

With that definition in mind, here are three essential types of a/b split tests.  These are three of the tools in the testing toolbox that you should consider when putting together your optimization campaign.

1. Template test: Test the same general content (hero shot, copy, and button color) with a different layout and/or creative treatment.  The goal is to have a new template that better emphasizes the value proposition, improves readability and sets up well for a multivariate test.

   

   Use this when… you want to make sure you have a solid design, before or after testing messaging.  The majority of the time this should be your first test.

2. New concept test: Test a totally new approach.  Don’t let anything hold you back, test what you think will work best and see if it beats the original.  The approach here is to break out of the box and create a page that’s holistically designed around a new marketing concept.  Sometimes this involves introducing new functionality, animation, interactivity and other dramatic steps.  However it can also be on the smaller scale, such as introducing new messaging that requires a complete redesign.

   

   Use this when… your current page has already been tested many times and beating it has become difficult or you believe the way to really grab your visitors is through a big change.  This should only be done when the benefits of multivariate testing (knowing individual factor influences) are outweighed by the possible gains.

3. Funnel test: Send users to different multi-page experiences, e.g. no registration vs. requiring registration (below) and a one page form vs. a 3 page form. A funnel test can also be done with a multivariate but is simpler as an a/b split test.

   

   Use this when… you want to test content that extends past one page.  This should be done earlier in the testing process so that you don’t end up optimizing a page and then find out it’s a suboptimal experience.  It can be more technically demanding to do this sort of test though.

Every optimization campaign is different and so knowing what kinds of tests are available is one of the most important places to start.  For my next post, I will talk about the different ways to use a multivariate test.  Please post in the comments if you have any questions or contact me via Twitter @billysblog.

You and your competitor’s all have the same landing pages.  You have a hero shot of the product, a big call to action button and short, punchy copy.  Or maybe you’re already ahead of your competitors and have run a few tests on your page, picking up more conversions on the way.  In either situation, you’ll eventually hit a wall and struggle to get additional lift.  So how do you continue to improve?

Go for broke.  Try something you’ve never tried before.  It might end up being a total failure, but it also might give you the lift you want.

The gamble you make with optimization can end in 2 ways:

• You lose X amount of conversions over the week or two that the test is running
• You gain X amount of conversions for the effective lifetime of the page

The possible upside dwarfs the downside by a large margin and, either way, you learn something new and can optimize the next test more successfully based on what you learned.

Luckily, with skill and experience, the risks of testing are minimized, however beating a strong page is never easy or guaranteed.  But when you do find something new that works or see that your current page still is a champ, you can rest assured that you’re doing all you can to drive conversions.

AdAge asked everyone on their Power 150 blog list to contribute a short snippet about “what technology marketers should be paying most attention to in 2008.” I got included in the list and found it a good way to introduce someone to testing. If you have any marketing colleagues that still aren’t sure why testing is valuable, here is my full, unedited submission:

Site optimization is already becoming a focus of every marketer in 2008. Why? Because testing and tuning websites is a natural extension to the proliferation of web analytics. Marketers know how important analytics are to their campaigns, but even with all the valuable data gathering tools out there, there is not a straightforward next step to improve websites. Site optimization through multivariate and split testing helps turn analytics into action, allowing websites to improve and grow with their audience.

In addition, the rising costs of PPC makes each click even more important. Using site optimization, the large gains that come from optimizing SEM are now arriving at the post-click stage. These gains are not isolated to SEM either; site optimization improves ROI across the board. As the market matures, the need for site optimization will continue to grow, making it essential that every marketer considers it in 2008.

This comic is both hilarious and insightful. Take a second and see if you’ve experienced something similar to this situation:

I came across this before starting my work at Widemile, but stumbled upon it again while browsing Marianina’s Web Analytics Princess blog. Surprisingly, I found it extremely relevant to the multivariate testing I do at Widemile now.

The comic is a joke about how hard it is to do what customers want. Even when you have the luxury of a customer telling you everything they want, they don’t always do it correctly and no two people interpret their instructions the same. I like this comic because it shows how difficult it is to create good experiences and products for customers, no matter the amount of help you have.

Now, imagine how this applies to your website. Visitors want your product presented to them in a certain way with certain offers. Before testing, you’d listen to visitors with analytics, surveys and usability studies even. From here everyone would make their own “panel” of the comic; the creative team would have a go, the marketing team would revise it and the CEO would change a few words around. Eventually a page is made and your business puts it online, not knowing whether the page is what your visitors truly want.

Take that same situation and add multivariate testing. Now we get the same perspective that we have when looking at the full comic strip. In the previous situation, all you could see was your individual panel and your customers could only choose the one you gave them. With a multivariate test, different customers see different versions and the trends that show up in your data, reveal the “whole comic” and point you to which one should be the last panel… what the customer really needs.

If you look at the first panel and the last panel, it’s immediately obvious that they are different. Your customers can tell what they like, even if they can’t articulate it. Start listening to their true feelings by testing your pages.

For more of these comics, check out the official Project Cartoon site.

MarketingSherpa is giving us a lot of love recently with The Weather Channel case study and now one on Smartsheet. If you were too shy to sign up for our case study on the Widemile website, then please check out the one at MarketingSherpa.

I especially like the last 4 points in the story:

#1. Don’t sit pat on your conversion rates. Colacurcio didn’t know that her original 5%-7% was above the industry average going in and is glad she didn’t. “If I would have said, ‘My conversion rate is pretty good’ and done nothing, I would have totally missed the opportunity to double it.”

#2. Conducting just a few multivariate test and applying them to a greater number of landing pages works. Simply put, you don’t have to test each landing page individually. “Sometimes it seems overwhelming when you think of multivariate testing, but you can cut corners. There are a lot of things that are low-hanging fruit — things that can be applied across landing pages.”

#3. Redesigning shouldn’t stop with your team’s new ideas. “You really have to get your organization into the mindset: ‘We are testing. We are not just going to spit out the next five Web pages.’ ”

#4. Even if your higher-ups are impressed with the initial results, Colacurcio says, marketers should expect to face organizational barriers when they start their second round of testing.

Don’t forget to check out the before and after in the creative samples.

If none of that catches your eye, Janet Meiners at MarketingPilgrim wrote a great summary on the case study.  She also makes a great point that, “I’ve been in heated debates about the best course of action but testing works best – assuming you’re humble enough to go with the data over your ego.”

Trust your data and watch your numbers build your ego.

Most decision makers start multivariate testing because it makes sense in terms of ROI. While I may talk about helping marketers make good decisions about their pages, my real job is to make my clients look good by getting them a result that looks good online (a quality web page) and on paper (quality returns.)

The value of multivariate testing and landing page optimization lies in the fact that SEM is rapidly becoming more expensive as competition rises.

How much would you have to spend in PPC to get a sustainable 30% raise in conversions? To be sustainable, you would have to spend that much money month after month.

• Sustainable lifts even after optimization ends

With testing, the increased conversion rate will typically be sustainable over a significant period of time. Pay to optimize once and you’ll receive that raise in conversions for free for months and even possibly years after you finish your first test. Of course the best way to go is continual testing to squeeze even more out of your pages, but the results from any testing come pretty quickly and make good business sense.

• More efficient SEM spend

A bonus is the natural cycle that occurs from optimizing. Once your page is optimized, your advertising spend will be more efficient. This allows you to either spend the excess money towards other campaigns or feed it back into the optimized campaign to further drive conversions.

• Large increases in conversion rate

Lastly, while this only happens occasionally, conversion rates often shoot through the roof after testing. We have more than doubled conversion rates for some pages, which typically is not possible even with dumping money into advertising.

If you have a solid SEM campaign but aren’t seeing results you want or once had, or if you just want to stay ahead of the competition, multivariate and split testing are the necessary next steps.

I work in this field for two reasons: 1. I love marketing and testing, 2. I know that in a few years, every business will be testing.

SEM will be around for a long time and must be done right, but it is not getting cheaper anytime soon and most businesses already participate in it. Testing is still nascent and probably where SEM was 3-4 years ago. Don’t get left behind, the future is in testing.

I stumbled upon a great post from Andy Edmonds at Always Be Testing about ways testing can be used. Here’s the list he came up with:

Business Acceptance Testing

Example: You want to add yet another shortcut on the homepage for a new sub-audience. Use testing to validate you didn’t mess up the other functions of the homepage.

Value Estimation

That new search function is going to cost you X thousands of dollars. How long will it take to provide a positive return on investment.

Design Choice Determination

The boss thinks the logo should be purple. You don’t.

Customer Understanding

Multivariate methods are really valuable for this use case. Say you wanted to ask the question, “Is copy at the top of our product pages worthwhile? Or should we just drop it and get more products above the fold?” A multivariate test can let you vary the size & quality of the copy, along with other elements that push the product down the page, and assess the general impact of products higher on the page as well as the general impact of good copy.

Off the top of my head, I would change the definition for Customer Understanding though. You can learn more than something as specific as, “Should I put copy here or there or remove it?” Test different messaging by testing fear based versus lifestyle based content. It allows you to find out which messages really hit at home with your readers. Not only will that help you improve that page, but it will teach you why your users want your product or service in general.

In short, testing answers “Does this work or not?” Give the customer choices and let them do the hard work deciding what works best. Makes being a marketer easier doesn’t it?

Photo credit: e-magic via Flickr

We marketers get to have a lot of fun. We have a variety of projects that we get to plan, develop and execute. Wrapping up every project is an accomplishment for us. Or is it?

Unfortunately, many projects are started without tangible metrics to strive for other than finishing it on time and within the budget. One part of being a marketer, or any businessman or woman, is showing that what you do is valuable and has a solid number attached to it.

Because of that, every marketer should make a resolution to measure and test their campaigns rather than just finishing them.

Achievements like site redesigns and campaign launches should be celebrated, but they also should be effectively tracked and analyzed from beginning to end.

Jason Burby wrote a fantastic article at ClickZ about ringing in the new year by defining and executing based on goals for 2008. In it he describes how sometimes projects succeed in some measures but fail at others, so overall is it a success?

He even mentions multivariate testing as part of the solution: “One part of this new way of thinking is to ensure you have a simple, easy to use testing platform to try different things and to measure the impact of tests based on goals. If you haven’t invested in an A/B or multivariate tool, 2008 is the time.”

What I’ve found in doing optimization and working with marketers in web analytics is that defining success in every campaign is key to being a quality marketer. And the only way to know if you met those goals is to measure what you do and how much it drives those goals.

For web page testing, the goal typically is to raise conversion rates and so its easy to keep that in mind. However if I never changed conversion rate as my ultimate goal, I might miss out on other valuable goals. A great example is if there are two pages with different subscription levels for a product, then possibly the lower converting page has a higher lifetime ROI.

The great side benefit of having a goal and measuring what gets you there, is that you can learn from success and failure. Even when I test a page and get an improvement, I exam what factors were the worst, along with what did best. It tells me what scares away and attracts customers so that I can look out for those things in other parts of the page and overall funnel.

Never assume that changing or making a new version of anything is always better. Figure out what you want to improve, make the changes and see if you find that improvement. As long as you have a measurement for success and take the time to track your progress, you can learn from your campaign and in the end, get closer to reaching and beating your goals.

Sometimes people ask me why landing pages are such popular targets for optimization and testing. Why not optimize a home page? A product page?

Actually, we can optimize those kinds of pages, but almost all businesses come to us with a landing page that needs help. Beyond the demand for landing page testing though, is the fact that landing pages inherently are fit for testing. Let me explain.

One of Widemile’s landing pages

The strength of multivariate and split testing is in pulling out the best page possible for what you want to do. With a homepage you have multiple things you want to do (e.g. show your products, company history, customer service, get people to spend X minutes browsing) and with a product page you are typically working with a CMS template for your whole site. Those factors complicate things a bit since you have to figure out what counts as improvement for the homepage and for the product page, you have to work with a CMS system and make changes that improve the majority of products pages using that template.

These things are not impossible to do or even difficult in many situations, but a landing page is usually totally independent of everything and has only one goal.

In a technical sense, a landing page is more simple to deal with. In a measurement sense you only need to improve one metric, the conversion. With a landing page, we don’t have to make copy and creative that works okay for all situations, we simply make copy and creative that is optimal for one situation.

This makes testing really fun and easy, since you can test and find out who your audience really is and what causes them to convert.

Now that I’ve answered this question, in the future I’ll move onto a more interesting post: Why you should optimize everything else too.

Online testing is a bit different from other marketing data. It uses live traffic to find out what works. Analytics is the same, measuring what’s occurring at the moment. So why is that important? Well you can infer all you want from surveys, usability studies and demographics, but in the end you can’t argue against what real users are doing.

Avinash Kaushik, a popular analytics blogger, summed up the juiciest bits of a presentation by Jim Novo at eMetrics. In it, Jim asked, “What data yields insights that can be actioned the most?” The answer:

“[A]ctionability, relevance of insights that can be actioned decreased as you go down the slide”

He makes the point that the farther away you get from the top of the pyramid, the harder it is to accurately predict your users actions. Yet often times too much value is put into the bottom levels of the pyramid. Even when marketers do test and measure actual behavior, they go about it the wrong way because they stick to all this other data too much and end up testing things that are all alike, defeating the purpose of testing.

Think about it, can you really tell if a red button will work better than a blue button if all you have are demographics? There are places for all of these types of data, but there should not be a fear of actual behavioral data. Yes we are using live traffic, yes the data is driven by technology (online visitors, javascript, cookies, rather than people filling out a survey), but those numbers tell a story unlike any other data.

Make the most out of all types of your data, but don’t die by one or the other. Use what’s best for every situation, but realize that you will never know you are right until you test it out.