Webtrends Optimization

It’s been a fun journey here at Widemile and although Widemile, the name, is disappearing, our mission to bring optimization to the forefront of online marketing has not ended.  I will be moving this blog over to the Webtrends domain very soon, for now, please see my post on the Webtrends blog for my thoughts on the acquisition.

Thanks for being a loyal reader of my blog, I will be posting a lot more once I switch over to Webtrends, so don’t forget to check it out.

-Billy

Webtrends is hosting a completely transparent marketing campaign, called The Open Campaign.  In the campaign, Webtrends and a few partners, including Widemile, are conducting their various specialties and sharing the results with the public.  The idea is to let you all learn about all the great marketing technologies out there and how it ties together with Webtrends, as well as seeing the performance of the campaign itself.

Check out the Site Optimization section where I will be discussing the test we designed for The Open Campaign and the results.  See for yourself exactly what its like to do optimization with Widemile Optimize and how my brain works throughout the process.

A huge part of doing optimization well is knowing what to test (put garbage in, get garbage out), so keeping up with good design philosophy is extremely valuable.  While brushing up on web design, I came across a Smashing Magazine article on UI design trends by Janko Jovanovic.  He uses a lot of great examples of good design, some of which are perfect for illustrating some optimization options.

With that in mind, I’m going to examine one of the sites mentioned and discuss the good, the bad and the testing opportunities I see.  The (lucky?) site I picked was Picnik, which has done a commendable job on their registration strategy.  (Also, like Widemile, they are a Seattle start-up.)  I only wish the site wasn’t flash based, which is more difficult to optimize.  Despite that, my thoughts on test variations and best practices are still applicable to it and any other registration campaigns.

Quick summary: Picnik is an online photo editing application.  You can upload photos and do easy photo editing all within the browser.  You can try out the app, even exporting and saving photos, without registration.

Let’s get started by checking out their form:

Although a bit busy, I like the way the form assists users.  It has a green highlight to for the selected field and dynamically pops up error messages (see the username alert below).  Additionally, it hides and locks the “again” fields until there is valid input in the corresponding field.

One highlight is how this is a good example of when a lightbox/page overlay type form might be appropriate (note that behind the form is the page I was working on, which has been darkened).  Why is it appropriate?  Because this is the form that pops up after the user clicks “Register.”  It makes sense to be direct and reduce additional marketing if the user indicates they want to sign-up by clicking directly on the register button.

Is this right for your site/landing page/microsite?  It’s hard to say, but I would recommend testing it.  This would fall into the category of a funnel test because it eliminates a page in the registration funnel.  As long as your full page and lightbox form don’t have any glaring issues, you should quickly see the influence of whether a small and direct lightbox form works, or if a whole page with additional information is necessary.

In terms of testing this overlay form, there are a few big opportunities for improvement.

• Testing title and intro copy. Use “free” in the headline and as the first word, e.g. “Free registration”, then list a few benefits rather than saying “All we need is a username, password, and email address.”
• Eliminate typing passwords and emails twice. Test this to see if it has a negative impact on registrations and if it creates lot of nonstarters (people who register but never return to the app.)
• Change the color of alerts to red instead of green because green is the site’s hyperlink color and also used for highlighting the selected field.
• The button should stand out. Call to actions typically work better when they are a different color from the rest of the site.  The button copy should be amped up a bit to “Get Started Editing”, “Save your photos now” or something similar too.

So how does Picnik capture users that don’t click register directly?  They offer it after a photo is saved:

As you can see, this page has a lot more content than the lightbox form since its a full page.  It has the job of pushing someone into registering after having used the product.  This is a good technique (mentioned in Javanovic’s article), but there’s always the question of if you’re offering too much or too little.  Testing how much to offer would be a very interesting and fruitful optimization campaign.

Overall, I’m not a huge fan of this page, but I do like the approach.  It has continuity at the top, showing the actual photo edited, and the form and main registration benefit (”Want Picnik to keep a copy?”) are prominent.  Also, they have structured the page to prioritize their conversion goals, keeping the focus on registration but still advertising the opportunity for people to print their photos or sign up for premium service below.

Here are a few recommendations to improve this page:

• What’s the clock icon for? Make the headline bigger or put in an informative image that will help encourage registration.
• Make the bullet points more prominent. The bullets disappear once the form begins to be filled out, using the same alert and field revealing technique I described with the previous form.  I would make sure the bullets stay on the page.
• Test all the copy.  It’s hard to know what feature is most important to users without testing.  Uploading more photos might be more appealing or saving their connections to Flickr and Facebook.
• Change the buttons. “Close photo” and “Create my account” look the same, they should be differentiated to emphasize their individual actions.  With a primary call to action, it needs to stand out.  Also, I would make the “Close photo” and “Continue editing” buttons much smaller to discourage immediate attention and clicks on those buttons, the point being to drive people to read the registration benefit copy.

Optimizing for registration involves many steps, beyond just improving the registration pages.  You can delve into when to ask for registration, test the ROI of emphasizing different products and then executing  segmentation focused pages as well.  However the easiest returns will come from some simple fixes like I’ve discussed above.

I hope this was helfpul talking over a real example, let me know if you’d like me to do more of these and if there’s any great sites out there I should look at.

If you missed it, see Part 1 (A/B Split Testing) and Part 2 (Multivariate Test Basics).

With the basics of part 2 down, it’s time to start designing a multivariate test.  Every optimization project has different challenges and goals, luckily though, there are a few rules that apply to every multivariate test design.  These rules fit into two categories: technical rules and content rules.

Technical rules:

1. Choose the appropriate multivariate test type (full or fractional factorial)
2. Determine the number of factors and levels that can be tested based on estimated conversion traffic (choose a test array)
3. Stop the test when it has stabilized, not based on your earlier estimations

These rules ensure statistical significance by constraining the test to the appropriate size at the beginning and then letting the test gather the proper amount of data at the end.

Running a test full factorial, if your traffic supports it, may be a good choice if you’re testing content that you believe to have many interactions or if you only want to test 2 factors with 2 levels each.  (Note: the smallest fractional factorial test size is 3 factors with 2 levels each.)  Typically though, you’ll want to run a fractional factorial test to save time and expand the number of factors and levels you can test.

In order to find out how many factors and levels you can test, you need to have some idea of your predicted page views, conversions, as well as an estimate of lift.  The reason that lift matters, is that a large lift will get you more conversions and so your test will stabilize quicker.  Because of this, I would be conservative with lift estimates to ensure that the test is not designed too large.  At Widemile, we have a large list of arrays available to our tool and have calculated the approximate conversions needed to stabilize, allowing me to look at the three criteria I listed and find the arrays that are statistically viable for testing.  You should look for something similar with your tool of choice.

To figure out when a test is stabilized, I prefer to primarily look at level influence stabilization with experiment conversion rate stabilization for support.  Widemile Optimize shows this using graphs, so I simply look for horizontal trending of lines, meaning winning levels and experiments stay winners and their level of influence or conversion rates stay fairly constant (look horizontal) over 3-5 days.  If you don’t have graphs available,  the historical cumulative conversion rate for your experiments and see if there is a lot of variance between the latest few days of your test.

Content rules:

1. Every item you test should answer an important question
2. Test variety not quantity
3. Test opposites first then refine
4. Remember you can run more than one test

The content rules are closely tied together.  In effect, they ensure that the items selected for testing have purpose and that they don’t needlessly expand the size of your test, reducing its efficiency.  I begin designing tests by creating hypothesis regarding issues with the page and then choose factors and design levels to address those issues.

An example hypothesis is “Having a hero shot on the right side of the page causes users to ignore the important value proposition on the left side.”  To test this, I would choose hero shot position as a factor and then have “left side hero shot” as the baseline level and “right side hero shot” as the second level.  This example also illustrates that, other than headlines and images, testing layout is possible with creative use of CSS and sometimes JavaScript.  As long as you can revert from one to another and it matches the other factors and levels, you are at liberty to test anything.

Coming back to the rules, make sure that you are testing as few items as possible to find out what you need.  Before testing a collection of lifestyle hero shots, choose one and test it against an iconic hero shot.  This will save you the time of going down a path of testing something that may not work.

Lastly, you aren’t going to be able to get the best page on the first run or even second, third, etc.  If you knew what your audience liked 100% of the time then you wouldn’t need testing.  Remember to think of your overall test plan beyond just the first run, so that you can answer all the questions you need without having to force everything into one test.

In summary, determine what you’re trying to achieve, select the proper testing method to meet those goals and then make sure to be purposeful and efficient with the content you end up testing in front of your visitors.  Testing and optimization is not difficult, although it can be tough to start.  Follow these rules and you’ll be on your way to conquering conversion rates, bounce rates, funnel drop-offs and many other metrics.

Photo credit: Aranda\Lasch (CC)

I recently read Douglas Bowman’s blog, Google’s former Visual Design Lead, about why he left Google.  In it, he describes how the engineering culture contributed to his decision to leave:

When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. [...] that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.

He then references Google testing 41 shades of blue and a recent debate he had over “whether a border should be 3, 4 or 5 pixels wide” and was asked to provide data to back up that decision.

Bowman’s post brought up some feelings of disappointment towards Google because despite having their own optimization tool, they did not create a culture to encourage their lead designer to expand his work and actually drove him away.  Optimization and testing is still in its early stage, so mistakes will be common, however I hope news like this doesn’t scare others away from testing.

Rather, I hope companies can learn from Bowman’s experience.  Instead of holding designers to every detail, testing should allow them to explore, learn and refine their ideas.  Testing should not prevent “any daring design decisions,” I feel it should actually encourage them.  As I said before, gamble with your conversions to raise them.

In the end, it’s all about having an understanding of how testing should and should not be used.  You can use testing to find the best shade of blue, but that doesn’t necessarily mean that’s what you should be testing right now.  Don’t be afraid to take a step back and try something new rather than fiddling with details, testing tools give you that freedom.  Big risks, reap big rewards in optimization.  Not taking risks leads to inefficient testing and, in Google’s case, a designer’s resignation.

Photo credit: i-marco (CC)

Selecting the right page to test is possibly the most important decision of an optimization campaign.  You can have great ideas, the technology and talent behind you, but if you pick the wrong page you could be doing a lot of work for minimal return.

So how do you get the biggest bang for your buck with testing?  Here’s a quick list of things to look for in a page:

• A single, specific and easy to measure conversion goal
• Sizable conversion traffic (at least 200 conversions in a week)
• A page that suffers from poor design or unclear conversion goal
• No large technical hurdles to implementing and executing the test
• A conversion rate that’s lower than comparable pages

Attacking pages with these attributes will get you some easy wins and help establish testing in your company.  Typically landing pages are the best pages to optimize, especially if they have the end conversion goal on the page, e.g. a form submission, download or click-out.

From there, I would move onto other pages in the funnel, taking a look at bounce rates to help determine where you need to help push visitors further into the funnel.  If there are no other pages in the funnel, find other poor or under performing pages on your site and take a look at them according to the rules above.

The main idea is to see that testing is a process and that just because you have ideas to improve a page, does not mean it is the best page to spend your time improving.

Photo credit: lepiaf.geo (CC)

If you missed it, see Part 1 (A/B Split Testing).  Update: Part 3 on Rules for a Successful Multivariate Test is here.

The technical and statistical aspects of multivariate testing can be complicated but in order to design successful tests you don’t need to know everything, just the basics of how it works and some guidelines.  I’m assuming you already have some understanding of multivariate testing, however I want to cover the basics and make sure we’re on the same level before going into how to design good multivariate tests.

Check out the wireframe below.  Pretty standard for a landing page, right?  To properly design a multivariate test, we have to look at the page in a certain way.  Using three key terms, factors, levels and experiments, we can break down a test and describe its framework.

Factor: An element of the Web page (headline, image, text) being tested.  The element can also be groups of content, e.g. left column, button and hero shot together, or all banner ads on the page.

Level: Content that is assigned to a specific factor to be tested.  For example, one variation of a hero shot.

Below are 4 factors from our example page (headline, hero shot, offer and button) and then each of those factors with 4 levels represented by the different colors.  Note that the levels of one factor do not have to relate in anyway to the levels of other factors.

The last term, experiments, makes use of both factors and levels.

Experiment: A unique combination of levels used during a test.

Here you can see 4 different experiments.  Each experiment is different and holds different combinations of levels.  Note that there actually are many more variations (4×4x4×4=256 combinations).

Essentially a multivariate test involves showing these experiments randomly to live traffic, while tracking how each experiment performs.  The one that performs the best wins.  Each experiment is shown to many people, but each person only sees one experiment.  (There is some complexity in this, if you are still confused or want to know more, go to my primer on full and fractional factorial testing.)

In my next post, I will use these terms to outline the rules to creating a great multivariate test.

Update: Check out Part 2 on Breaking Down Multivariate Testing and Part 3 on Rules for a Successful Multivariate Test.

Testing is not hard, but there are fundamentals that guarantee a successful optimization campaign.  To help get marketers up to speed with the basics, starting today, I will be writing about one topic per post and put together what I call Billy’s Optimization Guide.

The natural place to start is with a/b split tests, so let’s begin there.

First, a quick useful definition of an a/b split test: the competition of two distinct pages, where a portion of live traffic, usually 50%, is sent to one page and the rest to the other.  The winner is the page that provides the highest conversion rate, or whatever KPI is appropriate.

I want to emphasize that a good a/b split test requires distinct pages.  If that’s too vague, a simple rule that we follow at Widemile is:

You should be able to tell the difference between the 2 pages from 15 feet away.

Anything else isn’t a big enough change to be efficient in a split test and likely should go into a multivariate test.

With that definition in mind, here are three essential types of a/b split tests.  These are three of the tools in the testing toolbox that you should consider when putting together your optimization campaign.

1. Template test: Test the same general content (hero shot, copy, and button color) with a different layout and/or creative treatment.  The goal is to have a new template that better emphasizes the value proposition, improves readability and sets up well for a multivariate test.

   

   Use this when… you want to make sure you have a solid design, before or after testing messaging.  The majority of the time this should be your first test.

2. New concept test: Test a totally new approach.  Don’t let anything hold you back, test what you think will work best and see if it beats the original.  The approach here is to break out of the box and create a page that’s holistically designed around a new marketing concept.  Sometimes this involves introducing new functionality, animation, interactivity and other dramatic steps.  However it can also be on the smaller scale, such as introducing new messaging that requires a complete redesign.

   

   Use this when… your current page has already been tested many times and beating it has become difficult or you believe the way to really grab your visitors is through a big change.  This should only be done when the benefits of multivariate testing (knowing individual factor influences) are outweighed by the possible gains.

3. Funnel test: Send users to different multi-page experiences, e.g. no registration vs. requiring registration (below) and a one page form vs. a 3 page form. A funnel test can also be done with a multivariate but is simpler as an a/b split test.

   

   Use this when… you want to test content that extends past one page.  This should be done earlier in the testing process so that you don’t end up optimizing a page and then find out it’s a suboptimal experience.  It can be more technically demanding to do this sort of test though.

Every optimization campaign is different and so knowing what kinds of tests are available is one of the most important places to start.  For my next post, I will talk about the different ways to use a multivariate test.  Please post in the comments if you have any questions or contact me via Twitter @billysblog.

In elementary school, I played the game Hot or Cold in class.  The rules of the game are simple:

• One child is picked as the “searcher” and leaves the room
• The class collectively chooses an object in the room, like a marker or eraser, for the searcher to find
• Once the object is selected, the searcher returns to the room and has to find the mystery object as quickly as possible

To help the searcher out, the other kids in the room scream hot, if the searcher gets closer to the object, or cold, if they get farther.

To make the game more challenging, the searcher might be limited to only one clue, just hot or just cold.  Kids that were told both hot and cold found the objects fairly quickly, but if they were only allowed one type of feedback, it took them much longer.

For the same reasons that it is hard to find the object in the game without being told where it is closer and farther from, in testing, if you don’t design your tests with two distinct variations, you might go wandering for a long time trying to find what exactly your customer wants.

My metaphor fails in one way though.  In the game, the searcher does find the object eventually, even with just one type of hint.  However, If you don’t design tests correctly though, you may never find a page that resonates strongly with the audience.  You might test dozens of testimonials and find the most successful testimonial, but if you never test it against no testimonial or a review, you may be missing out on even bigger gains.

Let your audience tell you hot and cold by designing your tests intelligently and they’ll help you find the optimal page faster than ever.

Photo credit: Night Owl City CC

I will be at ad:tech New York on November 3-4 manning the Widemile Booth #922.  If you have any questions on optimization, landing pages or anything else, stop by and we can chat.

Also, I will be giving some mini-clinics on optimization best practices, how creative influences optimization and perhaps a case study or two based around those topics.  The last half of those clinics, I’d like to take a look at some live pages and review them.

But to be more effective, I’d like those live pages to be from volunteers.  If you have a page you’d like me to look at, send me a URL and a good time for you and I’ll try to make sure you get included in a clinic around that time.

Find me at ad:tech booth #922

If you catch me when I’m free, I also can take a look at your page and give you some of my thoughts on how to improve and test it.  Send me your URL’s beforehand for best results though.

Hope to see some of you there!