Webtrends Optimization

A huge part of doing optimization well is knowing what to test (put garbage in, get garbage out), so keeping up with good design philosophy is extremely valuable.  While brushing up on web design, I came across a Smashing Magazine article on UI design trends by Janko Jovanovic.  He uses a lot of great examples of good design, some of which are perfect for illustrating some optimization options.

With that in mind, I’m going to examine one of the sites mentioned and discuss the good, the bad and the testing opportunities I see.  The (lucky?) site I picked was Picnik, which has done a commendable job on their registration strategy.  (Also, like Widemile, they are a Seattle start-up.)  I only wish the site wasn’t flash based, which is more difficult to optimize.  Despite that, my thoughts on test variations and best practices are still applicable to it and any other registration campaigns.

Quick summary: Picnik is an online photo editing application.  You can upload photos and do easy photo editing all within the browser.  You can try out the app, even exporting and saving photos, without registration.

Let’s get started by checking out their form:

Although a bit busy, I like the way the form assists users.  It has a green highlight to for the selected field and dynamically pops up error messages (see the username alert below).  Additionally, it hides and locks the “again” fields until there is valid input in the corresponding field.

One highlight is how this is a good example of when a lightbox/page overlay type form might be appropriate (note that behind the form is the page I was working on, which has been darkened).  Why is it appropriate?  Because this is the form that pops up after the user clicks “Register.”  It makes sense to be direct and reduce additional marketing if the user indicates they want to sign-up by clicking directly on the register button.

Is this right for your site/landing page/microsite?  It’s hard to say, but I would recommend testing it.  This would fall into the category of a funnel test because it eliminates a page in the registration funnel.  As long as your full page and lightbox form don’t have any glaring issues, you should quickly see the influence of whether a small and direct lightbox form works, or if a whole page with additional information is necessary.

In terms of testing this overlay form, there are a few big opportunities for improvement.

• Testing title and intro copy. Use “free” in the headline and as the first word, e.g. “Free registration”, then list a few benefits rather than saying “All we need is a username, password, and email address.”
• Eliminate typing passwords and emails twice. Test this to see if it has a negative impact on registrations and if it creates lot of nonstarters (people who register but never return to the app.)
• Change the color of alerts to red instead of green because green is the site’s hyperlink color and also used for highlighting the selected field.
• The button should stand out. Call to actions typically work better when they are a different color from the rest of the site.  The button copy should be amped up a bit to “Get Started Editing”, “Save your photos now” or something similar too.

So how does Picnik capture users that don’t click register directly?  They offer it after a photo is saved:

As you can see, this page has a lot more content than the lightbox form since its a full page.  It has the job of pushing someone into registering after having used the product.  This is a good technique (mentioned in Javanovic’s article), but there’s always the question of if you’re offering too much or too little.  Testing how much to offer would be a very interesting and fruitful optimization campaign.

Overall, I’m not a huge fan of this page, but I do like the approach.  It has continuity at the top, showing the actual photo edited, and the form and main registration benefit (”Want Picnik to keep a copy?”) are prominent.  Also, they have structured the page to prioritize their conversion goals, keeping the focus on registration but still advertising the opportunity for people to print their photos or sign up for premium service below.

Here are a few recommendations to improve this page:

• What’s the clock icon for? Make the headline bigger or put in an informative image that will help encourage registration.
• Make the bullet points more prominent. The bullets disappear once the form begins to be filled out, using the same alert and field revealing technique I described with the previous form.  I would make sure the bullets stay on the page.
• Test all the copy.  It’s hard to know what feature is most important to users without testing.  Uploading more photos might be more appealing or saving their connections to Flickr and Facebook.
• Change the buttons. “Close photo” and “Create my account” look the same, they should be differentiated to emphasize their individual actions.  With a primary call to action, it needs to stand out.  Also, I would make the “Close photo” and “Continue editing” buttons much smaller to discourage immediate attention and clicks on those buttons, the point being to drive people to read the registration benefit copy.

Optimizing for registration involves many steps, beyond just improving the registration pages.  You can delve into when to ask for registration, test the ROI of emphasizing different products and then executing  segmentation focused pages as well.  However the easiest returns will come from some simple fixes like I’ve discussed above.

I hope this was helfpul talking over a real example, let me know if you’d like me to do more of these and if there’s any great sites out there I should look at.

Note: This is the third post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the other 2 articles on highly mixed data and the original page beating the new variations.

Ready for the toughest of all test results? I brought in Widemile’s Chief Scientist, Vladimir Brayman, for this post to help me with some of the concepts around this topic. The last of the three results is when the results just won’t stabilize.

How does this happen?
As long as you have homogeneous traffic and enough time, a test should stabilize. Unfortunately, this is not always possible and I don’t know anyone with unlimited time. The most obvious way this occurs is when a test is designed too large, meaning you don’t have enough conversion traffic for the number of variations you are trying to test.

Additionally, getting homogenous traffic is not always easy. If your sources are too different, you can have problems. Text, banner, e-mail ads and even Yahoo vs Google traffic may behave differently. The worst case is when these sources of traffic are added mid-test. I have had tests where an e-mail campaign was done at the end of a test without my knowledge (until I asked about the huge spike in traffic!)

You can’t control all traffic coming to your page from some sources like PR, blogs, seasonal events and news. This goes back to part 1, about highly mixed data; everything there applies to this case too.

A test also may not stabilize because the test is designed with elements that are too similar. The same thing can happen when 2 elements are different but have approximately the same amount of impact. In these situations, your data will go back and forth on which of them are the winners.

Anything outside of your page that has a large influence can destabilize your test, this includes pieces of your funnel. One symptom of this is when your clickthroughs are fairly consistent but the full conversions are not. If you are testing a landing page and the sign-up process after it is very kludgey and difficult for users then it can have a large impact on your tests’ ability to stabilize. This is especially true if the experience for visitors changes. An example of this is visitors bailing from a purchase funnel because shipping to their area is prohibitively more expensive than other areas. Although they would have converted if shipping was within the average price range, they ended up not converting because of something encountered outside of the landing page, skewing your results. This is in almost every test, but the magnitude of its impact depends on what exactly occurs.

What can you do to prevent this?

If you are using a testing tool different from what you normally track your conversions with, make sure you run a baseline test so that you can compare the numbers your testing tool gives you with the ones your conversion analytics produces. They should be within about 10%-15% of each other over about a week or so. Finding a large discrepancy here will save you from headaches down the line. This essentially double checks the expected traffic numbers by ensuring you are measuring your current conversion correctly, which allows you to design a test of the appropriate size. By size, I mean ensure that you have enough testing time and within that time you will get enough traffic.

While easier said than done, it is important to look for new traffic that may be driven to your page and to segment it out. Since this shares some of the same problems as highly mixed data, those solutions apply here too.

What can you do if this happens?

First, don’t cut your tests short unless you think more data won’t solve the problem. If you don’t reach stabilization, you are wasting all the time you tested since you have inconclusive data. Always try to be as conservative as possible and end tests only when you are very confident that the test is stabilized or that there is no other choice.

Think about restarting the test if it isn’t stable. Use a smaller design. Pick the important factors (pieces) and the levels (variations) that you think will perform and are drastically different from each other. This prevents elements from looking unstable as they flip flop as the optimal.

If your only problem is that 2 variations are vying for the winning position, then they likely perform about the same. It probably is not really worth your time to wait for them to stabilize and so stopping the test and going with either of them likely will have little difference to your conversion rates.

The problem of outside funnel influence is a bit harder, but not impossible to solve. The best solution is to segment the users that are determined to be unqualified. For example, if you only ship or work with US customers and businesses, then filter out any users that are outside of the US and do your analysis from there. This can be done either at the data level if you can tell where the data came from, otherwise this can be done with a splitter or qualification page that leads people into the appropriate funnel first. This may impact your overall conversions itself though, so careful testing around these methods should be done as well.

From my experience, the problems I’ve listed in these three posts are either preventable or unlikely to occur. The value of having an optimization expert is because they can avoid these situations or at the very least extract useful lessons when they do happen. Having said that, don’t be scared to test. Once you get the hang of it, it is a lot of fun and one of the keys to effectively growing and maturing your online marketing campaigns.

CC photo credit #1: ryaninc – CC photo credit #2: jurvetson

A big mistake in testing is to overlook variables inside and outside of the test that impact results. In an ideal test, the only variables would be the ones you are testing on your page. That usually isn’t possible though, but as long as you account for them in your analysis, you will get correct and actionable information.

If you test a seasonal page, then the optimal page you get for that season, probably won’t perform when the season ends. By not paying attention to those kind of variables, you are setting yourself up into thinking you’ve found the optimal page. The same type of mistake is made by grouping e-mail, print, SEM campaigns and event traffic, unless you know they react the same to your changes.

Even within segments, there might be more segments to uncover. Your only limitation should be traffic; don’t segment so granular that you can’t run a decent sized test in a decent amount of time.

One of my clients doesn’t get a lot of traffic, but the traffic he does get is very distinct. One converts in the single digits and the other converts in the teens. Although combining them would get me more data, it would be very confused data since they convert so differently.

A few things to look out for:

• The ad or offers visitors see beforehand
• Interactions between your factors (if you aren’t testing interactions)
• Technical problems
• Problems that occur before or after the tested page

A note about the last bullet, the problems can range from a technical problem to a problem with the overall funnel. If people get different experiences in the funnel that drastically impact whether they convert or not, it can add a noise to your test. Some examples are different checkout processes for registered and non-registered users or users being inelligible for service.

The purpose of testing is to find out if a certain element performs well under the conditions you provide. If you aren’t paying attention to all the conditions, then the results you derive will be incorrect without you knowing.