Webtrends Optimization

A huge part of doing optimization well is knowing what to test (put garbage in, get garbage out), so keeping up with good design philosophy is extremely valuable.  While brushing up on web design, I came across a Smashing Magazine article on UI design trends by Janko Jovanovic.  He uses a lot of great examples of good design, some of which are perfect for illustrating some optimization options.

With that in mind, I’m going to examine one of the sites mentioned and discuss the good, the bad and the testing opportunities I see.  The (lucky?) site I picked was Picnik, which has done a commendable job on their registration strategy.  (Also, like Widemile, they are a Seattle start-up.)  I only wish the site wasn’t flash based, which is more difficult to optimize.  Despite that, my thoughts on test variations and best practices are still applicable to it and any other registration campaigns.

Quick summary: Picnik is an online photo editing application.  You can upload photos and do easy photo editing all within the browser.  You can try out the app, even exporting and saving photos, without registration.

Let’s get started by checking out their form:

Although a bit busy, I like the way the form assists users.  It has a green highlight to for the selected field and dynamically pops up error messages (see the username alert below).  Additionally, it hides and locks the “again” fields until there is valid input in the corresponding field.

One highlight is how this is a good example of when a lightbox/page overlay type form might be appropriate (note that behind the form is the page I was working on, which has been darkened).  Why is it appropriate?  Because this is the form that pops up after the user clicks “Register.”  It makes sense to be direct and reduce additional marketing if the user indicates they want to sign-up by clicking directly on the register button.

Is this right for your site/landing page/microsite?  It’s hard to say, but I would recommend testing it.  This would fall into the category of a funnel test because it eliminates a page in the registration funnel.  As long as your full page and lightbox form don’t have any glaring issues, you should quickly see the influence of whether a small and direct lightbox form works, or if a whole page with additional information is necessary.

In terms of testing this overlay form, there are a few big opportunities for improvement.

• Testing title and intro copy. Use “free” in the headline and as the first word, e.g. “Free registration”, then list a few benefits rather than saying “All we need is a username, password, and email address.”
• Eliminate typing passwords and emails twice. Test this to see if it has a negative impact on registrations and if it creates lot of nonstarters (people who register but never return to the app.)
• Change the color of alerts to red instead of green because green is the site’s hyperlink color and also used for highlighting the selected field.
• The button should stand out. Call to actions typically work better when they are a different color from the rest of the site.  The button copy should be amped up a bit to “Get Started Editing”, “Save your photos now” or something similar too.

So how does Picnik capture users that don’t click register directly?  They offer it after a photo is saved:

As you can see, this page has a lot more content than the lightbox form since its a full page.  It has the job of pushing someone into registering after having used the product.  This is a good technique (mentioned in Javanovic’s article), but there’s always the question of if you’re offering too much or too little.  Testing how much to offer would be a very interesting and fruitful optimization campaign.

Overall, I’m not a huge fan of this page, but I do like the approach.  It has continuity at the top, showing the actual photo edited, and the form and main registration benefit (”Want Picnik to keep a copy?”) are prominent.  Also, they have structured the page to prioritize their conversion goals, keeping the focus on registration but still advertising the opportunity for people to print their photos or sign up for premium service below.

Here are a few recommendations to improve this page:

• What’s the clock icon for? Make the headline bigger or put in an informative image that will help encourage registration.
• Make the bullet points more prominent. The bullets disappear once the form begins to be filled out, using the same alert and field revealing technique I described with the previous form.  I would make sure the bullets stay on the page.
• Test all the copy.  It’s hard to know what feature is most important to users without testing.  Uploading more photos might be more appealing or saving their connections to Flickr and Facebook.
• Change the buttons. “Close photo” and “Create my account” look the same, they should be differentiated to emphasize their individual actions.  With a primary call to action, it needs to stand out.  Also, I would make the “Close photo” and “Continue editing” buttons much smaller to discourage immediate attention and clicks on those buttons, the point being to drive people to read the registration benefit copy.

Optimizing for registration involves many steps, beyond just improving the registration pages.  You can delve into when to ask for registration, test the ROI of emphasizing different products and then executing  segmentation focused pages as well.  However the easiest returns will come from some simple fixes like I’ve discussed above.

I hope this was helfpul talking over a real example, let me know if you’d like me to do more of these and if there’s any great sites out there I should look at.

If you missed it, see Part 1 (A/B Split Testing) and Part 2 (Multivariate Test Basics).

With the basics of part 2 down, it’s time to start designing a multivariate test.  Every optimization project has different challenges and goals, luckily though, there are a few rules that apply to every multivariate test design.  These rules fit into two categories: technical rules and content rules.

Technical rules:

1. Choose the appropriate multivariate test type (full or fractional factorial)
2. Determine the number of factors and levels that can be tested based on estimated conversion traffic (choose a test array)
3. Stop the test when it has stabilized, not based on your earlier estimations

These rules ensure statistical significance by constraining the test to the appropriate size at the beginning and then letting the test gather the proper amount of data at the end.

Running a test full factorial, if your traffic supports it, may be a good choice if you’re testing content that you believe to have many interactions or if you only want to test 2 factors with 2 levels each.  (Note: the smallest fractional factorial test size is 3 factors with 2 levels each.)  Typically though, you’ll want to run a fractional factorial test to save time and expand the number of factors and levels you can test.

In order to find out how many factors and levels you can test, you need to have some idea of your predicted page views, conversions, as well as an estimate of lift.  The reason that lift matters, is that a large lift will get you more conversions and so your test will stabilize quicker.  Because of this, I would be conservative with lift estimates to ensure that the test is not designed too large.  At Widemile, we have a large list of arrays available to our tool and have calculated the approximate conversions needed to stabilize, allowing me to look at the three criteria I listed and find the arrays that are statistically viable for testing.  You should look for something similar with your tool of choice.

To figure out when a test is stabilized, I prefer to primarily look at level influence stabilization with experiment conversion rate stabilization for support.  Widemile Optimize shows this using graphs, so I simply look for horizontal trending of lines, meaning winning levels and experiments stay winners and their level of influence or conversion rates stay fairly constant (look horizontal) over 3-5 days.  If you don’t have graphs available,  the historical cumulative conversion rate for your experiments and see if there is a lot of variance between the latest few days of your test.

Content rules:

1. Every item you test should answer an important question
2. Test variety not quantity
3. Test opposites first then refine
4. Remember you can run more than one test

The content rules are closely tied together.  In effect, they ensure that the items selected for testing have purpose and that they don’t needlessly expand the size of your test, reducing its efficiency.  I begin designing tests by creating hypothesis regarding issues with the page and then choose factors and design levels to address those issues.

An example hypothesis is “Having a hero shot on the right side of the page causes users to ignore the important value proposition on the left side.”  To test this, I would choose hero shot position as a factor and then have “left side hero shot” as the baseline level and “right side hero shot” as the second level.  This example also illustrates that, other than headlines and images, testing layout is possible with creative use of CSS and sometimes JavaScript.  As long as you can revert from one to another and it matches the other factors and levels, you are at liberty to test anything.

Coming back to the rules, make sure that you are testing as few items as possible to find out what you need.  Before testing a collection of lifestyle hero shots, choose one and test it against an iconic hero shot.  This will save you the time of going down a path of testing something that may not work.

Lastly, you aren’t going to be able to get the best page on the first run or even second, third, etc.  If you knew what your audience liked 100% of the time then you wouldn’t need testing.  Remember to think of your overall test plan beyond just the first run, so that you can answer all the questions you need without having to force everything into one test.

In summary, determine what you’re trying to achieve, select the proper testing method to meet those goals and then make sure to be purposeful and efficient with the content you end up testing in front of your visitors.  Testing and optimization is not difficult, although it can be tough to start.  Follow these rules and you’ll be on your way to conquering conversion rates, bounce rates, funnel drop-offs and many other metrics.

Photo credit: Aranda\Lasch (CC)

Update: Check out Part 2 on Breaking Down Multivariate Testing and Part 3 on Rules for a Successful Multivariate Test.

Testing is not hard, but there are fundamentals that guarantee a successful optimization campaign.  To help get marketers up to speed with the basics, starting today, I will be writing about one topic per post and put together what I call Billy’s Optimization Guide.

The natural place to start is with a/b split tests, so let’s begin there.

First, a quick useful definition of an a/b split test: the competition of two distinct pages, where a portion of live traffic, usually 50%, is sent to one page and the rest to the other.  The winner is the page that provides the highest conversion rate, or whatever KPI is appropriate.

I want to emphasize that a good a/b split test requires distinct pages.  If that’s too vague, a simple rule that we follow at Widemile is:

You should be able to tell the difference between the 2 pages from 15 feet away.

Anything else isn’t a big enough change to be efficient in a split test and likely should go into a multivariate test.

With that definition in mind, here are three essential types of a/b split tests.  These are three of the tools in the testing toolbox that you should consider when putting together your optimization campaign.

1. Template test: Test the same general content (hero shot, copy, and button color) with a different layout and/or creative treatment.  The goal is to have a new template that better emphasizes the value proposition, improves readability and sets up well for a multivariate test.

   

   Use this when… you want to make sure you have a solid design, before or after testing messaging.  The majority of the time this should be your first test.

2. New concept test: Test a totally new approach.  Don’t let anything hold you back, test what you think will work best and see if it beats the original.  The approach here is to break out of the box and create a page that’s holistically designed around a new marketing concept.  Sometimes this involves introducing new functionality, animation, interactivity and other dramatic steps.  However it can also be on the smaller scale, such as introducing new messaging that requires a complete redesign.

   

   Use this when… your current page has already been tested many times and beating it has become difficult or you believe the way to really grab your visitors is through a big change.  This should only be done when the benefits of multivariate testing (knowing individual factor influences) are outweighed by the possible gains.

3. Funnel test: Send users to different multi-page experiences, e.g. no registration vs. requiring registration (below) and a one page form vs. a 3 page form. A funnel test can also be done with a multivariate but is simpler as an a/b split test.

   

   Use this when… you want to test content that extends past one page.  This should be done earlier in the testing process so that you don’t end up optimizing a page and then find out it’s a suboptimal experience.  It can be more technically demanding to do this sort of test though.

Every optimization campaign is different and so knowing what kinds of tests are available is one of the most important places to start.  For my next post, I will talk about the different ways to use a multivariate test.  Please post in the comments if you have any questions or contact me via Twitter @billysblog.

In elementary school, I played the game Hot or Cold in class.  The rules of the game are simple:

• One child is picked as the “searcher” and leaves the room
• The class collectively chooses an object in the room, like a marker or eraser, for the searcher to find
• Once the object is selected, the searcher returns to the room and has to find the mystery object as quickly as possible

To help the searcher out, the other kids in the room scream hot, if the searcher gets closer to the object, or cold, if they get farther.

To make the game more challenging, the searcher might be limited to only one clue, just hot or just cold.  Kids that were told both hot and cold found the objects fairly quickly, but if they were only allowed one type of feedback, it took them much longer.

For the same reasons that it is hard to find the object in the game without being told where it is closer and farther from, in testing, if you don’t design your tests with two distinct variations, you might go wandering for a long time trying to find what exactly your customer wants.

My metaphor fails in one way though.  In the game, the searcher does find the object eventually, even with just one type of hint.  However, If you don’t design tests correctly though, you may never find a page that resonates strongly with the audience.  You might test dozens of testimonials and find the most successful testimonial, but if you never test it against no testimonial or a review, you may be missing out on even bigger gains.

Let your audience tell you hot and cold by designing your tests intelligently and they’ll help you find the optimal page faster than ever.

Photo credit: Night Owl City CC

You and your competitor’s all have the same landing pages.  You have a hero shot of the product, a big call to action button and short, punchy copy.  Or maybe you’re already ahead of your competitors and have run a few tests on your page, picking up more conversions on the way.  In either situation, you’ll eventually hit a wall and struggle to get additional lift.  So how do you continue to improve?

Go for broke.  Try something you’ve never tried before.  It might end up being a total failure, but it also might give you the lift you want.

The gamble you make with optimization can end in 2 ways:

• You lose X amount of conversions over the week or two that the test is running
• You gain X amount of conversions for the effective lifetime of the page

The possible upside dwarfs the downside by a large margin and, either way, you learn something new and can optimize the next test more successfully based on what you learned.

Luckily, with skill and experience, the risks of testing are minimized, however beating a strong page is never easy or guaranteed.  But when you do find something new that works or see that your current page still is a champ, you can rest assured that you’re doing all you can to drive conversions.

What are full and fractional factorial test designs? How do they relate to optimization and what about interactions?

Once you get down and dirty with testing, these questions matter. Whether selecting an optimization platform or trying to thoroughly understand the tests you are building, grasping these concepts will put you in greater control and allow you to design and analyze your tests more effectively.

As simply as possible, I hope to educate you and other marketers about full and fractional factorial test designs and why fractional factorial is the best choice for multivariate testing of online campaigns.

Note: “Partial factorial” and “fractional factorial” are the same. Also, if you don’t have a thorough understanding of experiments and interactions, please read those first.

The tests used in optimization are from the design of experiments field. (From Wikipedia: “Design of experiments is the design of all information-gathering exercises where variation is present, whether under the full control of the experimenter or not.”) The two types of tests I will focus on are fractional factorial and full factorial.

Here is an example I will use to explain these concepts. Below is a test matrix outlining a test for a landing page with 5 factors with 2 levels each. Don’t let the vocabulary scare you away, this means that there are 5 parts of the page being tested and 2 variations of each.

Recipe Matrix: 5 factors = 5 parts (hero shot, headline, etc.) and 2 levels = 2 variations

These factors and their respective levels make up the possible combinations for a landing page. The combinations displayed are called experiments.

Let’s calculate the total number of experiments possible (even if you know how to do this already, this is important to understanding the distinction between fractional and full factorial.) There are 2 levels for each factor, so you can have 2×2x2×2x2 (2 to the 5th power) = 32 possible experiments. This means there are exactly 32 combinations of hero shots, headlines, sub headlines, button text and main copy from our matrix outlined above. Note that if we add another factor, it becomes 2 to the 6th power or 64 possible experiments. Additionally, if you add 2 more levels to any of the existing 5 factors, it will increase from 32 to 4×2x2×2x2 = 64 experiments also.

In testing, each experiment must get a minimum amount of measurable conversions, known as the sample size per experiment. This ensures that there is enough data for a solid statistical analysis. Therefore the more experiments you have, the more conversions you need. You can think of conversion data as time also, since the longer you leave your web page up, the more data you get.

Now we’re ready to go back to the difference between the two test designs. Full factorial testing requires that every possible experiment combination is shown, so our 5-factor test would need to display all 32 experiments. This means that if there is a sample size of 100 conversions, 3,200 conversions will be required. Fractional factorial works differently, it displays a much smaller number of experiments, about 8 in this case, so it would need about 800 conversions.

Since full factorial gathers additional data, it reveals all possible interactions, but as seen by the numbers above, there is a trade-off. More data equals more information but more data also equals a longer test duration. The minimum data requirements for full factorial are very high since you are showing every experiment.

Even if you are using full factorial to get the same amount of information as a fractional factorial test, it will take more time since you need more data to see statistically relevant differences between the many experiments.

You might be wondering how fractional factorial can be accurate if interactions are possible?

Random interactions of high relevance are very rare, especially when looking for interactions of more than 2 factors. You really need to design tests where you look for meaningful interactions that are based on true business requirements rather than hoping for a random and low influence interaction between a red button, a hero shot and a headline.

Whatever the interaction is, you need to be able to understand your audience and infer why there was an interaction in the first place, only then are you ready to start designing for interactions.

Tests should not be filled with random levels, they should be carefully designed for success by focusing on testable hypotheses around the audience. Could a 1 pixel drop shade on a button interacting with the copyright statement ever be truly significant, and not a victim of random error? Is it worth sacrificing thousands of conversions to learn a lesson that won’t result in any relevant increase of real world conversions?

There are interactions that might make sense and those that should be avoided from being measured because of the amount of testing time it adds.

This brings me to fractional factorial. It is possible for fractional factorial tests to detect interactions. How so? Using our example of a 5-factor test, fractional factorial can include everything from only main-effects all the way to 4-factor interaction effects. Full factorial’s only difference is that it is the full extension and includes the 5-factor interaction effects.

Fractional factorial is not a one-trick pony, it is a continuum ranging from testing for no interactions (only main effects) to one factor less than full factorial. It is exactly what the name fractional implies; even one less is a “fraction” of full factorial. It gives you the power to make trade-offs between testing only main effects to testing for interactions based on intelligent test design.

Once you decide to test for all possible interactions, you are committing to a full-factorial test and incur the associated traffic requirements. I’d love to see a test design that is designed for full interactions and still makes sense! Not having the ability to reduce the number of interactions is a huge detriment rather than a benefit of solutions limited to full-factorial testing.

Radically shorter test times allow for many more smart marketing ideas to be tested and adapted based on what you learn from each test run. You, the marketer have the ability to analyze your results and tweak follow-on tests to capitalize on what you learn. This common-sense approach is what hypothesis-based testing is all about and is very powerful. Focus on testing smart ideas to increase your conversion rate – that’s what matters most.

The graph below illustrates how much information is gained and the amount of testing needed, based on the number of interactions tested.

In my experience, the red area shows how valuable the data is based on which effects are being tested, while the blue area shows the amount of data (or time) needed to gather the data to confirm those effects. The x-axis goes from left to right, from main effects to full factorial (5-factor effects).

At Widemile, we believe it is more effective to perform quick, successive tests detecting only main-effects rather than randomly hoping for interactions. While interactions might give you small or even large gains, it likely will never not trump the gains from additional testing, nor the time and money lost looking for random interactions. The additional time required for full factorial tests is large and not many marketers want to wait more than a month for a test to complete.

Fractional factorial is preferred by a few camps, including Widemile, Omniture’s Test&Target (formerly Offermatica) and Interwoven’s Optimost. Full factorial is used in Google’s free Website Optimizer and some tools offered by smaller providers.

Testing for all interactions sacrifices a lot of time. With the speed that audiences, marketing campaigns and seasons can change, it is important to get the most testing done in the least amount of time without sacrificing the quality of the data. Fractional factorial allows you to do just that, making it the wisest choice for multivariate testing.

A beginner’s mistake is to test every idea with every test. This is the most obvious way of being efficient. If I can test 50 things in a week, why not?

In my experience, efficiency has more to do with careful test design and doing things right the first time, than trying to test everything and rushing the process. By testing a few big ideas quickly and then designing the next test based on those results, you can do a set of small tests and get answers fast without having to risk your page to many bad ideas.

Every test should have specific questions its trying to answer. Not just “What’s the best performing page?” but questions that lead to that. A car salesman doesn’t blindly try every tactic in the book get you to buy a car, a real salesman probes you with a few questions and changes their technique accordingly.

That’s how you should design your tests.

Here’s an example test plan that works for most clients:

• Step 1 (Split Test) – Find an optimal template/design: What template and/or design effectively gets visitors to stick, click and convert? At this stage, you aren’t testing messaging yet, you’re merely re-skinning and moving elements around to find a good design. Some techniques to use are simplifying the page by de-emphasizing unimportant content (shrink company logo, move ads to the bottom of the page) and emphasizing core content (moving 3rd party validation near the call to action) and adding more whitespace to the page to enhance readability. These are in addition to a well done creative design. This test usually has the greatest impact, however it all depends on your current page and the audience. (Read more on template testing)
• Step 2 (Multivariate Test) – Find the biggest converting segment: This test focuses on finding the correct messaging by appealing to different segments that you know and hypothesize visit your page. If your product was Google Apps, you might test appealing to business users and freelancers. Or if you are selling a cell phone, you might test features against benefits.
• Step 3 (Multivariate Test) – Find the perfect way to communicate to the segment: Step 2 points you in the right direction, but this step helps you find the exact place you should be with your page. Use what you learned (freelance messaging won) and try variations on that winning theme to really grab your audience and give them what they want. Also, step 2 may have revealed 2 or more segments that are worth targeting. If you can segment them out, run multiple tests that are customized for each segment, and you’ll raise conversions even higher.

The alternative is to test 50 ideas of which many of the ideas overlap. Why test any ideas that are remotely similar until you know that they work in general? If I go to a dealership wanting a sports car and the dealer offers me 5 colors of minivans, I’m still not going to buy a minivan. Show me 4 types of cars, let me pick the one I like and then we might talk about color.

Let your visitors lead you!

This really is a simple process, but it drives results. Be methodical to be efficient. By course correcting in each test, you get closer and closer to what you need and don’t spend a lot of time testing losing elements. Follow a test plan like this and you’ll get results and learn a lot about your core converting visitors.

Note: This is the third post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the other 2 articles on highly mixed data and the original page beating the new variations.

Ready for the toughest of all test results? I brought in Widemile’s Chief Scientist, Vladimir Brayman, for this post to help me with some of the concepts around this topic. The last of the three results is when the results just won’t stabilize.

How does this happen?
As long as you have homogeneous traffic and enough time, a test should stabilize. Unfortunately, this is not always possible and I don’t know anyone with unlimited time. The most obvious way this occurs is when a test is designed too large, meaning you don’t have enough conversion traffic for the number of variations you are trying to test.

Additionally, getting homogenous traffic is not always easy. If your sources are too different, you can have problems. Text, banner, e-mail ads and even Yahoo vs Google traffic may behave differently. The worst case is when these sources of traffic are added mid-test. I have had tests where an e-mail campaign was done at the end of a test without my knowledge (until I asked about the huge spike in traffic!)

You can’t control all traffic coming to your page from some sources like PR, blogs, seasonal events and news. This goes back to part 1, about highly mixed data; everything there applies to this case too.

A test also may not stabilize because the test is designed with elements that are too similar. The same thing can happen when 2 elements are different but have approximately the same amount of impact. In these situations, your data will go back and forth on which of them are the winners.

Anything outside of your page that has a large influence can destabilize your test, this includes pieces of your funnel. One symptom of this is when your clickthroughs are fairly consistent but the full conversions are not. If you are testing a landing page and the sign-up process after it is very kludgey and difficult for users then it can have a large impact on your tests’ ability to stabilize. This is especially true if the experience for visitors changes. An example of this is visitors bailing from a purchase funnel because shipping to their area is prohibitively more expensive than other areas. Although they would have converted if shipping was within the average price range, they ended up not converting because of something encountered outside of the landing page, skewing your results. This is in almost every test, but the magnitude of its impact depends on what exactly occurs.

What can you do to prevent this?

If you are using a testing tool different from what you normally track your conversions with, make sure you run a baseline test so that you can compare the numbers your testing tool gives you with the ones your conversion analytics produces. They should be within about 10%-15% of each other over about a week or so. Finding a large discrepancy here will save you from headaches down the line. This essentially double checks the expected traffic numbers by ensuring you are measuring your current conversion correctly, which allows you to design a test of the appropriate size. By size, I mean ensure that you have enough testing time and within that time you will get enough traffic.

While easier said than done, it is important to look for new traffic that may be driven to your page and to segment it out. Since this shares some of the same problems as highly mixed data, those solutions apply here too.

What can you do if this happens?

First, don’t cut your tests short unless you think more data won’t solve the problem. If you don’t reach stabilization, you are wasting all the time you tested since you have inconclusive data. Always try to be as conservative as possible and end tests only when you are very confident that the test is stabilized or that there is no other choice.

Think about restarting the test if it isn’t stable. Use a smaller design. Pick the important factors (pieces) and the levels (variations) that you think will perform and are drastically different from each other. This prevents elements from looking unstable as they flip flop as the optimal.

If your only problem is that 2 variations are vying for the winning position, then they likely perform about the same. It probably is not really worth your time to wait for them to stabilize and so stopping the test and going with either of them likely will have little difference to your conversion rates.

The problem of outside funnel influence is a bit harder, but not impossible to solve. The best solution is to segment the users that are determined to be unqualified. For example, if you only ship or work with US customers and businesses, then filter out any users that are outside of the US and do your analysis from there. This can be done either at the data level if you can tell where the data came from, otherwise this can be done with a splitter or qualification page that leads people into the appropriate funnel first. This may impact your overall conversions itself though, so careful testing around these methods should be done as well.

From my experience, the problems I’ve listed in these three posts are either preventable or unlikely to occur. The value of having an optimization expert is because they can avoid these situations or at the very least extract useful lessons when they do happen. Having said that, don’t be scared to test. Once you get the hang of it, it is a lot of fun and one of the keys to effectively growing and maturing your online marketing campaigns.

CC photo credit #1: ryaninc – CC photo credit #2: jurvetson

Note: This is the second post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the first article on highly mixed data.

As an optimization analyst, this is probably the hardest result to bring to a client. Oddly enough, it actually is favorable to part 1’s highly mixed data and part 3. I am talking about optimization that determines that the original page is better than the tested variations.

How does this happen?
Sometimes a page just gets it right. How would you change Google? I looked for a few variations and came across one by Andy Rutledge and another by Valacar. They both are beautiful designs and a lot of thought were put into them, but at the same time, would they really make Google more profitable? It’s definitely a tough sell and there is a big challenge in improving this type of page.

The goal is for users to search. Yes, they want users to click on ads eventually, but there’s not a whole lot they can do for ad clicks on the homepage. The best they can do is get users to search as fast as possible. So would a redesign make it more usable and readable? Maybe. To a level that it would increase their revenues? That’s tough to say.

The more simple the goals of the page, the less information and messaging the users needs, the more likely that the page will be difficult to optimize.

What can you do to prevent this?
Be careful when choosing a page to test. Find a page where the user will take some time to look at what is going on. This is another reason why most landing pages are great places to optimize, because users naturally need to be introduced to the product and sold on why to convert.

The logical thing to do would be to simply refrain from testing pages that seem to be performing well, but this is rarely a good rule. Unless it is performing well because of a lot of testing, then you don’t really know if a page is performing well or not (see my post on conversion rates.) Testing always brings surprises and personal judgment is no replacement for a test; a good looking page can perform poorly and a page with subpar creative can perform great.

What can you do if this happens?

Because of the above reasons, you may actually plan for this scenario to occur. Many people believe redesigning an old page will provide improvement, but what if it is old and performing well? In that case, you may plan to try to improve but not expect to beat the old version.

In any case, if your original page wins, then you have confirmation of your page’s success. It is unlikely that all possible improvements were tested in one test run though, so it may take a few more runs to really confirm its solidarity, but the page has won against the initial best ideas and that is an achievement.

This lesson tells you that you can move on and that is progress in itself.

Moving forward, I would try drastically different approaches, either in layout or design and testing around offers. Otherwise, I would apply the successful original page to tests for other areas of your site.

I have to be honest when I say that this rarely ever happens. Almost every page has room for improvement at every step of the conversion funnel.

Whew, I will try to get the third and toughest optimization result next week.

CC photo credit: philosophygeek

Every test teaches something. Almost every campaign test has the goal of raising conversion rates, but really the power of testing is in answering questions. It is just a happy coincidence that it raises your conversions at the same time.

Here are a few of the questions that can be answered with testing:

1. Is giving away a free widget worth while? Would a cheaper widget perform better?
2. Does promoting a few key products work better than promoting them all equally?
3. How much should this product be priced?
4. Does e-mail traffic respond to the same things as SEO and advertising traffic?
5. What benefits are consumers interested in?
6. Will a discount offer make up in conversions, what is lost in revenue per sale?
7. Do banner ads reduce conversions?
8. Is an extra learn more page more effective than a longer page? Is extra information even necessary?

Answering these questions is pretty easy, but you have to think about them beforehand. Design your test so that it asks questions and offers answers. Figure out what questions you want to ask and based on those, give a few separate and distinct answers in the form of variations of your page.

The easy ones are the ROI questions, e.g. giving away widgets, discounts, banner ads, and pricing. Just include them on or off in the test and/or with different amounts. At the end, do a ROI analysis comparing the conversion rates of each variation you test.

If you are wondering if one group of traffic responds differently to a campaign than another, segmentation is what you need. Separately track those segments, but don’t forget to design your test with answers too. Make your variations appeal to different audiences and your segments will be pulled towards the one they like the most with their conversions as proof.

If you want to learn what your customers are looking for, do some market research through testing. Test different types of benefits and see which get the visitors to convert, e.g. technical (this camera contains 1 gb of memory) versus lifestyle (this camera stores 100’s of wedding and birthday pictures).

You can test these ideas in an A/B split test or a multivariate test, but a multivariate test is much quicker and will allow you to test multiple questions and answers simultaneously.

Testing gives you a lot of answers and the better you design those questions into your tests, the more sense those answers will make of your data. Conversion rates are always important, but focusing solely on them won’t get you great results time after time. Think about testing in terms of learning more about your audience and you will find continual improvement in your campaigns.