Low traffic is probably the most common testing problem. Since traffic equals data, it is essential that you get enough traffic so that you can trust your data.
Unfortunately, low traffic also is one of the problems where people feel they have no control. Unless you can spend more money on advertising to drive traffic, there really is no way to increase the amount of traffic you get. Because of that, there’s a strong temptation to use the data that has been collected and move forward assuming that data is accurate. While that is an option, it should be an absolute last resort.
Before using unreliable data, I recommend either:
• Trimming down your test recipe matrix and testing fewer items, or
• Waiting it out
With this Open Campaign test, we had no idea how much traffic we’d receive since this was a brand new campaign and so designed a test based on some assumptions about the amount of traffic we expected to receive. Unfortunately the test we’ve designed is too large and so we are taking a look at our choices at this point.
Let’s assume we decided to reduce the amount of variations we were going to test. The way I’d go about doing this is to, one by one, cut out the items that we believed would perform the worst. This can be done by either eliminating a whole factor, such as stop testing button color completely, or cutting a few level variations, such as green and orange buttons. This would be done until I had a test that would fit the amount of traffic I am receiving. At Webtrends, we created a tool in order to help find test sizes based on traffic and conversion rates.
Beyond just working off of what you believe is correct, since we do have some data, it might be interesting to look at that and see if it can help choose what items should be tested.
From the Influence Stabilization graph, we can see that Button Copy clearly has the greatest influence on the page so far. Based on this, we would keep Button Copy in the test to see if it kept its high influence.
On the bottom is the Input Button, so since it has a low influence, we might consider testing new levels or a different factor.
The problem with doing this is that it is similar to what I was trying discourage you all to do in the beginning of this post, which is to trust the data as it is today. In the end, I would trust your marketer’s instinct and reduce the test size based on that, with the data as support if necessary.
In the case of this test, we’re still hopeful that we’ll get enough traffic over the course of the campaign, so we aren’t going to be making any changes. If we decide to though, you’ll be able to read about it here.
Let me know if you have had any low traffic situations and what you did in the comments section or feel free to ask me questions about the campaign or anything else optimization related.
