Beyond Implementation: Trusting the Data

November 17th, 2008 | April Moore

Once you have WebTrends implemented, you will likely find yourself, or someone else in your organization (perhaps even several someone’s), pouring over the data to determine if it’s accurate. Ideally, this testing and validation stage takes place before you start distributing the data internally and in a development or staging environment. That gives you time to address any potential issues — missing tags or forgotten filters — and put forward the best quality data right from the start.

But not everyone has the luxury of a testing environment or the time to do it before deployment. Even if you do, your testing environment may not match your live site exactly, or perhaps something was simply overlooked. These things can lead to data that is less accurate than you had hoped, quite apart from the inexactitude that plagues any form of web data. The problem is compounded further when you are moving from a different measurement tool all together, or moving from log files to client-side tagging. These factors combined can result in data that varies widely — sometimes shockingly so — from what you were expecting. And that means you could lose internal credibility, making it even harder to get the resources and attention you need.

Here are a few steps you can take to mitigate these risks and ensure that you not only capture the best data possible, but that you can distribute it confidently.

1. Before you implement any analytics tool, embark on an internal requirements gathering effort. If the reports you implement don’t meet your organization’s needs for actionable data, whether or not the data is accurate is just not as important as it could be.
2. Set expectations about how and when the data will change beforehand. Whether you are switching tools or just making small changes to your WebTrends configuration, you need to set expectations about the differences between the old data and the new data and give users a time line for when the change can be expected. If you are decreasing your session length from 30 minutes to 15 minutes, for example, your visit count is likely to increase and your number of visits by returning visitors will appear to increase as well. If this change is not widely communicated, users of your data will draw the wrong conclusions. Setting expectations whenever a change is made to your analytics tool is particularly critical if your organization has already been trending data. It is better to set the expectation that the numbers may change from what was reported historically than to let potential misinterpretations be revealed in an important executive meeting. And be sure to communicate “why” the change is being made — usually, that’s to improve overall reporting accuracy. (I’ll discuss using historical data in a future post, by the way.)
3. Increase your users’ knowledge of best practices, as well as of common errors. Schedule a training session with those people who will be reporting the data you deliver to walk them through their reports and answer any questions they may have about the numbers. Often these conversations will reveal areas where they have doubts about the data, lack basic knowledge of the WebTrends interface, or have been misinterpreting what is being shown. Addressing such concerns proactively goes a long way toward turning Doubting Thomases into allies and, one day, evangelists for your analytics data. Don’t feel like you have what it takes to be a trainer? Then refer them to WebTrends’ web-based training modules in the Customer Center. These materials address many questions with only a minimal time investment. Best of all, they’re free.*
4. Agree on reporting standards. The more widely your WebTrends data is distributed throughout the organization, the easier it is for metrics and Key Performance Indicators to be reported inconsistently. Set standards for how these metrics will be measured and reported, which reports and profiles will be used to deliver the data, and then keep an eye on the reports that get distributed. If you find someone who frequently misstates numbers or garbles a calculation, it’s time to schedule a one-on-one training session.
5. Set a time limit on how long testing and validation of the data will last. Although it sounds nice to say your data is completely accurate, the vagaries of the web make 100% accuracy in web data over any substantial length of time an impossibility. Months, even years, can be spent comparing numbers from different tools or numbers from different reports, without ever achieving certainty. But how accurate does the data need to be for you to make a decision based on it? Your objective in purchasing a web analytics tool was to get actionable data, not just to churn out reports, right? Establish a comfort level for data accuracy and then strive to hit that target. Set a cut-off date for testing and validation and establish a start date for your benchmarking period. Once that line in the sand is crossed, make sure everyone is measuring and reporting metrics consistently. If you are measuring and reporting consistently, as long as the trend goes up or down, your decision makers can act on it. And if you do discover a problem with your configuration that requires correction, something that brings you below that comfort level, refer back to tip #2.

It’s important for the people using your web data to understand that you don’t have complete control over the factors that can affect its accuracy. Client-side tagging solutions rely on data originating in your users’ browsers; their own choices of things such as browser, OS, privacy settings and ISP will impact the data, whatever you do. Even web server data is not proof against this, since what is requested and how it is requested are dependent on these same factors, plus you have the uncertainty of caching added into the mix. This is the world of the marketer, not the accountant. Nothing is certain, but you can still move forward. Address the inaccuracies you can. Establish repeatable processes. Focus on the trends.