Hi again everyone!  I know I’ve been slow getting back to this thread, and for that I apologize.  It’s been a busy time here at Webtrends, and I was caught up in the whirlwind.  I haven’t forgotten my promise, though, to continue this thread, so here’s my next installment.  I’m skipping over the “How to read the Report” entries to post this one first about the numbers on the scenario analysis report.

What you’ve all pointed out in comments is true:  the numbers in this report don’t add up in the ways we might expect.  There’s a reason for this, though – actually, several reasons – that I’ll walk us through in this post (settle in; this is a thick read).   See, in a perfect scenario, all our visits would enter the process at the first step, convert through the steps in order, and complete the scenario without ever going anywhere else.  Most scenarios, though, have room for improvement – information is missing for the visitor, or a step is optional, or … well, you get the picture.  In such cases, it’d be useful for us to know how users meander in and out of a scenario, so we can identify steps we could improve upon, right?  That’s what this report is designed to do – and at the risk of sounding a bit self-serving, I have to say it does it quite well.  And that’s precisely why the numbers can seem so confusing – because this report follows the user’s meanderings, focusing on their activity rather than on totals.

Let me explain with an example (with many thanks to Xavier Le Hericy, who built the example!).  Let’s say I have three people visit my site and interact in a scenario.  Here are the paths each of them take:

Visit A:
(1) Views Page 1.
(2) Enters Step 1 (Product Page View) in the scenario.
(3) Views Page 2.
(4) Returns to Step 1.
(5) Views Page 3.
(6) Enters Step 3 (Started Checkout), skipping Step 2 entirely.
(7) Goes back to Step 2 (Cart Add).

Visit B:
(1) Views Page 1.
(2) Enters Step 1 (Product Page View) in the scenario.
(3) Views Page 2.
(4) Views Page 3.

Visit C:
(1) Views Page 1.
(2) Enters Step 1 (Product Page View) in the scenario.
(3) Goes directly to Step 2(Cart Add).
(4) Goes directly to Step 3 (Started Checkout).

None of them complete the checkout process for one reason or another.

Fairly straightforward visits, which lead to the following results in the scenario analysis report.  First, with step transitions:

Okay, let’s walk through our examples so we understand what we’re seeing here.  The text above from our engineering pals helps, but let’s translate it clearly into the examples we have above.

The Product Page View we see at the top left comes from Visit A. The two visits we see on the top right come from Visit A as well, though – in that visit, the user viewed a product page, then jumped to starting checkout.  So, if we try to add up numbers, we’d be literally tripling the number of visits – this entire line refers to a single visit, but show multiple paths through the scenario.  That’s one reason the numbers don’t add up.

Okay, moving on to Step 2.  Visit C converted from Step 1 to Step 2, so that’s the single visit we see next to the green arrow leading to Step 2.  But we also have Visit A represented once more – when the user moved from Step 3 to Step 2.  See how this is affecting the numbers?

Moving to Step 3, we see again that Visit C converted directly from one step to another – hence the 1 visit by the blue down arrow. Both Visit A and Visit C saw Step 3, which is why we have two visits at this step.  Visit A also is reflected on the left, since the user viewed a product page, then came to Step 3.  Finally, Visit A is also on the right, since it’s the visit that went back to Step 2 from this step.

So, this report shows us the flow of visits through the scenario steps, but there’s no way we could add up these numbers to get an appropriate number of visits to/from the scenario itself.  That’s just not what the report was designed to do!

Let’s check out the other view – scenario entry and exit pages:

Again, thanks to the Xavier for the explanations above – now let me walk you through the visits one more time.

Visits A, B, and C all enter the scenario from viewing page 1; therefore, they’re reflected both in the three visits on the left and the three visits to the product page view.  We don’t see the second visit to this step that took place; that’s reflected in the Step Transitions view instead.  On the right, we see that Visit B has moved on to another page and then never came back to the scenario – whoever they were, they played around elsewhere on the site, then headed out.

Visit C converted from Step 1 to Step 2, so that’s our one visit we see coming down to this step.  But we have two visits at this step, and none coming in on this step, which looks confusing until we take a look back at Visit A.  Ah – this visit did not convert from Step 1 to Step 2 (it jumped from 1 to 3 and back to 2), nor did it come into the scenario at Step 2 (it saw Step 1 first).  It was simply out of order – but it was still a visit to Step 2.  Hence, the two visits on this step – and we see another reason why these numbers won’t necessarily add up.

Visit A just keeps complicating things – this is the step from which this visit exited the scenario and either idled out of the visit or left the website entirely.  That’s the visit to the right of Step 2.

Visit C is now the only visit we have left, and it’s the visit that converted from Step 2 to 3 that we see.  However, again we have two visits on this step, and again it’s good ol’ Visit A that’s being reflected.  That was view #6 in Visit A’s progress on our website.  The visit we see exiting at this step is Visit C, since it didn’t move on to Step 4.

So, once again, these numbers won’t add up because of a couple of meandering folk on our website, weaving their way in and out of this scenario.  Multiply that by 10,000 visits or more, and you can start to see why these numbers seem out of whack on even infrequently-visited scenarios.

Let me say that I agree wholeheartedly that this lack of the clean-cut ability to add up numbers and see exactly how many visits were part of this scenario makes this report confusing.  However, that’s also what makes this report so valuable – we can see exactly what’s happening at any step in the scenario, and we can tell when visits meander through and out of the scenario.  That’s the actionable information this report is designed to provide – the kind of information that helps me optimize each step along the scenario, ensuring that my visitors always have the information they need to keep moving through the scenario and complete it.

Just to make you feel better, though, there actually are a couple of places in which the numbers add up.  Check out the image below:

Isn’t Xavier great? These are his notes, which were honestly a great relief to me.  I really wanted to be able to tell you that the numbers do add up, and here’s proof – some of the numbers really do!  Just not always the ones people ask you about.

So, I hope this post is helpful.  I’ve still got a couple more of these posts in the works, so I’d love to hear more about what you’d like to know about this report!

This entry was posted on Monday, June 15th, 2009 at 6:00 am and is filed under Customer Experience, User Experience. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.