Demystifying the Scenario Analysis Report, Part I: Understanding Fall-out

April 29, 2009 Sandra Elliot — 12 comments

One of the most complex reports we have available in Webtrends is the scenario analysis report. It’s also one of our most robust, and can provide you with worlds of good information to help you optimize scenarios on your site. But I’ve found that a lot of people aren’t exactly sure what the report tells them. They see the information, but they don’t understand what insights are being given.

That’s what this series of blog posts is about: demystifying the scenario analysis report and making it work for you.

Let’s start by talking about a concept that’s bandied about a lot when talking about conversions: abandonment. In a lot of conversion funnels, abandonment equates to visits that did not “funnel through” to the following step in the scenario. Not so with our scenario analysis reports; we focus, instead, on fall-out.

Here’s a sample scenario analysis report (click on it for a clear image):

At first glance you may think that this scenario has a 73.51% abandonment rate on step one, since that’s the first percentage to appear on the right. On this, you’re right, but it’s not quite what it seems. Here’s what that number tells you: Of the 38,232 visits to the “Product Page View” step in this scenario, 28,264 (or 73.51% of) visits did not on to the next step, but also did not entirely abandon your site.

From there, we get more detail on what happened with those 28,264 visits:

· 619 (2.19% of the 28,264) visits did actually abandon the site entirely. The product page view was the end of their visit, hence the “End of Visit” label.
· However, a full 27,645 (28,264 – 619) did not leave your site. Instead, they went elsewhere on your site. In the case of the first number (27,082, or 95.82% of the 28,264), they went to the Video Recorders page. How do I know this? I hover my mouse over the little blue name, and voila!

So, only 619 people from step one actually abandoned your site altogether; the others got distracted and did something else on your site, so they’re not completely gone yet.

How do I know what they did? Well, if they went elsewhere on your site, this view of the scenario analysis will tell you, and will provide you with truly actionable information. Here, for example, I see that the vast majority of people leaving my scenario on the very first step are looking for video recorders, so why not promote those video recorders on your home page? Or maybe you could set up a bundle: your most popular products with a video recorder at a reduced cost. Upsell!

Now, let’s shift to the Step Transitions view. I do this by clicking the “View Step Transitions” button above the report (again, click for clarity):

This view provides us with completely different information. Instead of telling you whether a visit ended or continued someplace other than the scenario, this process shows you two things:

· The number of visits in which the visitor did not continue directly on to the next step, yet remained within the scenario process as a whole, and
· Where that visit went when it left that step.

So, let’s look at what the numbers on this first step tell us here: 6,733 visits did not convert to the “Cart Add” step. However, they still interacted with the scenario: 6,731 (99.97% of the 6,733) viewed another product page (so, they were still shopping), and 2 (.03%) actually started checkout (which probably means they’d already added something to their cart and decided not to buy what was on the last product page they viewed).

So, we can say that, of the 28,264 visits that did not convert from step one to step two, 6,733 did not leave the scenario entirely. Instead, they either skipped a step or stayed on the same step; they didn’t abandon. That’s almost a quarter of the visits that didn’t convert to step two – and that’s a great opportunity to ensure that, now that they’re in the scenario, they stay there. Note, for example, that 317 visits went back to the “Product Page View” after they started checkout. Is that a result of your efforts to offer them similar items or accessories on your checkout page? Or, if you’re not making such offerings, could you increase that number by doing so?

Of course, these opportunities may seem fairly obvious; after all, we’ve been tracking shopping carts for a long time on the web. But imagine tracking your three-step application/registration process, or your five-step “Give me more information” process, and you can see how this information becomes useful quickly. You might be able to reduce the number of steps and increase conversions, or note where people are getting distracted and provide them, within your scenario, the information they need to stay on track. That’s so much more helpful than just tracking abandonment, isn’t it?

I’ll discuss the left side of this report in an upcoming post – stay tuned.

CATEGORIES: Best Practices Marketing Optimization User Experience

Click here to cancel reply.

12 Comments

Nick Potter
2009-04-29
06:36:06
A good start however hopefully in one of the future posts you'll provide a clear explanation of why the number don't add up.

This is probably the number one question I (and I'm sure many others) get asked with regards to scenarios. Its also one of the more difficult things to explain - even when you know (or think you know) the answer.
Julia Pond
2009-04-29
08:51:03
Great post--downright Socratic!! :-)
John Wood
2009-04-29
23:01:05
Hmmm! Actually I am now even more confused! Please could you clarify one thing for me? The documentation says "This report only shows exit pages if visitors did not return to any of the Scenario step pages during the visit." (https://product.webtrends.com/WRC/8.7/ResourceCenter/rc/library/pdf/aug/Tracking_Conversion_and_Abandonment_through_Scenario_Analysis.pdf - Page 14). So, i read this to mean that if the scenario consists of pages A, B and C and a visitor goes to A, D, A, B and C then we will not see page D to the right hand side of A - that was not the last page of the scenario that they visited in that visit.

To me this does not fit with your statement: "At first glance you may think that this scenario has a 73.51% abandonment rate on step one, since that’s the first percentage to appear on the right. But that’s not so! Actually, here’s what that number tells you: Of the 38,232 visits to the “Product Page View” step in this scenario, 28,264 (or 73.51% of) visits did not continue immediately on to the next step." Actually I think that is exactly what is happening - 73.51% never return to any scenario page after step 1?

Also, when you say "So, only 619 people from step one actually abandoned your site altogether; the others got distracted and did something else on your site, or they skipped a step, or something else." this also doesn't fit the way I read the documentation. If they skip a step then they don't appear in the right hand side at all...

In short either you or I have completely different views about the way it works, so please could you confirm that what you have written is correct and maybe explain how it fits with the documentation?

Thanks
Sandra Elliott
2009-05-04
08:17:17
@Nick - Thanks for the comment; I'll be sure to add in a post dealing with that subject.

@Julia - Thanks!

@John - Rather than just give you my off-the-cuff response and possibly confuse you more, I'm sitting down with our documentation team and talking about this confusion ASAP. As soon as I have more info, I will share here.
Sandra Elliott
2009-05-06
13:54:00
Hi again John. To be sure about things before I responded to your question, I’ve spent a couple of days talking to people in the company, and one of our engineers (Thanks, Xavier!) ran me a couple of tests to ensure that the information I’m giving is correct.

Here are the results: I did make a mistake above, and it’s here (I’m copying here because I’m going to ask that this blog post be edited, so it may not be above any more):

So, only 619 people from step one actually abandoned your site altogether; the others got distracted and did something else on your site, or they skipped a step, or something else.

The bold is where I’ve misled, though it was due to a misunderstanding on my part (which, by the way, has existed for a long time and persisted through a number of conversations with quite a few people about this report!). What you see on the right side of step 1 are only visits that never came back to the scenario – so, though they did not abandon your site, they did abandon the scenario.

The Step Transitions view, in contrast, will point out how visits meandered through the scenario process – we won’t see those visits on the Step Transitions view. This is just one of the reasons that the numbers in this report don’t add up.

So, in short, our documentation is correct – though I didn’t understand, at first, how we differed. And my understanding, which I’ve used both in training and as an analyst myself, was incorrect – for years, apparently – and I’m not the only one! These kinds of realizations are precisely why I wanted to write this series of blog posts; the scenario analysis report is incredibly useful, but hard to decipher.

Thanks for posting the question, John; without it, I may never have unearthed this understanding!
John Wood
2009-05-06
15:22:40
Thank you for that Sandra - you have expressed it much better than I succeeded in doing!
Selwyn de Koo
2009-05-15
08:48:09
Thank you for this post. It sure improved my understanding and it also raised some other questions. I'm looking into the relation between the scenario analysis (A) "Scenario Entry and Exit Pages" and (B) "Step transitions". Here are some observations:

- In the first step we see 38.232 visits. We see a total of 28.264 leaving the scenario, so obviously (38.232 -/- 28.264 =) 9.968 visits stay IN the scenario. Both graph A and B show a total of 10.966 visits proceeding to the next step. How come?

- The step transitions graph B tells us that 6.731 visits did NOT go to the next step, but to another product page. These visits need to reach another scenario step (otherwise they won't be mentioned in graph B?). Is it fair to suggest that these 6.731 visits are (eventually) part of the group of 10.966 visits that proceed to step 2?

- in graph A we have 38.232 visits in the first step, where 28.264 leave the scenario, of which 27.082 visits go to another product page. This implies that not all product pages are part of the 'cart add scenario' (because they are labeled as 'leaving the scenario'). In graph B we have another 6.731 visits to a (different) product page, that IS in fact part of this scenario. Are there different cart add scenario's in this example site?

- you're suggestion that 27.082 visits go to the "video recorder page" might be misleading. As I see it, you're looking at the Page Title. The Page Title of these kind of dynamic pages will be labeled by WebTrends with the last encountered Page Title in the analysis (or is it the first?). Is this assumption correct?

- No mather in what direction I try to add up numbers for a particular step transition, nothing works for me. Is this normal? For example step 2: 11.219 visits, 4.106 go to step 3 and 6.529 leave the scenario. Where are the remaining 584 visits? (Graph B shows that 845 visits go back to step 1)

Please enlighten me :)
Jacques Warren
2009-05-16
06:06:25
Hi Sandra,

As a consultant who's been working with webtrends for seven years, I must agree with Nick that the most annoying thing with Sceanrios is numbers not adding up. It very often defies logic. I don't understand why they always add up in Google Analytics or Omniture, and very rarely in webtrends. Is it because those apps do some kind of hocus pocus tricks and eliminate extra values? I don't know.

One could argue that it's not that important, but you would be quite surprised to see how much detrimental this situation is for those reports. Most people will discard Scenarios as being inexact, or just plain false, when numbers show important discrepancies, because the number situation is too illogical to them.

BTW, shouldn't the semantic debates about what means what be a clear indication that the report needs rework?
Sandra Elliott
2009-05-18
07:44:36
Hi Selwyn! I hear you and feel your pain, and I'm working on a separate blog post that should answer these questions. When I get it done, I'll come back here and answer these specific questions as well.
Sandra Elliott
2009-06-18
08:04:50
Hi again Selwyn! I've finally posted my second post on the scenario analysis report, which I hope helps. Now, as promised, here are the specific responses to the questions that you asked, courtesy of Dash here at Webtrends:

1. While 28,264 visits left the scenario and did not return (the flow to the right), that does not mean that they did not flow down to step 2 first and then return to the first step. In other words, some of the 10,966 Visits that flowed to step 2 returned to step 1 during that same visit and then exited the scenario. You can see this in the step transistion view when you see the flows to the right in the subsequent steps that go back to "Product Page View".

2. The 6,731 number just tells us the number of visits where they refreshed that page or so another page with the same scenario setup immediately following a "Product Page View." They may be part of the group that went directly to step 2 (10,966), but they could have also simply refreshed the "Product Page View" page and then exited the scenario.

3. Nope - this is one cart add scenario here. the "product pages" you see on this step are actually aggregate pages that contain links to a number of products.

4. The data is based upon pathing data so it would be a unique page URL and we would show the most recent title for that set URL. We would not roll pages with the same title together, only those with the same URL.

5. This ties back to point number 1. The 584 Visits you are looking for would be part of the 845 that went back to the "Product Page View." They could have then left from that step which would increment the flow by one to the right of "Product Page View."

I hope this helps!
Megan Douglas
2009-08-19
10:58:28
Hi Sandra,

I have read all three posts now and they were helpful. Just to clarify (again), a user is only represented once in the pages on the right and the left columns? They are on the left once to represent where they entered the scenario and once on the right when they left the scenario and never came back, correct? If they left the scenario but then came back are they in the number that resides between the step and the right column?

Thanks!
Sandra Elliott
2009-08-19
11:33:26
Hi Megan - thanks for reading!

You're right on the clarification, but let me just replace the word "user" with "visit." A single user could come into the conversion funnel through multiple visits, after all.

So yes - a visit will only display once on the left, when it entered the scenario, and once on the right, when it left the scenario and never returned. If they left and came back, they won't show up on the right until they leave for good...but you can see their movement through the steps on the Step Transitions view.

As for the numbers - that's a more tricky answer. They may, at some point, appear on one of the numbers to the right of the funnell arrows, but only if they directly converted from one step to another. Then again, if they jump steps, they won't show up on these numbers. It's one of the major reasons these numbers never add up.