Scenario Abuse, Improve Your Sequence Reporting

June 22nd, 2011

Topics: Digital Marketing

Scenario reports are useful to measure sequential processes.  If you have a check-out process that requires four steps, you can see the step with the highest conversion and dropout rates.  It is a very useful report for determining interface and process issues that impact moving customers from interest to purchase.

Scenario reports are designed to measure process steps.

Straight line sequence

Sequential Process

It lets you find that requiring the middle name of their fourth grade teacher in step two is causing some drop out.  Who would have guessed?!

However, many sites have processes that look more like:

Complex process

More Common Process

Multiple paths with dynamic pages, optional side paths, different number of primary steps depending on product, visitor segment, day of the week, phase of the moon, …  Admit it, you have (at least) one of these.

It’s enough to bring a scenario report to tears.  Don’t get me started on the people trying to read the Path reports on this type of process and expecting useable information on a customer segment.  What can you do?

Look at just a single sub-group of your entire data set. In other words, segment. Segment to find usable, actionable information.  Data for a single segment is clearer than displaying every possible variation at once.  Segmenting not only makes the data easier to understand but better targets the customers you are interested in.

One technique is use a scenario report as a dimension in a custom report.  So what, isn’t it the same data?  Well, you can put extra dimensions on the reports (campaign id, entry page, traffic source, etc.) for segmentation inside the report.  You can also add report filters; for example, filter for only visits that include the “Step 1 Start” to see what visits from that entry point are doing.

Let’s look at an example.  Here is an unfiltered Scenario Report showing the visits for a process similar to the second picture above:

Step Visits Continuing to next step Visits In Visits Out Step Conversion
Step 1 1,437 236   1274  
Step 2 2,446 225 2214 2177 170%
Step 3 254 151 31 79 10%
Step 4 361   209   142%

If we just compared Step 1 visits to Step 4 we would see a total conversion rate of about 8.5%.  However, our steps show that we had an extra 2214 visitors come in from a different starting point at the second step.  The step conversion rates are useless because of the new visitors at every step (some of which are visits returning from side paths).  So how much did the 1437 visits on Step 1 really contribute to the final success rate?  What is the drop off point for these visits?  We can’t tell because they are getting lost in the crowd.

Let’s take the report above and use it in a report filtered for only visits that include “Step 1”.  We have:

Step Visits Continuing to next step Step Conversion
Step 1 1,437 258  
Step 2 258 8 18%
Step 3 8 5 3%
Step 4 5   63%

Only 0.3% of these visitors made it from Step 1 to Step 4.  We get a reasonable conversion at Step 1-2 but few are making it to Step 3.  Now we know the conversion and drop off points for this segment.  Further research shows that most of this segment didn’t make it past the third grade so had no fourth grade teacher to remember. Ha!

Another useful technique is to lay the report on its side and use measurement landmarks.  In our example, is knowing the visitor left Step 3 to look at your company info and then returned important?  Maybe not, but the scenario report considers it leaving the sequence.  You get an exit from the right and an entry from the left side of the report.  By using measurement landmarks you can see visit flow without a scenario report.

Create a measure which counts visits to a specific step.  Create measures for other steps.  These are the landmarks, the checkpoints that the visitor passes on the way, sub-conversion points.  Create a custom report using a dimension of your choice with the landmark measures to see how far different dimension segments make it in the process.  Want to see if visits from organic search go farther than visits from social sites?  Go ahead!

Webtrends Analytics is very flexible and powerful.  If you want a report on the second process above, it will give it to you. Careful, though: Sometimes more data is not better data. Sometimes, just because you found a new and interesting fact doesn’t mean you need to share it with the nearest executive. But part of the power is letting you segment in ways that provide easy-to-understand numbers to see how visitors are using your site.  Numbers that let you take actions to improve your marketing.

  • Peter Olsson

    Hi Curtis – What type of filter did you use to only include visits that touched step 1? Thanks!

  • John Mitchell

    I have a question on this if I may.

    I have a scenario where a branch in the scenario occurs at step 3 around the payment option. Credit card, invoice, phone call. depending on what the user selects changes the number of following pages.

    So the scenario looks like

    Name —> Address —> Invoice —> Complete
    Name —> Address —> Call Me
    Name —> Address —> Credit card —> Payment details —> Complete 

    Should the whole scenario including the branches be called the same thing,and then use new sequential numbers for the various payment typesso name and address pass a scenario name of Order and steps 1 & 2then invoice uses steps 3&4Call me uses step 5Credit card uses 6, 7 & 8The scenario report will be garbage but I create a custom report to filter on the final steps to get a better breakdown of various processes.

  • http://www.webtrends.com Curtis Smith

    John,

    I didn’t cover the different possibilities clear enough.

    Correct, If they leave a step and return to that same step before continuing the Visit based report shows the flow to the next step. But if they skip a step or return to a different point in the sequence (common in dynamic sequences) the visit leaves from the right and returns from the left side of the reports. The information in the report is correct, just hard to interpret.

    The “abuse” part is trying to use a scenario report in a dynamic process. There are just too many variables for the report to be clear. The basic information is to simplify the data by segmenting the information so you get down to a set that you can use. Frankly, I’m a bigger fan of tracking micro-conversion points and a final conversion than trying to force scenario reports in to where they don’t belong. But that’s a whole blog (if not longer) by itself.

  • http://Website Mark

    Thanks Curtis that was really helpful, I was able to get the visits for each step without the left and right side visits – Scenario Analysis as dimension and include filter for only visits from Step 1. However I am a bit perplexed with how to managed to configure a measure such as “Continuing to next step”. It would be great if you can throw some light on that.

    • Curtis Smith

      Continuing to next step is the value shown in the downward arrow in a scenario report. In the scenario report it shows how many visitors at this step went to the next. What it doesn’t show is if this group came from the step before or if a significant part entered from the right. That is where visit filtering clears up the report making sure we only include visitors who start at the beginning (or the segment of your choice).

  • http://Website Rahul

    Hi Curtis,

    I am trying to create a custom report based on a scenario, similar to the second table you have above in your blog post. Not quite able to get around the “use it in a report filtered for only visits that include Step 1″. What I would like is a simple custom report based on the scenario parameters that excludes all those lateral visits from each of the steps. Only the sequential visits from each step to the next one and their conversions.

    • Curtis Smith

      Rahul,

      Try the “View Step Transitions” button on the scenario report. That might give you the view you want.

  • http://www.matraxis.co.uk John Wood

    Useful technique, thanks Curtis! However, I have one question. You state:

    “…is knowing the visitor left Step 3 to look at your company info and then returned important? Maybe not, but the scenario report considers it leaving the sequence. You get an exit from the right and an entry from the left side of the report.”

    However, I thought Webtrends scenario reports are Visit based and ignore pages which are not part of the scenario unless the Visitor does not re-enter during the same session, so there will be no exit and re-entry to the scenario?

  • http://www.klm.com Frans

    We have created 3 new funnels with different entry steps. It works really good! Thanks Curtis!

  • http://www.webtrends.com Curtis Smith

    A standard scenario report shows the visits that enter (from the left) and leave (from the right) the scenario sequence. In a basic linear path this is very useful information. However, the more side entry/exit paths and loops you get the more confusing it becomes to interpret. Filtering on a scenario as a dimension lets you focus on the segment.

    Filters remove data going into a report, reducing the data to the segment you want. But applying a filter that segments the main path would also affect what URLs show on the sides. Often the interaction can be fairly complex rendering the side data unusable. You would still have the side paths but they would be missing data that was filtered out.

    Think of the second process above. Now filter out 50% of the arrows, or worse remove 50% of the visits from each arrow. Which 50% can be hard to figure out even if you realize they are missing!

  • http://Website Ulrik

    I wish you guys would allow filters ón scenario steps and scenario report. It would solve a large portion of these issues :-)