Through this blog, I will continue to speak about the virtues of fractional factorial testing and limitations of full factorial testing.

Multivariate testing of web pages is still growing, so there will be many conflicts on the best testing methodologies, best practices and statistical methods.

Best of luck,
Billy

I am afraid that you have some inaccurate comments and assertions in your reply to my comment above. I will try to address them each of them below:

1)

[Also, I am not sure what exactly you mean by “very large tests” but the strength of fractional factorial design is being able to run large tests in shorter periods of time than full factorial.]

This is a bunch of B.S. – all parametric approaches build a model of the expected conversion rate and then “score” each possible recipe in the test to try to predict which is the best. This predicted-best should then be run in a head-to-head against the original to see the actual amount of improvement. If you are doing a main-effects analysis only, there is absolutely NO SPEED ADVANTAGE at all of fractional versus full factorial data collection. In fact, you get better main effects estimates in the (almost inevitable) presence of variable interactions, and you have the ability to create models with interactions to see if your main effects model is appropriate. I cover all of this in detail in my Taguchi whitepaper at http://SiteTuners.com/downloads.html

2)

[Part of being an expert is being able to draw out what interactions may come up.]

This comment is off-base as well – you can’t “draw out” interactions which are not captured by your fractional factorial model in the first place. If you do not collect data on the interactions (by using full-factorial or at least very dense (as opposed to the commonly used very sparse) fractional factorial sampling, there is simply no data to build the required models.

3)

[Although you may find all the interactions by doing full factorial test designs, it is highly likely that many of those interactions are low influence and are not worth the time or effort to discover.]

This is very misguided. In marketing we want to create synergies among the page elements. In other words we want the variables in the test to interact. So to say that the interactions “are not worth the time or effort to discover” means that you are willing to leave a lot of money on the table. We recently completed a test for RedEnvelope using the Google Website Optimizer. If we had used only a main-effects analysis without interactions the test showed only two out of the eight variables as being significant. When we re-ran the analysis with simple two way interactions, we found that there were four significant main effects, and eight very strong interactions of similar magnitude. The bottom line is that by ignoring the presence of many wickedly strong interactions, you would have not found the best answer.

http://sitetuners.com/site_tuners_re_case_study_detailed.pdf

4)

[The better alternative is for an expert to choose which interactions they believe may occur and to test only for those.]

This is not possible. There are no “experts” capable of understanding the psychology of an anonymous audience visiting your site. Even if you could, there would be inevitable conflicts and contradictions because the needs of each individual are different. Domain subject matter insights in creating the model are only helpful when you are dealing with physical processes like manufacturing (which is where the Taguchi Method originated many decades ago). If you have access to such “experts” for the human brain, I will hire them in a heartbeat.

5)

[In regards to parametric vs nonparametric, 100 conversions per day is a very high amount for a single page. With that amount of traffic, fractional factorial design can create very large tests also.]

100 conversion per day is not a lot – we work with many clients that have many orders of magnitude higher than this as a data rate. Also, as I have already said repeatedly, fractional factorial design for large experiments would have to be very sparse and ignore most variable interactions. If they include even low (second and third) order interactions, the number of parameters in the model would grow geometrically and prevent you from making any kind of reasonable predictions.

6)

[In addition, the information gained about individual factor influences through parametric MV tests is VERY important. The influence measurements I get from parametric testing helps me design future tests and ensures that I don’t throw away ideas that only barely lost to the optimal or think that marginally important factors are very important. Since nonparametric design only tells you what wins and not the actual influence levels, you lose the knowledge of knowing why the optimal is the best.]

This is a tired and incorrect mindset that I encounter often. So-called “learnings” are of very limited value. You basically have a choice: run large scale tests and find the best answer among millions of landing page versions, or run small “toy” tests with inaccurate simple models and pretend that you understand something about why your predicted answer is better. There is no free lunch – you can either devote your data sampling to finding the best answer or to building a model – you can’t have it both ways. As I’ve also said many times, this kind of main effects after-the-fact meaning making will lead you to the wrong answer (see the RedEnvelope case study in #3 above).

I hope this helps,

Tim Ash

Thanks for your post. We obviously disagree about the best method of testing, but I am glad that there is discussion around this.

I am “splitting hairs a bit” when I describe Taguchi as inadequate since it is a form of fractional factorial test design, but I just wanted to make clear that it is not the exact same thing as what we do at Widemile and that the differentiation is both important and relevant to getting the best test results.

However, I have to take point against a few of your statements. Fractional factorial designs can detect interactions, up to the total number of factors in the test. Fractional factorial design does not inherently mean that no interactions are taken into account. As Vladimir, Widemile’s Chief Scientist describes, “practically any level of interactions (up to the total number of factors in the test) can be specified. The fractional factorial designs must be viewed, in fact, as a continuum of the designs, starting from the main effects (no interactions), then 2-factor interactions, and so on, up to the full factorial (interactions of all the factors in the test). The designer then chooses the appropriate level of interactions depending on his/her insights and the time constraints of the test.”

Also, I am not sure what exactly you mean by “very large tests” but the strength of fractional factorial design is being able to run large tests in shorter periods of time than full factorial.

I never assume that interactions don’t exist, however I also think the extreme of assuming all interactions exist in all situations is a mistake also. Part of being an expert is being able to draw out what interactions may come up. Although you may find all the interactions by doing full factorial test designs, it is highly likely that many of those interactions are low influence and are not worth the time or effort to discover. The better alternative is for an expert to choose which interactions they believe may occur and to test only for those.

In regards to parametric vs nonparametric, 100 conversions per day is a very high amount for a single page. With that amount of traffic, fractional factorial design can create very large tests also. In addition, the information gained about individual factor influences through parametric MV tests is VERY important. The influence measurements I get from parametric testing helps me design future tests and ensures that I don’t throw away ideas that only barely lost to the optimal or think that marginally important factors are very important. Since nonparametric design only tells you what wins and not the actual influence levels, you lose the knowledge of knowing why the optimal is the best.

-Billy

I agree with you that the Taguchi Method is the wrong way to go for landing page optimization tests. I am currently working on a whitepaper called “The Truth About The Taguchi Method for Landing Page Optimization”. It will be available as a download here: http://SiteTuners.com/downloads.html

However, I believe that you are splitting hairs a bit. As you mentioned, the Taguchi Method is a subset of fractional factorial design of experiments (DoE). All fractional factorial designs basically suffer from the same constraints as Taguchi. They assume that there are no interactions (or at best can detect very few low level ones). The other big drawback (and this applies to all parametric “model building” approaches), is that the number of parameters in the model grows very quickly as the size of the test grows. So it becomes impossible to run very large tests.

Landing page optimization tuning methods consist of two main activities: how you collect data, and how you analyze it. Your choice on the data collection approach often dictates and restricts the possible kinds of data analysis that you can conduct. Fractional factorial data collection is absolutely the wrong way to go in the presence of strong variable interactions. Variable interactions are very common in online marketing, so to assume that they do not exist is silly and dangerous. It often leads you to the wrong answers).

The proper way to collect data is full factorial. This has two advantages: better estimation of main effects (even if you do not model interactions), and the ability to detect and model important interactions. Google Website Optimizer does full factorial data collection. Even though their reports are currently only main effect reports, at least you can export the raw data and model interactions.

However, both fractional and full factorial data collection is still commonly analyzed using parametric methods. This leaves the problem of scaling up the size of your test. Basically you will hit a wall with any of these methods if you try to run a very large test.

That is why we took a long time to develop our own proprietary TuningEngine technology. It is a non-parametric method that takes variable interactions into account, and also allows you to run much larger tests (1000 to 10,000 times larger on the same data rate). It does have the disadvantage of needing at least 100 transactions per day to operate. But if you have higher data rates, it allows you to get results in one test instead of several back to back parametric multivariate tests.

I wrote a whole chapter in my Landing Page Optimization book ( http://LandingPageOptimizationBook.com ) about various tuning methods. It discusses the trade-offs above in a lot more detail.

The TuningEngine is available as part of our full-service engagements or as a technology license for interactive agencies and companies that run many landing page tests in-house.

What you described as “adjustments” is close to what I am saying. The Taguchi method is a part of fractional factorial design, but is not all encompassing. Marketers that say they are using “the Taguchi method” probably are just doing fractional factorial design and incorrectly attributing it to the Taguchi method.

I just want to clarify that the true Taguchi method is not made for marketing and should not be sought as the answer to answer multivariate tests.

- “There is no possibility of detecting interactions among the control factors”
+ In Roy’s book he covers this topic in many sections, starting in Section 5-5-2 “Interaction Affects”. He also publishes Taguchi’s triangular tables and linear graphs which provide information on how to isolate interaction effects of the control factors in the orthogonal arrays.

- “However the confounding structure for the control factors is the same as that of an array of size N1. This implies that the same resolution can be obtained with much smaller number of runs. ”
+ The full factorial array is N1 * N2 in size, but isn’t that the point of the research done by Taguchi – that by using orthogonal fractional factorial arrays that you can use a fraction of the number of experimental runs to capture the essential information to determine main effects and interaction effects?

I have heard the comment in the past that the Taguchi approach does work in marketing, but there are “adjustments” that need to be made to make it useful in that context. Have you any idea what those “adjustments” might be?

Thank you in advance for your expanded comments in reply, I am looking forward to advancing my understanding of optimization techniques applied to marketing.