Multiple Views and Variations on Multivariate

|

I’ve been in the software industry for over 20 years and in that time I have seen plenty of term confusion and misuse.  But it’s hard for me to remember a time when I saw so much of it from a single space.  Let’s clear the air on a couple of definitions (for now), beginning with the liberal use of the term multivariate.

Regardless of the terms, there are really two fundamental forms of testing and optimization in our space: split tests, as I refer to them, and multivariate.  Split tests, a.k.a. A/B tests or A/B/n tests, apply when you have a small number of variables and values.  For instance a single variable, such as an image, may have two values, Image A and Image B.  This most basic example is where the term A/B test comes from.  Add in Image C, Image D and Image E, and you can see why the A/B/n shorthand is used.  In these instances the number of values is low enough to permit a large number of trials to be performed on each, providing us with a clear statistical winner with a reasonably sized population of trials.

Now let’s throw in another variable, such as a text block.  You might have two versions of the text to go along with two versions of the image, making four combinations.  This is where the confusion enters in.  Some vendors have gotten into an annoying habit of calling this multivariate.  It’s not.  You can use the exact same split test approach as before – it just so happens there are two variables involved.  I like to call this multi-variable split tests, but frankly I don’t care what we call it, so long as it’s not confused with multivariate.

So what is multivariate optimization?  It’s a form of statistical variance analysis.  The Taguchi methods (named after its inventor Genichi Taguchi) is one form of multivariate statistics reportedly used in several behavioral targeting, testing and ad optimization solutions today.  Where multivariate statistics come into play is when the number of variable/value combinations is so large that it prohibits more than a small number of trials being run against a single combination of values (i.e. it would be too time consuming or too costly).  Multivariate statistics permit us to infer, statistically, the singular values and/or combinations that lead to the desired outcome (e.g. conversions) based on a relatively small population of trials.

Now that we have that cleared up, let’s move onto the term “portfolio” as it applies to PPC campaigns. Back in the ‘70’s and 80’s, Harry Markowitz published his works on Modern Portfolio Theory (MPT).  His work earned him the Nobel Memorial Prize for Economic Sciences in 1990. Essentially MPT provides the body of mathematics used to create diversified investment strategies, and in so doing providing the greatest possible return within a given risk tolerance (notwithstanding global economic calamities, that is).  Or conversely, the lowest amount of risk for a desired return.  It’s all on Wikipedia, so you can read all about it there.

More recently some really smart people from Efficient Frontier, Inceptor (RIP) and WebTrends applied some of the principles of MPT towards the problem of optimizing large-scale PPC campaigns.  In fact Efficient Frontier derives its name from one of the main concepts of MPT.  It is also how the term “portfolio” came to be used in the context of search marketing.  Today I’m only aware of three vendors that use MPT-style mathematics in their PPC optimization solutions: Efficient Frontier, eSearchVision (I think) and WebTrends.  If there are others, please comment below and let me know.

So now enter the confusion.  Several vendors advertise “portfolio-based bid management” capabilities.  What is meant by this is that you can apply a bid rule against a group of keywords.  The term portfolio, in this instance, is used as an English synonym to group or collection.  Grammatically accurate?  Yes.  Intentionally designed to mislead?  Absolutely.

Of course vendors get away with this because the detailed understanding of these technologies is locked in the heads of a relatively small number of people.  But hopefully they are influential people who have a belief in transparency and truth in advertising.  So if you’re one of those people, please help educate the market, starting with me.  If you see errors in fact or have differences in opinion from my comments or in WebTrends’ messages, let’s hear it.

  • http://www.match.com Jim McDonald

    Interesting. So – my understanding of MPT (granted I drank a lot during grad school and didn’t sleep much with a newborn) was that it was designed to mitigate risk (with stocks/investments) through diversification. I understand how risk can drive a $100 stock to $1 in a short period of time (and how diversifying helps); I’m unclear how risk can affect a $1 bid price, or the return thereof, in a similar fashion. If you bid $1 CPC, there is no (Wall Street) risk – you know what the return will be, and typically it has small variance over time. E.g. – if I spend $100 on the term “dating,” I know I will get X amount in revenue and X changes very little.

    Barry – Can you provide insight into how WebTrends or other vendors that utilize MPT when risk really isn’t an inherent component of search (in the same way as on Wall Street); after all – shouldn’t everyone spend based on marginal cost of acquisition and maximum ROI for any derivative of any given search term/phrase?

  • http://gotanalytics.blogspot.com Chris Grant

    This is a great post and first comment. That’s all I have to say. Continue please!

    Oh. Wait. I do have something else to say. (surprise) I’m glad that some terms are getting defined; it’s overdue.

    And I agree with Jim that the term “risk” needs more definition, or perhaps a substitute word, in the SEM context.

  • http://blog.jimnovo.com/ Jim Novo

    Please let’s not forget the “null” format of A/B/n, which is rarely used in Web Analytics but has important implications for any customer contact testing.

    A lot of WA testing on web site pages pits new ideas against the existing treatment as “control”. This is fine and proper. But when you get into outbound customer contact, and especially with e-mail, the more important idea is “null control” – doing nothing – as opposed testing against the existing something.

    Just try it some time – take a random sample of 10% out of the e-mail drop. If people receiving the e-mail generate 5 cents per e-mail, and people who don’t generate 2 cents per e-mail, what is the real effect / profit of your campaign?

    Similar, if you are into the “abandoned shopping cart recapture” e-mail thing, take out a random sample and send them nothing. If 30% of these folks come back and place an order anyway, and you are offering discounts to get the others (who received e-mail) to come back, then what is the real effect and profitability of your recapture program?

  • http://www.webtrends.com Barry Parshall

    Hey Jim,

    In MPT risk is the standard deviation on the expected return of the asset. The same applies to bidding, where the risk on a given keyword is equated to the variance of the return (e.g. conversion rate). While this may not seem like risk, in the true Wall Street sense, it is risk. If you bid $1 on a keyword that gets you 100 clicks, that’s $100 you’re “risking”. If it does not return the ROAS you were expecting, it constitutes a loss, relative to the return that the same $100 could have gotten you on other keywords (and considering we all have finite budgets).

    For head terms like “dating”, in your case, the risk may not be perceived — you have so much data on that term that the return variance is negligible. But for mid and tail terms the lack of data imparts a higher variance. Make sense?

    For all the attention portfolio theory gets in PPC advertising, the hard part of the problem is inferring the return and risk of a given keyword, especially with sparse data. This is where WebTrends Ad Director really shines, with its application of hierarchical Bayesian inferencing, as opposed to the kinder-math used in bid tools, and is thus one of many reasons Ad Director will significantly out-perform a bid management solution.

    - Barry