Debunking Sentiment : NLP or A Turk?

|

Sentiment comes up more and more in the conversation about social media measurement as the demand for insight in this arena increases. What savvy marketer doesn’t want to understand the sentiment of their audience? With the rise of social media, the voice of the consumer has more venues to be heard than ever before. Marketers have had to rely on highly sampled attitudinal response, largely collected through direct survey and other traditional market research tactics, so the promise of broader sentiment measurement is highly appealing.

The problem is that no one has solved the problem of sentiment, which is compounded by the fact that many vendors are claiming to have sentiment (Forrester Wave: Listening platforms). We wanted to provide some clarity around sentiment and explain what we know to be accurate.

At the moment there are two emerging approaches to determining sentiment: humans and natural language processing (NLP). Let’s take a closer look at the two.

Human Power Sentiment
Because automating sentiment analysis is a difficult problem to solve, some vendors are using people to determine sentiment. Many use the service Mechanical Turk, which is an API that connects to people. The name comes from the 18th century chess-playing automaton “The Turk”. An example, if you wanted to know if a comment was positive, negative, or neutral (establish sentiment) you would send that comment to a Turker who gets paid per request to decide the comment’s sentiment. To ensure accuracy, three Turkers are asked to assess the comment.

The inherent problem here is that the process doesn’t scale. Large volumes of data can only be sampled. Analysis also isn’t real time and delays can be in terms of days. To scale, more bodies need to be thrown into the analysis. Since three Turkers are paid per request, the cost becomes prohibitive to deal with high volume.

Algorithmic Sentiment
Natural language processing is the algorithmic approach to text analysis that doesn’t have the cost and scaling problems of human powered sentiment. Companies like Google are working on this problem, but they still haven’t cracked the A.I. nut to accurately map text to the author’s sentiment. Currently NLP technologies fail to accurately decipher the sarcasm, slang, and irony that we so frequently push out in our blogs, tweets, and pithy status updates.

Don’t get us wrong, we love the idea of sentiment analysis! We’re eager to hear more about progress made in this area, let us know when you hear about it. We look forward to this maturing into something that will help all of us marketers better understand and communicate with our customers. We’re following sentiment technologies closely because we want to incorporate the technology into our solutions. For now, we’re not employing automated sentiment because the human powered approach doesn’t scale, and the algorithmic approaches are as unreliable as voice recognition. If companies are promising you features that depend on automated sentiment, be sure to ask them probing questions about their approach so you don’t get sold on vaporware.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We are always working hard to improve your experience with WebTrends. Let us know how we’re doing – provide feedback from within the products, comment here on the blog, user forums, Twitter or contact any of us directly.

  • http://zebrabites.com Katie Harris

    Hi Eric

    Nice to see I’m not alone in my thinking!

    http://zebrabites.com/2009/01/18/fools-gold/

    Katie

  • Tony Jones

    Great post and and I totally agree many folks can lose sight of the inaccuracy of NLP but it is so true that it is the only way to scale this analysis. For a company like mine there really is no other option. I do believe there is hope though many folks are working on some very cool stuff. The biggest hurdle I see is scaling it worldwide. Getting this to work in English is hard enough but try Chinese!

    Bottom line I’d say NLP is the way we need to move but the technology is still in the crawling phase right now and it’s going to take sometime to get to running.

  • http://www.socialmetrix.com Sebastian Rosefeld

    Hello everyone. At SocialMetrix we tested different techniques and, as you properly pointed, the main issue with human power is not just cost, but scalability. In addition to that, we found that even a “human moderator” often fail to recognize some cases of irony or sarcasm on comments posted by other humans.

    Our current approach is to work with a mixed solution. NLP algorithms, allow us great scalability, but in order to obtain good levels of accurancy in sentiment detection we work on a particular set-up for the language processing for each client. We use humans to “train” the algorithm into the specifics of each industry, sector, and country slangs (our tool works in English, Portuguese and Spanish, so mexican spanish is quite different from chilean spanish and so on…) and we are getting very acceptable levels of accuracy in sentiment detection (more than 80%).

    So, each client have a set-up period were a human moderator manually determine the sentiment of certain comments, and use that as the starting point for the tool, then periodically check the level of accuracy by taking a portion of the NLP processed comments and verify them manually. With the ones that were incorrect, we re-train the NLP algorithm for that particular client.

    We also agree with Tony’s last comment, NLP is certainly the way we need to go and even though technology is still limited in some ways, there are several companies and academic institutions making huge progress in this field. At the same time we already see marketers around the world embracing these technologies because most of them already take advantage of the insights obtained by its use.

    Having said that we expect more exciting industry improvements in the near future.

  • http://www.learningnlp.org Mike Hill

    This was an interesting read… I’ve never heard of “sentiment” being used in this way.

    From the things you have read about how long do you forcast something like this being available or better yet more reliable?

    It will be interesting to see all the powerful tools that come out once people figure out A.I. Google will probably figure it out sooner or later… LOL