Sentiment Monitoring Services: A Case of “Best Solution Available”

May 8th, 2009
Author: Nazim Uddin

When it comes to sentiment monitoring, you are left with only two options: “do-without” or “do with what’s available”.  But realistically, “doing without” is not an option for companies and organizations in the digital era, especially when the speed and volume of communication can be overwhelming and the risk of ignoring the conversation is enormous.   However, what is available is not that great either.  I have a lot of concerns with currently available services and the claims many of these services make.   I know from my own experience  in doing qualitative research, specifically framing analysis of text which is far beyond what is required for comparatively simple sentiment monitoring of an issue, organization or brand, that content analysis is a messy business.

I  tested Twitrratr, a website which promises that it “can distinguish negative from positive tweets surrounding a brand, product, person or topic.”  The results were mixed.

For example: a search of “Obama” returned this tweet as being negative towards him because of the word “problem”:

“obama does get it. he understands the basic problems tha americans have while mccain is very out of touch”.

The search also returned this tweet as being positive towards Obama simply because it contained the word “good”:

“palin asks a really good quesion: what do all these radicals and terrorists see in obama?”

Twitrratr is only matching up flagged keywords and categorizing the tweets accordingly but quite oblivious as to what those keywords are actually referring in the context of the conversation. Content analysis is extremely difficult and the methodology required can be different from project to project and even evolve within a given project based on the results being generated.  Sentiment/Attitude/Tonality, all virtually analogous, is just one of the dimensions that falls under the content analysis umbrella.  At least the service is free. Companies such as Scout Labs, Sysomos, and Wise Window all offer products that supposedly overcome this, but my experience of the first two has been less than satisfactory.

My main problem is that while these companies promise 80-90% accuracy when compared to human coders, there is no independent verification of these claims.  Add to that pressures to sign up to lengthy contracts, you end up getting the eerie feeling of being at a used car dealership.

Unfortunately, this problem will continue to plague sentiment monitoring for the foreseeable future whether for blogs, micromedia or otherwise. What is needed is advanced  algorithms that can gauge sentiment from a contextual basis. As it is, this nascent technology can be describe as the “best solution available”, but whether it meets any objective standards is another question all together.

Related Posts

Tags: , , , , ,

This entry was posted on Friday, May 8th, 2009 at 5:23 pm and is filed under Public Relations. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • Hi Nazim,

    You are right. There are some folks (like Twitrratr) who are just looking for the presence of positive and negative words, and those methodologies are always going to be disappointing. Scout Labs uses "entity-specific" sentiment detection, so we are indeed parsing the text and doing part-of-speech tagging to make sure that we only label negative posts that are negative ABOUT OBAMA. This methodology has higher accuracy rates, but in the world of social media "high" only means agrees with humans 73-75% of the time. We test our algorithms regularly against human scorers and that's our level of accuracy (we can give all the detail on our methodology if you like). For larger brands, a 75% accuracy level definitely gives you something that is strongly directionally correct. You will for sure see negative spikes on negative buzz days and huge positive spikes on very positive days. And if you recall, Nazim, you can override / change any sentiment value in our system, and that becomes a labeled piece of the data that we can use to improve our algorithms over time (add terms to the dictionary, add linguistic rules, etc.) So you will see our algorithms improve over time as thousands of users across the Scout Labs network just use the app. And of course, if you a have a client for whom getting sentiment exactly right is essential, we can offer you human services that can check new posts daily for you and get them just right, but its people, so its going to cost you (and usually agencies prefer to be the ones offering that service to their clients).

    Social Media language is messy. In fact, our testing shows consistently that three humans only agree that a post is negative (or positive) 85% of the time. So why try? Because for many brands, there is such a high volume, that they need technology to help. And 75% correct on a first pass in real-time for $250 per month is, in most cases, very worth doing. And besides, you can quickly and easily make make it perfect if you like.

    So you should question our methodologies and you should take us for a test run. I assume your comments about the used-car-sales feeling of various vendors were not directed at Scout Labs, as we offer a full-functionality free trial with the ability to cancel any contract at any time with full proration if you ever choose to cancel.

    We'd love to "talk sentiment" with you, so give me a ring, Nazim. jennifer<at>scoutlabs<dot>com. </dot></at>
blog comments powered by Disqus