Since the recent New York Times piece on sentiment analysis, it seems everyone has an opinion on sentiment analysis (how appropriate, yes?). Without actually counting, I'm getting the impression that the overall score is negative, although with the colloquialisms and subtle innuendo, I'm not always sure. :-)
This is a round-up post, so I'm going to start linking to posts I've seen in a minute, but first, we have a problem: a buzzword alignment problem on what to call companies who monitor and analyze social media content. The article uses sentiment analysis to refer to the industry, but sentiment analysis is better understood as just one of the types of analysis used in the field.
This industry has a history of picking up a new label almost every time someone new writes about it. Forrester Research has called it brand monitoring and listening platforms, depending on which year and analyst you ask. I picked social media analysis when I had to choose, but even that is more limited than the state of the art tools and services. I don't have an answer to that one that makes me happy just yet.
Scoring the conversation
Oh, OK, I'll count. Really, how could I resist? Isn't this the obvious way to collect the posts on this topic?
Positive
- Sentiment First, Ask Questions Later, Joseph Fiore (RepuMetrix)
- Text Analytics for Ye of Little Faith, Manya Mayes (SAS)
- Sentiment Analysis, Scott Berkun
- Sentiment analysis for online content: Honest?, Urs Gattiker (CyTRAP Labs)
- La tarte à la crème du sentiment analysis, Guilhem Fouetillou (Linkfluence)
- Sentiment analysis crap in social media, Laurent François (Ogilvy)
- Is Sentiment Analysis Reliable? Andy Beal (Trackur)
- Sentiment mining: new term, new field. A new web? Claude Théoret (Exvisu)
- The sentiment analysis party has plenty of elbow room, Christine Sierra (Lexalytics)
- Sentiment Analysis, Eric Friedman (Union Square Ventures)
This list is an example of document-level sentiment analysis by a human. Anyone want to make the case that it might not be 100% accurate?
You're right, it's an issue we have to face since the beginning of social media monitoring.
The thing is (and that's probably why I was so angry yesterday) that we're now shaping what will be the norms in analytical tools / next gen. opinions surveys, and that we cannot accept to start with wrong basis.
so...more debates to come!
take care, and glad to discover your blog
Sentiment analysis on sentiment analysis? So, meta sentiment analysis? Sounds like a great name for Porcupine Tree's next album.
You know I won't argue about the accuracy of human analysis. I just claim that its about 50% more accurate than anything we've seen. :) And I'm inclined to say that you are 100% correct in saying that it won't be resolved any time soon.
Nathan, great post and excellent approach to consolidating views on an increasingly important topic. Automation (ie: automated sentiment analysis) is only a tool and will always need human intervention to be absolutely accurate. But, the degree of accuracy needed is dependant on the business user's requirements.
Great post and summary of conversations on the topic, and so glad that you directed your visitors to ours.
I agree with you and those who have already chimed-in that this debate won't settle anytime soon. I believe the maturity of this topic can only be made possible with awareness and debate, and I really think we need to continue to raise more questions on the quality and reliability aspects.
I especially liked this commenters points which I'll link share here.
Thanks again Nathan!
Joseph
@RepuTrack
I wrote a case study on a measurement program that is just published by the Institute for PR -- every system needs input and adjustment, but the automated system detailed in the case needed quite a lot of revision and change. The paper ends with recommendations and thoughts about how to choose effectively.
http://www.instituteforpr.org/research_single/measuring_company_a_case_study_and_critique/
Wow, what a response! I guess I need to wade into controversial waters more often. Hmm. Puppies vs. Kitties, maybe? ;-)
Thanks for your comments, all. I'm nodding in agreement as I try to process all the points that you've made (both here and in related Tweets).
At Sysomos (which offer social media monitoring and analytics services), we use social media analytics and text analytics as the key pillars in determining sentiment.
The NYT article was interesting because it put the spotlight on one part of the social media monitoring ecosystem, which suggests the market is maturing as users and potential users try to get a better handle on what exactly is involved.
cheers, Mark
It's great to see this debate finally beginning to get some real visibility amongst the analysis community - it's been a major elephant in the room for too many years now. Earlier this summer, I debated the topic with another UK colleague, Mark Westaby of Spectrum, which, like your original post, Nathan, also generated a fair amount of comment and a significant number of hits from the market research community. It's here, for those interested - the URL gives a hint to its content :) http://www.research-live.com/features/tracking-online-word-of-mouth-the-people-vs-machines-debate/4000156.article
I just hope the debate reaches the eyes and ears of our most important stakeholders - users...
Yes, Mike Daniels and I had a very enjoyable debate on this issue and the conclusion was that there are pros and cons on both sides. Although pro-automation I completely accept it has to be used properly but that is just as true for human analysis. The main point I would make is that those who dismiss automation should review their thinking because (a) it can be used very effectively and (b) human analysis is always assumed to be accurate when that is often far from the case. Indeed, we've conducted tests where our automated analysis delivered far, far more accurate results than a media analysis company, which used human analysts. So much so, in fact, that they revised their analysis completely, in line with our automated results.
We decided to take part in the great sentiment analysis discussion as well. Due to the impossible task of tagging right all the complex comments people make, we believe that it is best to analyse the sentiment levels of the studied brands for example on a monthly basis. One can also compare shares of negative and positive discussions to see if there are any changes in the usual ratio between them.
Then, a more detailed drill-down analysis then enables the marketer to understand what has caused the change in the ratio between positive and negative.
You can find an example sentiment analysis about British Airways and competitors in our latest blog post http://www.whitevector.com/blog.php.
Essi
Whitevector