What's In That Sentiment Score?

| 1 Comment

Stoplight smileSentiment is the stoplight chart of social media analysis. It offers red and green candy for the boss, and a useful filter for the analyst who's moved beyond the mood ring. Still, sentiment analysis is the surest source of disagreement in social media analysis. Why is that?

The human vs. machine debate has been going on for years, because the software's always been close to the frontier of the science. I started writing about it in 2007; five years later, you can still find companies working closely with university researchers to find better technologies for scoring text. The lag between the lab and the commercial product is virtually zero.

The tradeoff for bringing new technology to market as soon as possible is that it won't be good enough at first. You could read The Innovator's Dilemma to see how that tends to play out. As long as text analytics remains an active area of research, today's products won't be as good as next year's, either.

Look more closely at the tools
The obvious question is, "What's good enough?" But you can run tests and evaluate your options, and I'm not usually a fan of the obvious questions. Instead, let's look at some questions that help you get under the hood of tools as you consider them. There's more to it than the usual discussion points suggest.

  • Who or what scored the data?
    Start with the obvious question: is the content scored by a person or a computer program? If it's human, is it by your user, by a vendor analyst, or crowdsourced? If it's automated, is it the vendor's own system or another company's?

  • How does the automated scoring work?
    If the system provides automated sentiment scoring, how does it work? The engine that does the scoring could be as simple as a word match or as advanced as one of those research projects that just left the lab. Listen for descriptions of machine-learning approaches or systems that parse the structure of individual sentences. For machine-learning-based systems, can users correct scores, and if so, does the system learn from the changes?

  • At what level is sentiment scored and reported?
    Does a sentiment score reflect a document, a sentence, or a statement within a sentence? How are document-level scores determined? How does the system handle, for example, a positive statement about Brand Y in the midst of many negative statements about Brand X? How does it score documents with mixed sentiment (multiple statements with opposing sentiment)?

  • What's the scale?
    How many points are used on the sentiment scale—three, five, 100? If there's a number associated with sentiment, is that an intensity scale or a confidence score?

  • Does the system go beyond sentiment?
    Does the system analyze statements of opinion beyond sentiment? Can it identify emotions, preferences, or intent?
We could probably get well-informed people to debate each of those topics (sounds kind of fun, actually). Remember that this is an area of continuing research and development, and what's not possible today may be common next year. There's a reason I didn't take a position on the best approach in this post.

It's not sufficient simply to check the right boxes, especially with sentiment analysis. You need to stick with the topic for the long explanation from each vendor, if you want to understand what you're looking at. Let them make the case for their preferred approaches, and then you can make an informed choice.

Photo by Blue Funnies.


1 Comment

Great job Nathan, those five bullet points are strategic in selecting sentiment analysis software AND for answering the C-suite questions that inevitably surface about two slides into most sentiment analysis presentations.

I use sentiment analysis for competitive intelligence consulting; it has been extremely useful though sometimes it is much like dancing on a razors edge.

Thanks for your help!
mike

Comments are now closed for this entry.


About Nathan Gilliatt

  • ng.jpg
  • Voracious learner and explorer. Analyst tracking technologies and markets in intelligence, analytics and social media. Studying complexity and futures.
  • Principal, Social Target

Subscribe

  • Subscribe by email