So many times, I've heard that (pick one) only humans can identify sentiment in text, or the software is very good now. I don't call it a debate, because I don't see the sides talking. Setting aside the question of software maturity, what is it we want the computer to do, and how far along are the tools?
When I need to explain the concept of text analytics for the first time, I usually summarize it as "teaching the computer to read," which—no surprise—isn't an original phrase. It goes back at least to the early '90s. But reading for meaning is still more of a goal than the current reality. Today, the tools are somewhere on a continuum, which I think looks something like this:
- Content discovery
The challenge is social media analysis starts with the attempt to "read the Internet" (all of it). The simple approach to selecting the part we care about is to use a keyword match or Boolean query, but probabilistic and semantic approaches are out there.
Success criteria: recall (completeness) and speed
- Filtering for relevance
Source data is cleaned, removing spam, duplicates, and off-topic items. Company names that are also words make relevance filtering important and a point of differentiation for some vendors.
Success criteria: precision (% relevant content)
- Extracting concepts
Natural-language processing (NLP) yields a list of key words and phrases, which generates those brand-association and leading topics reports. Also very useful for grouping items for end-users of the system. More advanced approaches group related topics and synonyms.
Success criteria: usefulness (low noise), accuracy
- Extracting facts
NLP identifies factual statements based on grammatical analysis of content. This is helpful for understanding the reason behind sentiment and potentially huge for competitive intelligence and finance applications.
Success criteria: accuracy, useful summarization
- Determining opinion
If you want to start a good argument, bring up sentiment (although I never seem to find opposing viewpoints on human vs. machine analysis in the same place). It's popular as a PR metric and useful as a filter, so it's one of the usual metrics in social media tools. Some vendors go beyond tonality (positive/negative) and provide an analysis of the emotional content of the text.
Success criteria: accuracy, consistency, depth
- Reading for meaning
What we really want: the computer reads mountains of text and, after accounting for source reliability and influence, delivers an accurate summary and metrics, cross-references sources, and synthesizes an accurate view of the situation.
Success criteria: not holding my breath.
I hear claims of 90% and better accuracy on sentiment, but a test would require a comparison with imperfect, human coding. In any case, one text analytics provider I talked with said that specific accuracy rates are not a client concern. Their focus is on the value of the resulting analysis, and good enough is good enough.
I'm not a scientist, and somewhere out there is a computational linguist whose left eye is twitching over some mistake I've made. Comments are open—go for it.