Recently in Measurement Category

Why is it that so many people talk about the effects of social media on reputation, but so few mention the more interesting models for measuring reputation? Instead, we argue over how to read the sentiment mood ring, or which media-oriented measurement tracks reputation. In most cases, I don't think we're measuring reputation at all. Instead, we're measuring media coverage.

Media analysis reports on published statements. In the recent past, it focused on media created by professionals, but even as it includes media created by everyone else, it's still mostly about reporting aggregate data based on coverage. The usual metrics—volume, sentiment, topics, and voices—reflect that media-centric view, which is now adapting to summarize consumer's opinions in online media. It's good data for some applications, but it's the shallow end of the pool for understanding reputation.

Wading into the deeper water, we find some companies that take a more nuanced view of reputation. These models start with survey research and are typically calibrated to focus on the relevant attributes for a specific company or industry.

• Reputation Institute: RepTrak
  Measures 23 attributes of 7 dimensions: products/services, innovation, governance, workplace, citizenship, leadership and performance.

• Harris Interactive: Harris Reputation Quotient
  Measures 20 attributes of 6 dimensions: emotional appeal, products & services, social responsibility, vision & leadership, workplace environment, and financial performance. Harris recently released its 2009 report (PDF).

• APCO: Reputation Insight (PDF)
  Multi-factor models customized for each client.

• Millward Brown: BrandZ (more)
  Evaluates the financial return attributable to the company's brands, based on an analysis of financial data and consumer research.

Is this stuff not accepted? Is it too advanced? Maybe too confidential? Or is it just above the social media paygrade? The audience in the room where I first learned about this was rather senior.

Hat tip to Leslie Gaines-Ross for pointing out some research I hadn't seen.

Sentiment analysis is generating blog headlines again. After reading about the non-response bias of automated sentiment analysis, and that it has no place in social media monitoring, I decided to run a sentiment analysis on sentiment analysis (apparently I like that phrase). I have an account on Biz360 Community from my current project (look for it next week), so I tossed it a quick query and found that the recent buzz was mostly… positive? Hmm.

Here's the thing. Sentiment is not the golden metric. Virtually every social media analysis platform can show you a pie chart on sentiment. At best, it's a first glance—which way is up. Unless you go deeper into the data, all you're looking at is a mood ring.

At the very least, you need to compare sentiment across brands and over time. Yay, it went up! Aw, it went down.

Oh, look, a mood ring. Maybe there's a secret decoder ring somewhere around here that we can wear next to it.

Set aside the methodology question
The automated sentiment debate continues, but I want to focus on what to do with sentiment data once you have it. On scoring methodology, remember that it's not a simple question of human vs. computer (though this Attensity post explains more of the automation than most people have probably seen). Most of the social media analysis (SMA) platforms I've just reviewed allow users to edit sentiment scores, so when you find a post with the wrong sentiment score, you change it. About half of the automated sentiment processors learn from users' changes, too.

But today's topic is what to do with the sentiment data you have.

Trends, segments, and causes
Sentiment, by itself, is a mood ring—a happiness indicator. It's nice to see the happy color, but there's not much information there. If you dig into the sentiment data, though, it starts to contribute to useful analysis.

Take the trend chart. Direction is interesting, but what about slope? Sudden changes are especially interesting. Any spike—not just in sentiment, but in volume or anything else—is the chart's way of saying "look over here." A spike on a chart is a big ol' why, waiting to be asked.

Sentiment really gets interesting when you combine it with other measurements. Most SMA platforms use sentiment scores as a filter for segmenting the data. What are the prominent and emerging topics within negative-sentiment content (and again with positive)? How does sentiment compare within a topic, across different media types or specific sources? Is a topic emerging from a source that writes negatively about you, or is it a friendly source?

Crafting queries and combining filters could be a whole series of posts, or maybe a book. That's why insight isn't automated: what you do next depends on what you find. If you're looking at the mood ring and wondering what it means, you haven't even started.

Join me at the Sentiment Analysis Symposium (New York, 13 April), where I'll talk about how to make an informed purchase decision in social media analysis.

Measuring Social Media must be the new black. Everybody's doing it—the in-crowd is, at least, and the rest are starting to realize they're missing something. Just look at the agencies who've suggested their own take on the little black dress—that is, on how marketers should measure social media.

• Conversation Impact, Ogilvy PR

  Ogilvy proposes a framework with three sets of metrics that correlate to the traditional marketing funnel: Reach and positioning, based on a combination of web analytics, media analysis, and search visibility; Preference, based on media analysis and traditional research; and Action, based on measurable client objectives (such as sales conversions).

• Social Influence Measurement, Razorfish (with TNS Cymfony and Keller Fay Group)

  The SIM score, as introducing in the Fluent 2009 report, compares sentiment for a company to sentiment for its industry. The report also mentions share of voice and weighting for influence, although the formulas for the metric do not.

• Digital Footprint Index, Zócalo Group (with DePaul University)

  Evaluate a brand's online presence in three dimensions: Height, which represents the total volume of brand mentions; Width, based on consumer engagement with online content; and Depth, based on message saturation and sentiment.

It was my understanding that there would be no math.
—Chevy Chase, as Gerald Ford

Razorfish's SIM score is all about perception. How does the client look compared to its competitors and industry? Despite the "influence" in its title, this score is entirely about sentiment, with none of the usual indicators of influence. As a single metric, the SIM should be compared with the Net Promoter Score or MotiveQuest's Online Promoter Score, but I'll be honest here. I'm having trouble figuring out the significance of this ratio of ratios. I tried a few scenarios to get a sense for how the numbers change and got some strange results: divide by zero errors, negative scores... The intermediate Net Sentiment metric is the more meaningful number here.

Zócalo's DFI addresses PR effectiveness, as telegraphed by the "earned media" headline in the announcement. The focus on presence, engagement and sentiment pick up on important aspects of social media, but without more detail on the math behind the overall index value, this seems like another framework rather than a metric.

What are we measuring, exactly?
I'm not the first to say it: the golden metric that will answer every question does not exist. To be fair, the authors of these examples don't claim to have the ultimate answer, anyway. Social media initiatives can support diverse objectives, and so the metrics used to evaluate those initiatives or to answer questions will be equally diverse. But it is nice that we have people sharing their efforts to find appropriate metrics for some common objectives and questions. Thank you, and keep it up.

While working through the math, I was reminded of an old trick from undergrad science classes: if you're losing track of your formula, do the math with the units included. If the resulting units don't make sense (comments^2, for example), the value won't, either.

Photo by alist.

I'm working on a theme that's all about expanding our idea of listening—it's so much more than defensive monitoring, but we need to get beyond first steps. After the last post, Sam Flemming commented on the importance of distinct terms for communicating outside of the bubble, and he's right. After we expand the concept of listening, we need to break it into manageable pieces. Fortunately, the pieces will look familiar.

As a set of activities, listening breaks down into these five modes:

• Searching
  Search is so familiar that we don't always think about it, but look at the advice on getting started in social media. That first step: find out what people are saying, where they meet—you know, the 5 Ws—when you do that as a snapshot, that's search. Don't neglect the value of familiar methods.

• Monitoring
  The usual starting point for a discussion of listening. Through automated methods (typically a dashboard or RSS reader), find and read new posts, comments, tweets, etc. that are relevant to your business. Focus on individual items for action.

• Alerting
  Similar to monitoring, but the system notifies you through email, instant messaging or text when a new item is discovered. Alerts can also be based on measurement thresholds, such as a sudden increase in negative commentary. No requirement to revisit the platform to receive alerts.

• Measuring
  Add a quantitative element to monitoring. Whatever your choice of metrics or measurement silo, measurement is about aggregation and numbers. For the purposes of this list, I use measurement to refer to the generation of regularly updated metrics.

• Mining
  Add a quantitative element to search, and you have data mining, which looks for meaningful patterns in archival data. Although it has a lot in common with measurement (as used above), I'm seeing different practices and benefits that justify separating the two.

For example, if you're doing routine measurement—you're looking at meaningful metrics on a regular basis—is there an opportunity to find different value by taking a mining approach, looking for insight in a snapshot of historical data? A slim distinction, but the point is to step back, walk around a bit, and look at the data from another angle.

Actually, lots of other angles, but more on that later.

Photo by bdu

Everyone says that listening is central to social media success, but over time, we've fallen into a too-narrow interpretation of the metaphor. Think about it: if listening means monitoring, then we have too many words. Fortunately, they don't need to mean the same thing. We just need to expand the way we think about listening.

Here's the definition of listening implied by many posts and presentations:

Defensive keyword monitoring of social media for customer problems and complaints that need a communications or customer service response.

How can we expand the definition of listening?

• From a defensive posture to developing valuable market intelligence.

• From keyword monitoring to applying all of the technologies available to discover and analyze relevant online content and activity.

• From monitoring to metrics, mining, and interpretation. It's a metaphor, so there's no reason to be stuck with the word's literal meaning.

• From social media to all media and customer communications.

• From a focus on problems and complaints to an interest in all relevant conversations.

• From PR, marketing, and customer service to anywhere the information has value to the business.

• By collaborating across measurement silos to find the right methodology for the task.
  

One thing about sentiment analysis: it really stirs up the opinions. Apparently, it's good for attention, too, because yesterday's post has gotten a lot of it. So what is it about automated analysis that's so controversial, and what can human analysts do to offset the advantages of automation?

Automation offers four basic benefits:

• Scale
  Keeping up with all of the relevant conversations as volume grows.

• Speed
  Processing new items sooner; computers "read" faster than humans read.

• Consistency
  Software doesn't get less accurate with fatigue or mood, and it doesn't consider contextual knowledge if it's not supposed to. It just follows instructions, over and over.

• Availability
  Automated systems don't sleep, so they won't be the limiting factor in determing your 24/7/365 operations plan.

Competing with automated systems
Scale and speed are related, and the hands-on approach is simple: add more people to the process. Speed (latency) will still be limited by the ability of your analysts to read quickly. You won't compete in the sub-second latency market, but you can get ahead of the daily-update crowd.

Consistency in human scoring comes down to training and process. I won't pretend to teach the media analysis pros how to do that job, but it's going to be more formal than the eyeball ratings I gave out yesterday.

Availability is an interesting challenge, but it's not the first time companies have addressed the issue. If you're going to work through nights and weekends, your choices are to create some undesirable jobs at home or switch to follow-the-sun operations overseas.

The rise of offshore outsourcers
Combine the need for a lot of people (analysts) with the desirability of around-the-clock operations, and a lot of people will reach the same conclusion. From the beginning of the social media analysis business, some of the better-known vendors have had development and analyst groups in India. Now, I'm starting to hear from companies that are offering offshore human analyst services as a specialized service.

The interesting bit is that they're unbundling the content coding, so clients (or vendors) can add human-powered sentiment analysis to any platform that provides user coding or tagging.

This won't be an easy group to track. It's largely a traditional outsourcing approach, and any company that provides full-service social media analysis using human analysts could unbundle the coding piece, too. But if clients end up selecting human analysis over the automated version, expect more offshoring of the manual effort.

Since the recent New York Times piece on sentiment analysis, it seems everyone has an opinion on sentiment analysis (how appropriate, yes?). Without actually counting, I'm getting the impression that the overall score is negative, although with the colloquialisms and subtle innuendo, I'm not always sure. :-)

This is a round-up post, so I'm going to start linking to posts I've seen in a minute, but first, we have a problem: a buzzword alignment problem on what to call companies who monitor and analyze social media content. The article uses sentiment analysis to refer to the industry, but sentiment analysis is better understood as just one of the types of analysis used in the field.

This industry has a history of picking up a new label almost every time someone new writes about it. Forrester Research has called it brand monitoring and listening platforms, depending on which year and analyst you ask. I picked social media analysis when I had to choose, but even that is more limited than the state of the art tools and services. I don't have an answer to that one that makes me happy just yet.

Scoring the conversation
Oh, OK, I'll count. Really, how could I resist? Isn't this the obvious way to collect the posts on this topic?

Positive

• Sentiment First, Ask Questions Later, Joseph Fiore (RepuMetrix)
  
• Text Analytics for Ye of Little Faith, Manya Mayes (SAS)

• Sentiment Analysis, Scott Berkun
  
• Sentiment analysis for online content: Honest?, Urs Gattiker (CyTRAP Labs)
  
• La tarte à la crème du sentiment analysis, Guilhem Fouetillou (Linkfluence)
  
• Sentiment analysis crap in social media, Laurent François (Ogilvy)
  
• Is Sentiment Analysis Reliable? Andy Beal (Trackur)

• Sentiment mining: new term, new field. A new web? Claude Théoret (Exvisu)
  
• The sentiment analysis party has plenty of elbow room, Christine Sierra (Lexalytics)
  
• Sentiment Analysis, Eric Friedman (Union Square Ventures)
  

This list is an example of document-level sentiment analysis by a human. Anyone want to make the case that it might not be 100% accurate?

I've said that opposing viewpoints over human vs. computer analysis of social media content don't constitute a debate, because I've never heard both sides at the same time and place. Now, thanks to an email exchange between Mike Daniels (Report International) and Mark Westaby (Spectrum) for Research magazine, I have to stop using that little observation. It's now—finally—a debate.

Tracking online word-of-mouth: The people vs machines debate

After an exchange of the usual points and counterpoints (speed, accuracy, sarcasm, synonyms...), the discussion really gets going in the comments. Mark makes a point that may summarize why I find this stuff interesting:

Automated analysis should not be viewed as a replacement for human analysis. Rather, it is a different method that is opening up entirely new and tremendously exciting ways of analysing data.

The usual debate: a closed question
If you're looking at it from a media analysis perspective, this question comes down to quantity and quality. How much media can you analyze in a way that you will trust? The new technologies will let you analyze more media sources faster, if you accept the results. In a world bursting with new publishers, that could be a good thing, and that's where we find the usual—reminding myself to use the word now—debate.

Moving to an open-ended question
Speed and scale benefits come from the application of new tools to old questions—not a bad thing, but not terribly interesting. Coming at it from another angle, the rise of automated analysis suggests a question about the removal of obstacles: What would you do with online information if you could "read" all of it? We're seeing some early ideas; what else is it good for?

Which question are you thinking about? Is "good enough for media analysis" your standard, or does the prospect of a different set of capabilities (with new tradeoffs, yes) inspire new ideas?

Update: T.R. Fitz-Gibbon picks up the discussion on the Networked Insights blog: Social Media Analytics, Humans vs. Machines.

Photo by Narisa.

This morning, Dow Jones will release the first instance of its new Economic Sentiment Indicator (ESI), an economic indicator based on language patterns in news media. I've heard of several strategies for finding investment intelligence in media content, but this is the first I can recall that aims to predict the performance of the overall economy. Naturally, it reminded me of earlier discussions of which media metrics might be useful as economic tea leaves.

The math behind DJ's ESI is astoundingly simple—"a ratio between the number of appearances of the word recession (and some synonyms) and the number of appearances of the word recovery (and some synonyms)"—yet the company says it is a reliable, leading indicator of economic performance.

Predicting market moves
If a simple sentiment test based on a few numbers can predict the economy, what does it take to predict the performance of specific investments? I've heard of a few approaches at real companies. Naurally, what works and what doesn't work is somebody's trade secret, but here are some things they're trying:

• Volume of discussion
  More talk means something. I've heard of hedge-fund experiments with this simple indicator.

• Sentiment analysis of discussion
  Take the PR-research metrics and look for correlations with stock prices. Think about mining targeted communities versus the entire social media world.

• Sentiment analysis of influencers
  Ignore the crowd and focus on quotes and public statements of executives, analysts, and others with specialized knowledge of the company.

• Discovery of little-known facts
  Apply technology to read the impossible volume of daily information that may reveal—or hint at—a valuable fact. Keep your text analytics busy with securities filings, patent filings, court records and anything else that might hold material information. It's amazing how much is online now.
  

What else can it do?
If you think about the information generated by most social media analysis companies, it's not hard to imagine looking at the dashboards or reports with an investor's eye. Both quantitative and qualitative views can tell useful stories. If you're the communications person, you might try comparing your media metrics with your company's stock price, in addition to financial metrics. Wouldn't that be an interesting chart to have in your back pocket?

You might try thinking of how SMA might benefit other functional areas, too. Certainly, vendors I'm hearing from are applying similar techniques outside of marketing and communications. Apply a little rocket science and consider that the value of this information might show up somewhere other than where everyone is looking. It's way too interesting to stay in a sandbox for long.

On-topic message volume and its comparative derivative, share of voice, always show up in metrics and graphs of social media activity. When the media being measured were print and broadcast, the units were reasonably consistent—articles and mentions aren't too hard to figure out in newspapers and broadcast news. Social media complicates the units with new forms that are less comparable. So what do volume and share mean in this environment?

I started thinking about this as I read Bill Ives's review of Techrigy SM2. He pointed out a chart that breaks down mentions by channel:

You can also sort the mentions by their channel to see where the conversations are occurring. The screen shot below shows an example where Twitter leads the pack for this brand with 190 mentions. Following are a number of blog tools (LiveJournal, Word Press, Blogger, Typepad that in aggregate total more than Twitter. Next in order are: Ning, Wikipedia, Flickr, MySpace, YouTube, and Facebook.

Measuring corn
Think about how you might measure corn. On a small scale, you could count ears or kernels. On a large scale, you could use weight (tons) or volume (bushels). If you're going to compare it to wheat, you need to use the same units. If you're going to compare it to beef, you need to back up and understand the question clearly before proceeding.

Used carelessly, a comparison of raw message counts across platforms might be more distracting than useful. A conversation in the form of back-and-forth one-liners on Twitter might show up as many items, while a longer conversation in the comments of a blog post might show up as one item. Even if you count comments individually, one long comment might contain more content—more statements of fact or opinion—than half a dozen tweets.

Direction, not evaluation
Again, not to pick on Techrigy—message volume is a standard metric—but the profusion of new forms is changing the meaning of some of our metrics. Volume, in particular, needs to be considered in the context of how specific online media work. Frequency distribution by media type can be helpful in suggesting areas for further exploration, but when comparing raw volumes across fundamentally different forms, the actual numbers probably don't matter.