Recently in Measurement Category

I think I've figured out the source of the difficulty—and controversy—in some of the measurement discussions around social media. It all starts when we talk about measuring things that can't really be measured, because they can't be observed. If we called it what it is—modeling—we'd see that differences in opinion are unavoidable.

Take influence. As a concept, it's not all that hard to define, and I don't think there's a lot of disagreement on what it means. But have you ever seen a unit of influence?

What did it look like? A lot like persuasion? What does that look like?

How about reputation? Have you seen a good one lately?

How about engagement? That's all about attention, and interest, and emotion, and focus, and—well, nothing that you can actually see, even with the best instruments.

Measurement requires observation
We don't argue about the definitions of all online metrics. Many of the basics—page views, unique visitors, even hits—have precise definitions, so the discussion moved on to their relevance and the reliability of available data. The shared characteristic is that they're based on observable events. A web browser requests a set of files from a server, and the computers exchange information that can be tracked.

In survey research, the survey itself provides an observable moment. You might question the validity of the questions or sample, and interpretation is open to—um—interpretation, but you do math on people's responses.

We have discrete events in social media, too. People connect to each other on social networks, they like, tag, or share things, and they publish their opinions. These are all actions that can be observed, though what they mean can be the start of a heated discussion. The frequently misleading labels can confuse the interpretation of the data, but the starting point is a set of observations.

Enter the model
With influence, reputation, and engagement, we're dealing with the abstract. None is particularly hard to define, but none can be observed directly. When you can't measure directly what you need, you look for something you can measure that relates to it somehow. You need proxy data, and that's where disagreement begins. What's the right proxy?

Models can be simple or complex, but they all have this in common: each represents the modeler's estimate of how measured characteristics relate to the desired property. Models are abstractions—equations that use measurements to derive values for characteristics which can't be observed or measured.

A model might be based on someone's intuition or extensive research, it may be strong or weak. But here's something else they have in common: the model is not the thing.

The map is not the territory.
—Alfred Korzybski

The reason we don't have standard metrics for such desirable commodities as influence, engagement, and reputation is simple. We can standardize measurement, because we define what is being observed. Modeling defies standardization because it seeks to measure that which cannot be observed, and in the process of defining a model, we incorporate elements that do not apply to every situation.

Modeling for a reason
Models reflect the opinion of the modeler and the objectives they support. Because apparently simple concepts might be used for different purposes by different specialists, we end up with diverse models using the same labels. In essence, we talk about the labels, because they represent familiar ideas (influence, et al), but the models represent what we really care about (such as positive word of mouth, leads, and sales).

If you understand that the label is just a convenient shorthand for a model that takes too many words to describe in conversation, it's not a problem. If the model generates useful information, it's doing its job. Just don't assume that any one usage of the label is the correct usage. Modeling requires judgment, interpretation, and prioritization in context, which are incompatible with standardization.

Photo by gilhooly studio.

As the measurement clubs start to work out their competing standardization efforts for measuring social media, the battle to define influence is flaring up in all the usual places. And while I won't attempt to settle the debate over how to measure influence, I want to point out that the topic is more interesting than whether Klout scores mean anything. A growing group of companies is experimenting with different approaches. Influence, apparently, is the new gold rush.

At Defrag this year, I saw several new companies with new variations on analyzing influence and profiling people. One startup founder described an entirely new—and promising—approach that he's about to take into alpha testing. To his credit, he preferred that I not use the influence buzzword to describe his business.

We call it influence, because that's what it's not

Dance like no one's watching. Sing like no one's listening. Tweet like no algorithm is coldly deciding your social worth.
—Chris Sacca (@sacca)

I'm not comfortable with the influence label, because it's not really what anyone measures. Influence—the real thing, not the black-box metric—isn't hard to define, but it's practically impossible to measure. So everyone uses proxy data, and the proxies vary by company.

A few years ago, I heard Barak Libai speak about the use of agent-based modeling to calculate the value of word of mouth, and I suspect that influence is essentially the same question. But I haven't heard anybody going down that path in the commercial market. It's probably too hard for practical use. Instead, everyone uses some combination of network connections, topic analysis, and audience reaction, which—obviously—equals influence when combined with pixie dust in the correct proportions.

As I started this post, I reached the chapter on influence in Duncan Watts's recent book, Everything Is Obvious: *Once You Know the Answer, and he fairly demolishes the whole idea of measuring influence. In all but the most trivial, contrived scenario, influence is just too complex. It seems the influence controversy isn't limited to the social media discussion. Even in the sociology lab, they use proxies.

If people want "influence," let's sell it to them
If we dial back the expectation that metrics represent precisely what the label says, we might find some use in the growing crop of "influence" tools. We have a selection of single-purpose tools, of course, but it's also common for these companies to provide hooks to connect into other programs. They provide a filter for finding people who have more followers, or whose words seem to lead to more action online, and so one or more of the influence proxies frequently shows up in social media tools.

Here's what I've seen so far. Where available, I've linked to useful information about APIs, FAQs, and how the scores are generated for each company. As always, once you start looking for more companies, you find that they're different in interesting ways.

• Appinions
  Find and profile influencers relevant to topics defined by Boolean queries. Uses text analytics to understand statements by, and about, influencers and specific topics. (api, faq)

• Connect.Me (beta)
  A reputation-scoring system based on individuals recommending each other. Tags link recommendations to specific topics. Connect.Me promises not to mine or sell user data, so it's not an option for developers looking for influence scores.

• Identified
  A career-oriented marketability score based on how well Facebook profiles match what employers search for on social network sites. (how)

• Klout
  A single-score influence metric based on social network activity. "The standard for influence," at least in the sense that it's the one everyone's arguing about. (api, faq, how)

• Kred (beta)
  PeopleBrowsr's single-metric scoring system based on online influence and outreach. (api, how, intro)

• PeekYou
  A search engine for people with a single-score influence metric based on online activity. (api, faq, how)

• PeerIndex
  Influence analysis with scores broken out by topic and activity, audience, and authority subscores. (api, faq, how)

• PROskore
  Business-oriented reputation and experience score based on social network activity, career profiles entered on the site, and on-site engagement. (faq, how)

• Spot Influence
  Contextual influencer identification and analysis based on reach, topicality, and impact. (api, faq, how)

• Traackr
  Influencer search and profiling based on reach, resonance, and relevance. Traackr can also monitor and measure online activity by influencers for campaign management.
  

Lack of a standard never stopped companies from selling their stuff. If we're going to argue about the value of "influence," let's at least consider more of the options.

More posts in the "Build or Buy?" series:

• Building Blocks of Social Media Analysis
  
• Professional-Strength Social Media Aggregators
  
• Text Analytics in the Cloud

Judging from the way people are talking about it, social media analysis is segmenting into at least three subspecialties. As usual, we're using multiple labels that occasionally overlap, so the potential for miscommunication is great. Whatever the utility of any one approach, companies need a complete set of tools, so let's keep these emerging specializations in context.

In 2007, I asked for opinions on a generic term for social media monitoring, analysis, research, etc. I settled on social media analysis as an existing term that could stretch to fit the tools and services then on the market. Since then, I've also argued for an expansive interpration of the listening metaphor. Lately, though, I'm seeing a lot more of these labels:

• Social media monitoring
  In 2005, companies started to learn that people were talking about them online and they needed to pay attention. Today, we have tools and case studies, and more companies are prepared to notice and respond when someone mentions them. The response might come from a customer service or PR function, but the basic idea is what Radian6 calls "the social phone:" social media represent a new customer-service touchpoint, and companies need to respond to every mention that merits or requires a response.

• Social media analytics
  Every 15 minutes, someone announces a new tool for measuring social media. Most of these focus on the structured data of social media: seemingly hard numbers, such as friend/follower counts, mentions, shares, likes, and Facebook pageviews. This approach blends social media and web analytics, and it's good for questions such as, "is my Facebook campaign working?" If your ROI comes from online sales, this approach is an especially powerful tool for managing social media marketing efforts.

• Social media intelligence
  Analyzing the content of what people say online—topics, sentiment, emotions, and the trends and underlying causes—is starting to be called social media intelligence (I refuse to use the unfortunately abbreviated buzzword, social intelligence, in this context). This is perhaps the least consistently applied label, but whatever you call it, measuring and analyzing online content looks increasingly distinct from measuring online activity (the analytics view).
  

This doesn't begin to cover all of the variations in terminology we're using, and these categories aren't even mutually exclusive. But they do represent a division I'm seeing in both the thinking about, and the capabilities of the tools for, listening in social media. We're getting better (?) at talking past each other, which is not making it easy for beginners.

Update: All that and I forgot to mention social media research—thanks to Annie Pettit for the reminder in the comments. Also, here are a few of the many posts that inspired the topic:

• Social Media Monitoring is Not Research, Tom O’Brien & David Rabjohns (MotiveQuest)
  
• The Difference Between Social Media Monitoring and Measurement, Eric Melin (Spiral16)
  
• When to say Monitoring Social Media means nothing, Gianandrea Facchini (Buzzdetector)

Photo by Dan Thompson.

Way back when MTV played music videos, I was on the radio. On the creative side, we got excited about the ratings—especially when we topped the market. When I was on the sales side, I learned about how the business uses those numbers. In the course of writing a review of Social Media Metrics, Jim Sterne's excellent overview, I thought of some similarities in how easy it is to focus on the least useful numbers.

One number for the public, many for the pros
When the radio ratings came out, everyone knew that 12+ ratings were for newspaper articles. Somebody had to have "the most popular" station. Anyone using ratings in the business—whether their job was sales, promotions, or programming—knew to look into the breakouts.

Instead of looking at overall audience size, we wanted to know how we ranked with important demographic groups—and not just rating, but time spent listening and cumulative audience size. We looked at specific dayparts (times of the day) to see how the programming held up, and we were very interested in how we did against stations with similar formats. The 12+ number that always made it into the local paper never came up in the inside discussions.

Drill into the social media numbers
Social media generates some feel-good numbers, too. Friends and followers numbers might not mean much, but they're hard not to notice. So the web analytics angle is decently complicated, and a lot of people are working out how to find something meaningful in influence, traffic, and engagement. Those aren't the numbers I want to pick on.

I've been paying attention to the listening business since 2006, and I've heard a couple (hundred) explanations of what people are doing there. There's a lot of variety out there, and tools with significant number-crunching heft. But the enduring argument about sentiment analysis suggests that we're still not over the mood ring.

Whatever tool you use, the first number it gives you—followers, post count, overall sentiment—is candy for the boss. The useful insights are deeper in the numbers, in the filters and cross-tabs that you use to slice and dice the data. Look at topics within sentiment within demographics on a subtopic deep dive. Compare with competitors and ask the why questions suggested by sudden changes. Then you might find something you can act on.

Ratings books were printed on paper then, too. Let's just ignore that, mmkay?

Photo by D'Arcy Norman.

Customer ratings are useful things, aren't they? They help when you're making a purchase decision, even if you end up disagreeing with the rating. And they're showing up more and more—in the form of stars, likes, favorites, and thumbs-up buttons showing up everywhere. Naturally, all this structured data makes us want to analyze it, but what do the ratings really mean?

What if people have figured out that these scores do something, and they're trying to make the algorithms work better for them?

Tipping the DJ
Here's an easy example. Pandora is a wonderful service. Tell it the name of an artist or song that you like, and it starts playing music that you'll probably also like. The new list becomes a "channel" that you can listen to any time you like.

As Pandora selects songs for you, you can rate the tunes with thumbs-up/down buttons. Thumbs up? You'll hear the song more often when you listen to that channel. Thumbs down? Never again. If you're going for a particular sound, you might give a thumbs-down to a song that you like, in order to remove it from that channel.

Before you analyze the ratings (assuming you could get the data), what do they mean? In this context, it does not mean "I like/don't like the tune." It means that the user wants to hear more or less (none) of that tune on this channel. If you create a quiet music channel, you might give your favorite artist/song a thumbs-down, because it doesn't fit with the channel.

iTunes does something similar with its option to play higher-rated songs more frequently in its shuffle mode. But its five-star ratings are global within a user's playlist, so it's trickier to fine-tune individual playlists. Still, more stars don't really mean "I like this more" if the system interprets them as "play this more often."

Playing favorites
Media-sharing sites (YouTube, Flickr, SlideShare, and the like) give users the ability to mark an item as a "favorite." Which means the person really, really, likes it—or wants to be able to find it again. You'd need to do usability testing to figure out how people use the feature. My take is that favorite is like friend: overused and meaning something different in the online context.

Ever favorite a tweet? Was it really one of your favorites, or is that just how Twitter's bookmark feature works? Thought so.

Like a discount?
Who doesn't like fans, right? They're almost like friends. But a survey by ExactTarget and CoTweet found that 40% of Facebook fans are there for discounts and promotions. Maybe they're a lot like Facebook "friends," in that the label doesn't really describe what's happening.

Sometimes, a star is just a star
I'm more inclined to believe the product ratings people give on shopping sites, though there's always the risk that a "customer" review is from an employee (high score) or competitor (low score). Usually, the 5-star scoring system seems to be what it claims: people's opinions of the product.

Still, it's worth considering whether product ratings might be considered in sites' recommendation engines. I know that Amazon, for example, recommends products based on things you've bought or looked at, and I don't always want to see more of what I just bought. I don't know if product ratings figure into the mix, but what if people think they do? How does that affect your interpretation?

Photo by Adam Arroyo

Buzzword bingo bonus: use these words in a sentence: social, ratings, sentiment, influencer, customer, tomato.

Today's Wall Street Journal had Twitter abuzz about social media monitoring and privacy in closed communities ('Scrapers' Dig Deep for Data on Web). Specifically, a health discussion board and a social media analysis vendor using individual accounts to access personally identifiable health information. It's obviously an ethical question, but whose ethics apply? As far as I can tell? Nobody's (yet).

People are sharing personal stuff online, sometimes sharing more than they realize. We need to be careful about how we handle this information, but from what I can see, the ethical standards are just as siloed as the measurement standards. People brought along whatever ethics they subscribed to before they started dealing with social media, but the existing standards don't really cover the new activities.

Think about the different functional roles where you might find companies using social media data:

• Market research
  Market researchers have strong ethical standards that come from social science research. They get into things like informed consent, but does that really apply to data mining of publicly available data? Do they apply if the data is aggregated, and no personally identifiable information is preserved? What ethical standards apply to desk research?

  Jeffrey Henning wrote about the etiquette of eavesdropping and presented a webinar on consumer attitudes towards social media market research. The short version is that people persist in expecting privacy in their online conversations, despite the public nature of the forums they use. But does their expectation of privacy online translate into an ethical obligation for researchers?

  Update: IMRO and CASRO guidelines may apply to social media research.

• Public relations
  PR ethics say a lot of being honest and transparent in public statements, representing the client and the profession well… but what about the ethics of monitoring and measurement? A recent discussion of ethics in PR measurement suggests that that conversation has only just begun.

• Marketing
  WOMMA takes strong positions on its members' marketing activities, but the closest it comes to mentioning monitoring or research is when it commits to "promote an environment of trust between the consumer and marketer." Other marketing codes I found had a similar emphasis on outbound marketing over inbound information collection.

  Update: WOMMA also calls for members to "respect the rights of any online or offline communications venue (such as a web site, blog, discussion forum, traditional media, and live setting) to create and enforce its own rules as it sees fit."

• Customer service
  Is customer service sufficiently organized as a discipline to have its own code of ethics, or does it simply inherit the company's overall standards? I'll bet you that any existing ethics deal with one-on-one interactions with customers.

• Human resources
  HR ethics related to personal information are based on information that companies aren't supposed to use in hiring decisions. danah boyd shared some thoughts on regulating the use of social media data in hiring.

• Strategy/intelligence
  SCIP's code of ethics doesn't commit to much more than obeying the law. Other types of intelligence organizations get some leeway even on that. If you don't want competitors spying on you, your only real defense is to learn about INFOSEC.

• The source of the data?
• What you do with it?
• The job title/professional affiliation of the user? What if the labels themselves lack agreed definitions?
• No ethics, just laws?
• Nothing—there are no rules?

Update: Is it time for Ethical Standards for Listening Vendors?

Related:

• The Ethics of Listening
  
• Ethics, Open Sources and CI

Bruce Schneier's taxonomy of social networking data (via Tim Finin) provides a helpful starting point for thinking about the various ways that personal information finds its way online.

...and I'll tell you what your job is. Especially if you tell me "how to measure social media."

Measurement silos are alive and well, and the measurement cliques they foster are working hard to perfect their craft. They're coming up with increasingly sophisticated measurement approaches and tools for measuring social media. The unintended secret is that the metrics and objectives they embed in their work are designed for the needs of specific functional roles, and that's not usually stated.

31 flavors with the same name
Have I told you the story about the unspoken modifiers of marketing? Think about all of the subspecialties within marketing. Unless you work in a small company, marketing tends to be divided into more specific roles.

As a product manager and product marketing manager at large technology companies, I worked with product marketing, marketing communications, channel marketing, field marketing, event marketing, promotions… I didn't even know the corporate marketing people who did the high-level branding, advertising, and PR.

Later, an outbound telemarketing manager at another company asked me if I had done "marketing," and I said yes. But when she said "marketing," she meant direct marketing for lead generation—email blasts that would feed the call center. The modifiers that she applied to marketing were unspoken, but they were crucial to understanding what she meant. We were using the same word to mean entirely different things.

Measurement is the same. When people tell you how to measure social media, they're telling you what they are responsible for measuring—what they're responsible for doing.

You don't see what you don't look for
You may believe that you can't manage what you don't measure. I'd like to add that you don't measure what you don't manage (why would you?). But how does that work when the world changes and the old boundaries blur? When the same channels are used for PR, branding, promotions, and customer service (to name a few), whose metrics do you use? How do you measure one environment for multiple objectives?

So far, the answer seems to be that old ways of measuring create blinders that we take to new situations. So the web analytics club, the PR measurement club, the WOM club, the customer service club, and the BI club are all meeting after school to define social media metrics. Their definitions are based on their job responsibilities, but they aren't labelled that way.

Moving up a level
At an individual level, you measure what you're trying to manage, for all the right reasons. At a company level, you measure all of it, and you look for ways to use what you learn here to make improvements there. Companies aren't supposed to be limited by one function's objectives, but that's how we're talking about measurement.

If you're trying to measure social media, don't be limited by what you've used before. Think and, not or, and look around for useful tools, strategies, and metrics that have developed in adjacent silos. That other measurement club is doing good work, and they just may have the tool you're looking for.

Photo by Jonathan Cohen.

Too much information. And increasingly, too many disparate sources of data, many with their own analytical tools. So it's interesting to see a new crop of startups offering tools that pull analytics data from multiple sources into a single dashboard for analysis and reporting.

This is one of those posts that started as a more detailed look at a few tools, but as I was looking around, more platforms kept popping up on the radar. So it's become a list, which is probably just as well. Some of these guys are semi-hidden in beta testing, so any detailed description is going to be out of date soon, anyway.

If you're spending too much time trying to corral performance data from multiple online sources, try these on for size:

• Argyle Social
  
• Cyfe
  
• DigMyData
  
• Leftronic
  
• Metricly
  
• Social Report
  
• Sprout Social
  
• StatsMix
  
• Swix
  
• Unilyzer
  

Analytics mashup
What these platforms have in common is the ability to create charts and dashboards that combine data from web analytics and social media sources (Leftronic is different, because of its emphasis on large-screen dashboards for public view). So if you want to see the correlation of Twitter followers and website visitors, you can. If you want to track multiple accounts on one dashboard, you can. If you want to stir in data from your internal databases, some of them will let you do that, too.

What if
Remember the RSS tricks post from a couple of years ago, how you can assemble useful applications by using RSS inputs and outputs as a pipeline between services? With so many APIs going in and out these days, one of these dashboards could be the user interface for some interesting manipulations. For example:

What if you were to combine online sources (social media) and internal company data, run them through some text analytics, and pipe selected metrics out to one of these dashboards to mash them up with web analytics (which you've already linked to business performance). Would you find the elusive social KPI you've been looking for?

It's a list. I've missed somebody. Tell me, and I'll add them.

Why is it that so many people talk about the effects of social media on reputation, but so few mention the more interesting models for measuring reputation? Instead, we argue over how to read the sentiment mood ring, or which media-oriented measurement tracks reputation. In most cases, I don't think we're measuring reputation at all. Instead, we're measuring media coverage.

Media analysis reports on published statements. In the recent past, it focused on media created by professionals, but even as it includes media created by everyone else, it's still mostly about reporting aggregate data based on coverage. The usual metrics—volume, sentiment, topics, and voices—reflect that media-centric view, which is now adapting to summarize consumer's opinions in online media. It's good data for some applications, but it's the shallow end of the pool for understanding reputation.

Wading into the deeper water, we find some companies that take a more nuanced view of reputation. These models start with survey research and are typically calibrated to focus on the relevant attributes for a specific company or industry.

• Reputation Institute: RepTrak
  Measures 23 attributes of 7 dimensions: products/services, innovation, governance, workplace, citizenship, leadership and performance.

• Harris Interactive: Harris Reputation Quotient
  Measures 20 attributes of 6 dimensions: emotional appeal, products & services, social responsibility, vision & leadership, workplace environment, and financial performance. Harris recently released its 2009 report (PDF).

• APCO: Reputation Insight (PDF)
  Multi-factor models customized for each client.

• Millward Brown: BrandZ (more)
  Evaluates the financial return attributable to the company's brands, based on an analysis of financial data and consumer research.

Is this stuff not accepted? Is it too advanced? Maybe too confidential? Or is it just above the social media paygrade? The audience in the room where I first learned about this was rather senior.

Hat tip to Leslie Gaines-Ross for pointing out some research I hadn't seen.

Sentiment analysis is generating blog headlines again. After reading about the non-response bias of automated sentiment analysis, and that it has no place in social media monitoring, I decided to run a sentiment analysis on sentiment analysis (apparently I like that phrase). I have an account on Biz360 Community from my current project (look for it next week), so I tossed it a quick query and found that the recent buzz was mostly… positive? Hmm.

Here's the thing. Sentiment is not the golden metric. Virtually every social media analysis platform can show you a pie chart on sentiment. At best, it's a first glance—which way is up. Unless you go deeper into the data, all you're looking at is a mood ring.

At the very least, you need to compare sentiment across brands and over time. Yay, it went up! Aw, it went down.

Oh, look, a mood ring. Maybe there's a secret decoder ring somewhere around here that we can wear next to it.

Set aside the methodology question
The automated sentiment debate continues, but I want to focus on what to do with sentiment data once you have it. On scoring methodology, remember that it's not a simple question of human vs. computer (though this Attensity post explains more of the automation than most people have probably seen). Most of the social media analysis (SMA) platforms I've just reviewed allow users to edit sentiment scores, so when you find a post with the wrong sentiment score, you change it. About half of the automated sentiment processors learn from users' changes, too.

But today's topic is what to do with the sentiment data you have.

Trends, segments, and causes
Sentiment, by itself, is a mood ring—a happiness indicator. It's nice to see the happy color, but there's not much information there. If you dig into the sentiment data, though, it starts to contribute to useful analysis.

Take the trend chart. Direction is interesting, but what about slope? Sudden changes are especially interesting. Any spike—not just in sentiment, but in volume or anything else—is the chart's way of saying "look over here." A spike on a chart is a big ol' why, waiting to be asked.

Sentiment really gets interesting when you combine it with other measurements. Most SMA platforms use sentiment scores as a filter for segmenting the data. What are the prominent and emerging topics within negative-sentiment content (and again with positive)? How does sentiment compare within a topic, across different media types or specific sources? Is a topic emerging from a source that writes negatively about you, or is it a friendly source?

Crafting queries and combining filters could be a whole series of posts, or maybe a book. That's why insight isn't automated: what you do next depends on what you find. If you're looking at the mood ring and wondering what it means, you haven't even started.

Join me at the Sentiment Analysis Symposium (New York, 13 April), where I'll talk about how to make an informed purchase decision in social media analysis.