Why You Can't Measure Influence


RulerplaneI think I've figured out the source of the difficulty—and controversy—in some of the measurement discussions around social media. It all starts when we talk about measuring things that can't really be measured, because they can't be observed. If we called it what it is—modeling—we'd see that differences in opinion are unavoidable.

Take influence. As a concept, it's not all that hard to define, and I don't think there's a lot of disagreement on what it means. But have you ever seen a unit of influence?

What did it look like? A lot like persuasion? What does that look like?

How about reputation? Have you seen a good one lately?

How about engagement? That's all about attention, and interest, and emotion, and focus, and—well, nothing that you can actually see, even with the best instruments.

Measurement requires observation
We don't argue about the definitions of all online metrics. Many of the basics—page views, unique visitors, even hits—have precise definitions, so the discussion moved on to their relevance and the reliability of available data. The shared characteristic is that they're based on observable events. A web browser requests a set of files from a server, and the computers exchange information that can be tracked.

In survey research, the survey itself provides an observable moment. You might question the validity of the questions or sample, and interpretation is open to—um—interpretation, but you do math on people's responses.

We have discrete events in social media, too. People connect to each other on social networks, they like, tag, or share things, and they publish their opinions. These are all actions that can be observed, though what they mean can be the start of a heated discussion. The frequently misleading labels can confuse the interpretation of the data, but the starting point is a set of observations.

Enter the model
With influence, reputation, and engagement, we're dealing with the abstract. None is particularly hard to define, but none can be observed directly. When you can't measure directly what you need, you look for something you can measure that relates to it somehow. You need proxy data, and that's where disagreement begins. What's the right proxy?

Models can be simple or complex, but they all have this in common: each represents the modeler's estimate of how measured characteristics relate to the desired property. Models are abstractions—equations that use measurements to derive values for characteristics which can't be observed or measured.

A model might be based on someone's intuition or extensive research, it may be strong or weak. But here's something else they have in common: the model is not the thing.

The map is not the territory.
—Alfred Korzybski

The reason we don't have standard metrics for such desirable commodities as influence, engagement, and reputation is simple. We can standardize measurement, because we define what is being observed. Modeling defies standardization because it seeks to measure that which cannot be observed, and in the process of defining a model, we incorporate elements that do not apply to every situation.

Modeling for a reason
Models reflect the opinion of the modeler and the objectives they support. Because apparently simple concepts might be used for different purposes by different specialists, we end up with diverse models using the same labels. In essence, we talk about the labels, because they represent familiar ideas (influence, et al), but the models represent what we really care about (such as positive word of mouth, leads, and sales).

If you understand that the label is just a convenient shorthand for a model that takes too many words to describe in conversation, it's not a problem. If the model generates useful information, it's doing its job. Just don't assume that any one usage of the label is the correct usage. Modeling requires judgment, interpretation, and prioritization in context, which are incompatible with standardization.

Photo by gilhooly studio.


Excellent post! The missing ingredient is theory and hypothesis testing. Scientists theorize about the unobservable all the time. But the hypotheses derived from theory involve the impact of the unobservable on the observable and measurable. One well known example is CERN and the Large Hadron Collider.

Marketing should take a cue from scientists — real scientists, not "data scientists" — and deduce what observable outcomes should occur and when as a result of their marketing and start there.

Thanks, Theresa. I wasn't entirely sure I was being clear, but you're right there, and you've added a logical extension I wouldn't have thought of.

Let's see if I can translate the terms appropriately: A theory of influence suggests hypotheses, which are built into the model ("this is a useful proxy for influence"). Marketing initiatives are experiments to test a hypothesis, and the outcomes tend to confirm or disconfirm the hypothesis. Successful experiments tend to support the theory, and unsuccessful ones tend to disprove the theory. Am I getting the terms right?

In other words, design your metrics to measure outcomes, and then you'll know whether your initiatives are working. If they work, maybe your theory is correct. Getting hung up on a label (influence, etc.) is a distraction.

BTW, has LHC created a black hole that will destroy the world yet? I may have missed the news…

Great post, Nathan. Another missing element (aside from @theresa's hypothesis testing): most of the "influence" vendors don't actually offer much insight into how they build their models. Tell us the formula, or at least be specific about the elements that make up the model—here's how online influence is different from real world influence—and it would be a lot easier for the shorthand to make sense or have a practical application...

Good point. If you're buying a model (and with influence, that's what's for sale), it'll be hard to make an informed decision without a good idea of what the model runs on.

Hi Nathan,

I thought this was pretty insightful in that you've identified some of the common assumptions of data-modeling that most people over look. The whole idea that the data model represents what the data-modeler thinks is smart & correct.

SocialMatica has an open model that's tuneable...but - does this change the mixture and create the ability to measure semantics within an ecosystem - in your opinion?

Thanks, Joshua.

I'm not sure I get the question. Measuring semantics is possible, whether the model is adjustable or not. I suppose the difference is in whose priorities are built into the model. If it's more flexible, it's more likely to reflect the user's thinking.

The components that go into the models for influence, engagement, and reputation are definitely measurable. That's the point of using proxies and models.

Comments are now closed for this entry.

About Nathan Gilliatt

  • ng.jpg
  • Voracious learner and explorer. Analyst tracking technologies and markets in intelligence, analytics and social media. Studying complexity and futures.
  • Principal, Social Target


  • Subscribe by email