Along with her kind comments about my work, Havi Hoffman wondered if half the money is wasted in social media/word of mouth metrics, too.
What if we really need to throw out half our metrics, half our assumptions? Which half would you pitch?Evaluating metrics is tough. Just Google IAB comscore netratings to see what's going on with online audience measurement. It's one thing to ask a vendor what services they provide; going under the hood to evaluate their methods is a bigger challenge, both technically and because of the obvious sensitivity issues.
At least audience metrics are generally defined. In the social media/WOM measurement discussion, the definitions of some important terms aren't quite settled (quick, what's influence, precisely?). So, for now, we have qualitative measures and intuitive definitions that may vary across vendors. It doesn't mean they're not useful, but they're not yet standard.
Alan Wilensky's post raises some thought-provoking questions on the usefulness of social media metrics (read the related posts while you're there and at BlogWhine). Be sure to see Matt Hurst's response, including the comments.
Evaluating the current state of this young industry and its practices is an interesting topic, but for client decisions, three steps really matter:
- Understand what you're trying to accomplish. What information do you want to collect, and what do you want to do with it?
- Look at the existing companies and what they actually offer—services, analysis, deliverables and applications. That's what the Guide to Social Media Analysis is all about.
- Look for potential matches between available services and your objectives. Those are the companies to ask for more information.
Nathan,
I like your point about 'influence'; the same goes for 'sentiment', 'authority' and so on. One of the key insights I've taken from working in this space is: if you can't define a term, it's the wrong term. Underneath most attempts at influence or authority are simple link counting methods. That being the case, customers are better served by metric names which reflect what they actually measure.
Thanks, Matt. This has me thinking about the tradeoffs involved in choice of language.
The nice thing about labels like influence is that they suggest what the data are useful for. The downside is that it's imprecise and can be confused with other definitions of the term (see the comment on this post at Social Media Today for a negative reaction to influence).
I do think that precise definitions are more important if your objective is to create solid metrics than for the more qualitative applications on the market. As you pointed out, this will get worked out by vendors and clients as they identify the real value and refine the process.