The Net-Savvy Executive

Trust is an issue for an industry based on extracting meaning from what people share in social media. People don't have to use these services, and if they decide that their information might be used against them, they can stop. This week's revelations about the US intelligence agency monitoring social networks (among other sources) creates a massive trust issue for everyone who works with social media data. What now?

(This won't be an analysis of the NSA and Prism. I'm working from the same sources you are, and we'll probably have new information by the time I finish writing, anyway.)

The world reacts to US actions
David Meyer points out a threat to cloud computing vendors as customers and governments react to the news. US-based vendors can expect special challenges selling in Europe, where privacy is more protected and signs of a blowback from Prism are already appearing. In cloud computing, the trust issue relates to custody of the data—do you trust your vendor to keep your data safe and secure?—and the government version translates as a question of US-based vendors' ability to keep commitments to foreign governments.

But cloud computing is essentially just data center outsourcing. What does it mean for an industry that exists because of people's willingness to share publicly?

Access to the data is everything
The challenge to the social data industry is different. It's indirect, but potentially existential. What happens to your business as a result of the reaction to Prism? Will social networks tighten their terms of use to block data mining? Will EU safe harbor agreements create new requirements to protect user data (possibly by keeping it outside the US)? Will new legislation designed to limit government abuses include new limits on private-sector users?

Secret collection of private data by government agencies is fundamentally different from social media monitoring outside government. In business, we're working with publicly available data, which anyone can access without breaking the law or hacking a system. It's not espionage, but the facts aren't the problem.

The problem, as ever, is perception. The NSA is all over the news, and in the heated environment of a breaking story, subtle distinctions can get lost. The risk to the social data industry is that a reaction to government surveillance could become a problem for anyone doing the less intrusive type of monitoring.

How will you respond? What's your plan for minimizing the overreaction if it starts to get out of hand?

Responding as an industry
At its Big Boulder conference this week, Gnip announced the Big Boulder Initiative, which is an effort to start an industrywide discussion of the issues it faces. Trust is one of five issues they highlighted as starting points for discussion. The other news this week highlights the wisdom of the choice.

I'd go farther and ask a question I've asked before: should the companies who work with social data form an association to coordinate these discussions, codify standards, and speak for the group?

The ethics of social data
Let's go back to trust and consider the ethics of working with social data. Bob Gourley at CTOVision recently gave me a copy of Ethics of Big Data, a short e-book that lays out a process for establishing ethical limits to the use of big data. It's a worthy challenge, but I think the first step in the process—exploring an organization's values—will lose everyone. The Friedmanesque view that a business exists only to make a profit is common, which leaves only the law as a restraint on what can be done. "Be profitable" isn't the sort of value that will drive a hearty discussion of ethics.

I do think it's possible to have ethics of listening, but I don't see an existing standard that really applies. I don't see, for example, how ethical standards for social scientists, with their strict limits on personally identifiable information (PII) apply to social media monitoring in customer service. The standard for competitive intelligence boils down to "don't break the law," which appears to be the relevant limit on secret government programs, too.

Here's a starting point for discussion
I suggested a set of ethical standards for listening vendors in 2010 as a starting point, but the discussion went nowhere. Maybe it's time to try again. Comments are closed on the old post, but I'd welcome any discussion of the draft here.

The usual defense of social media monitoring in the private sector is that we're working with publicly available data, but monitoring public data can still be creepy. What's the plan for protecting public-source data mining from an overreaction to something far more invasive?

Photo by Marsmettnn Tallahassee.

Asking a computer to make sense of everyone's written opinions is a big challenge, but it's not the last one that social media will impose on anyone who wants to analyze it. We're sharing a lot of pictures in our virtual hangouts lately, which means it's time to update the old question. Instead of "what are people saying about us," the new question is something like, "what do people's pictures tell us about what they think of us and how they use our products?"

Just as the shared images give us access to new types of information about people, their tastes, and more, emerging technologies offer the promise of helping us understand the images at scale. To the vocabulary of text analytics or natural language processing, add computer vision. As with its text-processing cousin, it's not as evolved as your eyes, but it doesn't blink, and it doesn't sleep.

Looking at the photo directly
Let's say you want to track publicly shared photos that contain your company's logo. Without image analysis, monitoring depends on keywords in posts and photo descriptions, filenames, tags, and other metadata. It's better than nothing, but it has limitations. You're going to pick up images that don't actually include your logo, and you'll miss photos that include your logo but aren't about your logo.

If your tool can "see" product logos in photographs, you get access to a different type of information. You start to catch products and logos in the wild, where people really use them. The brand protection guys will like enhanced abilities to track counterfeits and parodies, but maybe this opens the door to a new kind of online ethnography, too.

Finding the technology
As demand picks up , you can expect the serious competitors in social media analysis to add image search capabilities. Already, Ninestars has added image recognition from a partner, and Meltwater's OculusAI acquisition suggests future capabilities with images. They won't be the last.

These companies are going at the image recognition challenge directly:

• Curalate

• Ditto

• gazeMetrix

• LTU Technologies

• TinEye
  

What's next?
Computer vision has lots of potential beyond spotting logos in photos. I imagine that this sort of product/logo identification will extend to video, though I'll need to talk to an expert to understand when to expect that.

And then there are people. We already have identity tagging in Facebook, and big money is going toward advancing facial recognition. I also found Real Eyes, a company that analyzes emotional responses from video, so visual analysis of faces isn't limited to identifying their owners.

The computers aren't just reading. They're starting to watch, too. Can you do something good with that?

This is one of those list posts that will grow as people point out more companies. Who'd I miss?

When people ask me what I do, I usually say something about exploring the edges of the market for intelligence and analytics capabilities, starting with social media data. I also like to connect threads from separate topics and look at things from unusual perspectives. With that as a warning of sorts, let's pull some threads from new methods, old metrics, and emerging science to see what they do together. It may sound like so much theory so far, but this is all about practical analytics for management.

Thread one: A new view of social media in a customer journey framework
It started with a briefing from SDL on their new Customer Commitment Framework (CCF). I'm always interested to see people do something different with social media data, and I give bonus points for tools that provide quick and clear access to useful information.

SDL's approach is to monitor business performance at key points of customer journeys by analyzing what they have to say in social media. They want to know what people are thinking as they progress toward a decision, whether that decision is about buying a product, telling others, or becoming an advocate for the product. CCF's analysis is always presented in the context of a customer journey, so—in theory, at least—its numbers provide a drill-down into the performance of different parts of a company's marketing and operational performance as experienced by customers.

I haven't tried CCF and its dashboard component yet, but if it works as promised, its alignment to identifiable business levers could make it a valuable analytical tool.

Thread two: Exploring possible futures with agent-based modeling
Call it simulation to avoid scaring people off, but the complexity science tool of agent-based modeling has come to market. When I saw that Icosystem had spun up a company, Concentric, to offer ABM tools for marketers, I knew I needed to learn about it.

Concentric's book, How Customers Behave, was a good start, but some of my earlier reading and the Santa Fe Institute's MOOC on complexity made the background sections somewhat redundant. One key takeaway I've found is that, despite the complexity label, this stuff isn't too complicated to understand.

Where SDL looks for signals about what has happened, Concentric starts by building models of customer journeys, playing out the decisions faced by individuals in the market ("agents"). Once the model can "predict" the past, it's ready for use in simulating the effects of different strategies and tactics. Depending on your needs, their software can incorporate social media and other online data sources, or it can look broadly across media types and operational data.

Is this what you expected?
If you put threads one and two together, you get simulations to explore possible outcomes of different strategies, and measurement of customer opinion at critical points to indicate actual performance. One looks forward to explore what may happen, and one looks at the recent past to understand what has happened.

It seems like a powerful combination to me, but let's add one more thread. What about hard data?

Thread three: Marketing analytics and the view of the process
I once worked on a project for one of the big phone companies that was concerned about customer churn in their high-speed Internet business. They were adding new customers as fast as they could, and they wanted to avoid losing existing customers. Our analysis rested on the insight that sometimes you lose the sale even when the customer wants to buy. Before it was trendy, we looked at the post-purchase customer journey and found some measurable issues.

I usually see customer journey models that assume that customer attempts to purchase virtually always succeed. If you're selling online or in retail, that's probably close to true. But do you know that it's true, or do you assume it?

For the phone companies, DSL service circa 2002 was constrained by geographic footprint, technical limitations, compatibility issues, and customer ability. Each of these added another step in the journey and another opportunity to lose the sale. Given the operational metrics they already had—order attempts, accepted orders, activations, etc.—you could track post-order performance as a series of multipliers between zero and one. A simple step-down chart would show you where you were losing customers, so you would know where to invest to improve the process.

For a subscription-based business, recurring revenue is everything, so you need to pay attention to customer retention and anything that drives them away. This is already a long post, so let me point to a post from Keith Schacht about customer acquisition and retention, and a VOZIQ post on customer issues in telecom. The point is, your customer's experience may not end with the purchase, and your measurement of the experience shouldn't, either.

When you combine the effects of attracting new customers and losing existing customers, you end up with a survival analysis, which brings us back to complexity and the potential of agent-based modeling. The fact is, a given rate of customer additions and losses combine to set a ceiling on your possible customer base, so it's crucial to understand where and why you lose them.

Really, it all comes together
Pull these three threads together, and we start to see the potential of forcing analytics out of their comfortable silos. Agent-based models offer a tool for understanding how things might (not will) play out under different scenarios, including the possible outcomes of different marketing strategies. Analytics based on similar models of customer journeys provide a view of the recent past to test expectations and continually reevaluate the models. Combining social media data with customer data and operational metrics allows us to see a more complete picture: Where hard data is unavailable, social media indicators can fill the gaps. Where hard data is available, it provides a test of the social media indicators.

Pull the threads together, and you get a view that combines future and past; what people say and what they do; what happened and why. Sounds pretty useful to me.

Another sketch from the whiteboard

A couple of years ago, a suggestion that I develop a maturity model for social media analysis led to a different kind of model. My approach to this space has always been to explore its edges, looking for what might be next. One effect I've noticed is that change circulates through the ecosystem of companies, their customers, and their suppliers. Where change keeps coming, everyone's learning together.

A linear maturity model defines development stages toward a known destination, but in a system where everyone is learning, the destination is still unknown. We react to others, and others react to us. Change reverberates through the system, and we don't yet know what maturity looks like.

What this means in social media analysis
If social media analysis were good for one thing, we could have a simple maturity model. The products would progress toward a theoretical ideal, and clients would mature toward efficient, effective business practices. But the technology stack is built on areas of active research, new platforms are driving new consumer behaviors, more business functions are showing interest in how to use social media to do their jobs, and vendors are trying new ways to distinguish themselves.

Virtually every piece of the puzzle is moving.

Let's go to the whiteboard to see if we can visualize it.

There's a lot going on here, and this is the oversimplified version. Here's the basic dynamic: on the right, new capabilities become practices; on the left, new expectations become requirements. In the overall system, we expect more from our suppliers as we adapt to new capabilities and adopt new practices.

Think about what is being learned in each loop.

• Tool vendors combine their own R&D with new capabilities from research labs and partner companies to expand their products' capabilities, enabling new tactics for their clients, who provide feedback and new requirements based on real-world use of the products.

• Consumer-facing companies experiment with new tools and capabilities, and they learn from both operational results and customer reactions.

• Customers react to companies' online tactics, adjusting their behavior to maximize their own benefit. When they find a practice they like, they may expect other companies to mimic it.

Who's learning fastest?
We know of some unintended lessons, such as teaching customers to complain publicly for a quicker response, and redefining like. But where do we look if we want to get ahead of the market? Try these key areas:

• Outside innovation - New research and inventions may provide answers to questions you've wanted to answer.

• Product capabilities - What's possible keeps changing, but don't look only at existing suppliers. Look at adjacent markets for capabilities worth adapting to new applications.

• Client requirements - It's always worthwhile to pay attention to what companies say they'll pay for.

• Client capabilities - Watch what companies are actually using, too.

• Competitor actions - Watch early adopters for practices that may become standard. Is there a better way to do it?

• Customer expectations - How are people reacting to new business practices? What issues are being raised? What new expectations?

More ideas from the whiteboard:

• Computer attention
  
• Omniscience

Do you have any of those mix-and-match books that let you remix parts of their pages? (It's ok; you can claim they're for your kids.) We once bought a story starter for my son (see, like that) that combines an opening quote, a character, and a situation. Put together a random grouping, and you have the beginning of a story.

That's sort of how I look at data and analytical methods.

Here's how it works: First, remember the basic building blocks of social media analysis: data, analytics, and application. Now, let's generalize from the social media example, because this isn't just about social media data.

We get three basic pieces:

1. Data
   Internal and external sources, open (freely accessible) and proprietary (paid). There's a lot more here than most discussions get into.

2. Analytic methods
   Sentiment analysis, topic clustering, source profiling, statistical analysis, geospatial analysis—the list goes on and on. This is a good area for And not Or thinking.

3. Applications
   In a software business, this usually refers to the product, its features and their benefits. Here, though, think about the work that can be enabled through the application of data and analytics. Think about functional roles and what they need to do, and then you may get ideas about what a software application should do.

We tend to get stuck in familiar modes of operation, thinking that a certain type of data implies a certain type of analysis, which is useful for a certain application. We fall back on social media + sentiment analysis + marketing. You might even think of it as a chemical reaction: social media + sentiment analysis -> value for marketing.

It's comfortable. It's familiar. It's not wrong. But there's more.

Time to mix it up
To find more value in the data and analytics, we need to start flipping the pages in the book. Which analytic methods could make this source of data useful for that function? I know what I know. What have I not yet found?

You can start with any piece first, and switching the order aids discovery. You might start with a functional role and ask what information would help them. You might start with a data source and think about how it might be useful. Or you could ask how an analytic method might turn data into something meaningful.

The secret is that each category has more options than you're probably using. More sources of data, inside and outside your organization. More analytic methods—some still being invented. More functional roles than the ones you're used to supporting.

Combine them, and you put familar data through unfamiliar analytics. New data through existing analytics. And you find ways to create value beyond the marketing, public relations, and customer service roles we associate with social media.

Do I have specifics? Sure, but not all in one blog post.

The mix-and-match book is similar to the Omniscience framework I proposed, which is all about understanding how intelligence and analytics can be useful at all levels in the organization.

Around the end of the year—or the beginning—I look at the numbers to see which blog posts people have looked at the most, and it's always the old posts that dominate the list. It's the same for 2012: only one of the top 10 posts in 2012 was something I wrote in 2012. Since the stats favor the old posts, here's a recap of some of the stuff I'd hope you didn't miss.

• Three Buckets of Social Media Data
  I've tried categorizing social media before, but this one is turning out to be more helpful than my previous attempts. When working around monitoring and analysis, think of social media as three types of data sources: about content, activity, and people. If you haven't considered all three, you have more work to do.

• Why Government Monitoring Is Creepy
  The meaningful distinction between private and public spaces is changing faster than our sense of privacy, both online and in the real world. The rise in drone activity around the world will make this an increasingly important topic.

• What Happens After Your System Notices Something Important?
  No matter how much intelligence we try to engineer into our analytics systems, most are still working toward putting data in front of a person. What if the system helped with the next steps?

• Can You Trust Social Media Sources?
  Finding meaningful insights in social media data is challenging enough, but there's more. Some of the sources you're finding may have been put there by people who intend to deceive you.

• The Four Locations of Social Media
  Putting social media data on a map is helpful, but remember that location might not mean what you want it to mean.

Previous years' lists
2011: Top 10 Posts, Revisiting 2011
2010: Top 10 posts, Thinking through 2010
2009: Top 10 posts

As the year winds down, it's time to see what people found on the blog this year, and once again, the most-read posts are generally older ones. Clearly, search-engine traffic favors older posts, and the visits add up through the year. But look at it this way: these are the posts on topics people searched for this year. Does that say something useful?

1. What Does Salesforce-Radian6 Deal Mean for Everyone Else? - March 2011 (#3 in 2011)
   
2. Applying Social Network Analysis to Social Media - September 2010
   
3. Human vs. machine analysis - April 2007 (#5 in 2011)
   
4. Visual text analysis - April 2007 (#6 in 2011)
   
5. Visualizing networks based on communication - February 2008
   
6. Global Social Media Usage Patterns - January 2011 (#4 in 2011)
   
7. Professional-Strength Social Media Aggregators - June 2010 (#8 in 2011)
   
8. Monitoring Social Media Before You Have a Budget - May 2008 (#2 in 2011)
   
9. Why You Can't Measure Influence - January 2012
   
10. Five Modes of Listening - September 2009

Previous years' lists
2011: Top 10 Posts, Revisiting 2011
2010: Top 10 posts, Thinking through 2010
2009: Top 10 posts

I've been tracking acquisitions in social media analysis for years. It feels like we've had a lot of deals this year, and based on what I've seen, it's true. The volume has gone up every year. This year, I thought I'd do something new: I wrote a recap of the activity, which you can find at The Year in M&A, Social Media Analysis 2012.

As with the big directory of over 400 companies, the list of transactions requires some judgment about which deals to include. The companies that offer turnkey platforms for monitoring social media are easy. Others offer some of the building blocks for developers who want to focus on other pieces or enterprises building their own tools. Most run on a software-as-a-service (SaaS) model, though some licenses software for on-site installation. The variation gives the market a fuzzy edge, so it's not obvious what to include.

For SMA, I've chosen to go with a coarse filter, which means that I tend to err on the side of inclusion. If I sometimes reach too far, its because I think there's value in knowing what's happening on the other side of the fence.

Looking back
2012 was about more than acquisitions, of course. The investment deal flow continues, which I plan to recap separately, and I'm still discovering new—and new-to-me—companies fairly regularly. At the other end of the lifecycle, I've noticed an increase in companies shutting down quietly and a few sales of "assets" (as opposed to operating companies).

In 2006, I thought I'd find every company in the world developing tools to work with social media data. By now, I think we've established that it's not possible, but it remains an interesting space to watch.

Update: the 2012 investment recap is now up.

Quick, name a social media monitoring tool that can monitor Instagram. Got one yet? Not sure? I found four in seconds. Here's how.

I launched Social Media Analysis in 2009 to move industry news coverage from my personal blog to its own site. A little over a year ago, I added a free directory of social media analysis companies, which continues to grow as I discover more companies in the market. In yesterday's webinar on choosing social media monitoring tools, I realized that the news archive is the better tool for finding specific product capabilities.

SMA's directory has its own search feature, which knows a few tricks, such as finding companies based on a search for their old names. But if you're searching for a feature, the directory is only as good as the descriptions that the vendors have written about themselves (in this challenge, a directory search found one result). For something as specific as covering a particular network, it's not likely to be a big help.

If you write long enough, you build a history
The good news is that the main site also has a search feature and nearly four years' archive of industry news. The weekly roundups of product updates are particularly rich in keywords for the search engine to use. A quick search for "Instagram" revealed four monitoring tools that have announced Instagram coverage.

Even industry observers who make a point of keeping up with the tools market can't remember every detail of what 400+ companies are doing. Is SMA on your go-to list of resources for keeping up with the social media tools market?

Vendors, have you checked your company's entry recently? Is it complete and up to date? Does it contain the right keywords for searchers?

I sometimes summarize the opportunity of social media analysis as using computers to "read the Internet." It's not an original idea, but it is one we still haven't mastered. I've seen many tools that find relevant content and apply some level of automated analysis, but we're not about to replace the analyst. One simple question I've started to think about is, "then what happens?"

The SocialSpook 9000 reads millions of blog posts, Facebook updates, and tweets every second. It finds every relevant mention in your space, extracting the facts, opinions, and needs that you're looking for. Its sentiment analysis engine provides 120% accuracy in 38 languages, and its graphics are so well designed that whole new awards contests have been created for it to win.

In 2007, I pointed out the need to link social media monitoring to customer service, because most of the problems that people were seeing as PR problems started with unhappy customers. Since 2010, I've been thinking about another application: blending social media data with other publicly available sources to create an automated view of what's happening in the world. It turns out to be a big challenge.

My own private news channel… or command center?
We can take this in several directions. At the low end, applications such as Flipboard generate personalized media based on activity in the user's social media accounts and selected topics or sources. In the middle, we might have a more dynamic version of the social media dashboard running in the conference room or reception area. It's the web-powered news channel that always shows something you might care about.

At the high end, we're looking at a valuable—but noisy and sometimes misleading—source of crowdsourced information about events in near-real time. The obvious applications are in government: national security, law enforcement, emergency management, and disaster response agencies are looking for fast and accurate information from social media sources. I see value in corporate applications, too, for functions like security, risk management, logistics, and business continuity that need information when things happen. Preferably without hiring an army of analysts to look at dashboards on the quiet days.

Now what happens?
The challenge in using social media for real-time awareness is that the volume level becomes overwhelming just when the information becomes most valuable. Forget looking for the needle in a haystack; this is the needle in the needlestack. Faster than you can read them, more messages arrive, and they're all relevant.

Existing tools generally emphasize either handling messages individually (think customer service or community engagement) or analyzing them in aggregate (think sentiment and leading topics). For this application, we want the system to help analysts deal with the volume without losing the detail, and that's where I started asking about what happens next.

For all the systems that can notice something happening and put it on a screen, I wanted a system that can notice and pay attention. So what would that look like?

Here's an idea (click to enlarge):

The inputs to this system can go beyond social media content; depending on your application, it might pick up data about natural disasters, weather, or market data. It might incorporate traditional news media, commercial intelligence services, or internal data. Its models will reflect the needs of its users, so a system that looks for, say, transportation-related incidents could be quite different from one looking for damage reports in weather emergencies.

This has a lot of moving parts, and it builds on what others have already built. The central idea is to go beyond the dashboard and think about how the system can relieve analysts of some of the burden of reading the alert queue. Step one is to consider what an analyst does with that information and how a computer could mimic that.

I'm sharing some of the frameworks that have been hiding on my whiteboard. Want the long version? Email me.