Recently in Social media analysis Category

What can you learn online? How about where someone is? Or where they live, where they work, where they hang out… One of the interesting ways to segment social media data is by the contributor's location, but it's a rare feature in social media analysis platforms because of the difficulty of doing it well. More than they realize, though, people are publishing their locations.

I've heard of two main methods to assign locations to social media sources. The easier method, which initially sounds more accurate, tracks down the IP network address of the associated computer. Every computer on the Internet has one, and in principle, the address corresponds somewhat to location. But the goal isn't to find a server, it's to segment online contributors by geography.

If addresses matched locations in some mythical past, they're useless for location now. Facebook is Facebook, wherever an individual user is. Blogs are hosted by a few big players; even with private domains, there's no guarantee that the web host is anywhere near the user. This blog, for example, is hosted on a machine in Pennsylvania—a long way from where I'm sitting. I have accounts on lots of social media sites, none of which are here.

So IP addresses might help you locate a computer server, but they're not a reliable indicator of where an individual user of that system may be.

Revealed deomgraphics
The more interesting process, which I've heard from a handful of SMA companies, is to extract information revealed by the user, linking profiles across services to develop a profile of the person. If someone links a blog to accounts on services like Twitter, Facebook, and LinkedIn, then the combined profiles can build a better picture of the person. Location is one of the major components of that picture.

People have lots of opportunities to announce their location in social media, especially in all those member profiles we fill out. The location field in Twitter might be misleading (remember all the people who changed their location to Tehran in a show of support last year?), but if it agrees with Facebook, Linkedin, or the About Me page on the blog, you have a location.

That's without getting into location-based services like Foursquare. Everyone using those is building a personal tracking database on purpose.

Are you uncomfortable yet? At least this is all based on information that people shared intentionally—so far.

Oops, too much information
The New York Times has an article today, Web Photos That Reveal Secrets, Like Where You Live, which discusses the location metadata attached to digital photographs now. Sarah Perez wrote on the same topic a few weeks ago (Researchers Warn of Geotagging Dangers - Are You Concerned?). Cyberstalking, meet cybercasing: how to reveal your home address on Craigslist.

Both articles emphasize the privacy concerns, as they should. In aggregate, the data creates what Marshall Kirkpatrick calls your new superpower; applied to individuals, it's just creepy.

So, how much location information do you want? Where's the line between constructive location and demographics data and creepy/dangerous? Finally, whose ethics apply to analyzing this data?

Photo by Silver Smith.

"Build your own listening tool" has been a popular topic, with suggestions usually building on free combinations of search feeds and RSS tricks. Twitter, especially, has inspired a whole constellation of free tools. But "build your own" has a deep end of the pool, too. Whether you're building a customized tool for specific, internal requirements or realizing your vision of the perfect entrant to an overcrowded market, building your own tool involves a series of build-or-buy decisions, starting with where you will get your source data.

There's a list below, but first, some background.

I first wrote about the building blocks of social media analysis in 2008. The short version is this: any system for monitoring or measuring social media has three basic components: data collection, analytics, and application. The differences are in the details, and each component can be the subject of its own build-or-buy decision. If your goal is to beat the industry standard in a particular area, you build, but if industry-standard is good enough, you don't have to.

At the time of the original post, all of the components were available separately for companies who were building their own systems, either for their own use or for commercial product development. Now, more options are in the market, especially in data collection. Lots of search engines offer RSS feeds. This is something else: services that aggregate social media data from multiple sources for business or commercial users.

More than just aggregation
The data collection step is about more than mashing together multiple search feeds. For the professional-strength aggregator, the finished product—like paid products in other categories—does something the free tools don't offer.

• Sources
  Social media comes in lots of flavors, and aggregators need to keep up with the introduction of new services. Commercial aggregators can also deliver content that simply isn't available without a subscription, such as full feeds from traditional media.

• Filtering
  Once you fill the pipe with incoming content, it's time to screen out the junk. Removing duplicate items is a start; removing near-duplicates (such as syndicated content or press releases) helps, too. Spam removal is a big deal.

  Another kind of filtering is prescreening the content for relevance to the customer. How that works is part of the aggregator's secret sauce and will differ by provider.

• Metadata tagging
  We spend a lot of time thinking about analyzing the unstructured text in social media, but most of the content also has structured data around it (such as the source, publication date, and number of comments). Aggregators can also pull information about posts from third-party sites to complete the picture.

• Speed
  Oh, and do all of the above quickly, please. Financial applications go to extremes to reduce latency (the lag between when content is posted and when it shows up in the aggregator), but it's a factor in less demanding environments, too. If you're monitoring Twitter, for example, you need to know in seconds or minutes, or you'll be too late to respond in that near-real-time environment.

The list
Oh, yeah, this is a list post. Companies who offer social media content aggregation as a service:

• 80legs
  
• Collecta - for public apps only
  
• Collective Intellect
  
• Context Voice (UberVU)
  
• CyberAlert
  
• Dow Jones
  
• Effyis (Boardreader)
  
• Extractiv
  
• Gnip
  
• Moreover
  
• PostRank
  
• SocialMention
  
• Spinn3r
  
• Superfeedr
  
• Viralheat
  
• Yahoo

Photo by identity chris is.

The aquisitions of Scout Labs, Biz360, and Filtrbox so far this year have people talking about what else is coming. Actually, I count seven acquisitions so far this year—apparently we need a score card.

If you weren't aware, I post industry news at Social Media Analysis. If you care about what the companies in SMA are doing, you should read it.

The M&A list is at SMA, too: Acquisitions in Social Media Analysis.

I've pointed out some of the elements of the learning curve for social media analysts. In the middle of looking at almost 30 social media analysis platforms for my recent report, I realized that the software itself isn't the main challenge—developing the analytical mindset to know what to do with the tool is. The question is, how much of that mindset can be taught? How do we teach people to ask penetrating questions using a simple set of analytical tools?

How's your logic?
Here's an example of the challenge. Most social media analysis tools use keyword searches to define topics or to segment the data with subtopics. The query typically takes one of three forms: a simple search, a Boolean query, or an advanced search that simplifies the process of building the query. (Boolean logic isn't the only technique used to define topics, but other methods are more complex, and the companies that use them set them up for their clients.)

Search using Boolean logic seems simple. You use operators like AND, OR, and NOT to include or exclude keywords from your results. Some tools let you get fancy with proximity operators (x within n words of y), and you can nest your statements for finer control. But many of us think we understand how it works.

So it could be a bit of a shock to see the queries presented by Integrasco's Aleksander Stensby at Monitoring Social Media Bootcamp last week. This little one finds the telephone company Orange in English-language content:

(Orange OR subject:Orange -subject:light -light -"Clockwork Orange" -subject:"Clockwork Orange" -"orange box" -subject:"orange box" -juice -subject:juice -fruit -subject:fruit -peel -subject:peel -"Orange Wednesday" -subject:"Orange Wednesday" -"orange county" -subject:"orange county" -"clock work orange" -subject:"clock work orange" -"orange ink" -subject:"orange ink" -"bright orange" -subject:"bright orange" -"dark orange" -subject:"dark orange" -"light orange" -subject:"light orange" -("color orange"~3) -subject:("color orange"~3) - ("style orange"~3) -subject:("style orange"~3)) AND ( (SMS OR MMS OR HDSPA OR "Mobile Phone" OR GSM OR GPRS OR 3G OR SIM OR handset OR "Sony Ericsson" OR Nokia OR HTC OR Motorola OR BlackBerry OR iPhone OR PAYG OR "pay-as-you-go" OR "Network Provider" OR UMTS OR WAP OR PDA OR "PAC Code" OR Cellphone OR OFCOM OR phones4u OR voda OR vodafone OR tmobile OR tmob OR "T-mobile" OR T-Mob) OR subject:(SMS OR MMS OR HDSPA OR "Mobile Phone" OR GSM OR GPRS OR 3G OR SIM OR handset OR "Sony Ericsson" OR Nokia OR HTC OR Motorola OR BlackBerry OR iPhone OR PAYG OR "pay-as-you-go" OR "Network Provider" OR UMTS OR WAP OR PDA OR "PAC Code" OR Cellphone OR OFCOM OR phones4u OR voda OR vodafone OR tmobile OR tmob OR "T-mobile" OR T-Mob) )

He showed another one, about nine times as long, that finds discussions in multiple languages of the form factor of a particular mobile phone. You can see that endless query in Aleksander's presentation.

So, yeah, we know Boolean logic, but wow.

It's not difficult, just hard
These intensely focused queries illustrate the difference between the two learning curves. A query like this could be pasted into many—maybe most—of the available tools for social media analysis. Working out the nested Boolean logic is the trick.

Eric Garland puts a competitive spin on things with this note from a discussion of the future of intelligence at GWU:

Asymmetry of analysis will be more important than asymmetry of information—it’s not who collects the most data, but who is the best at deriving insights who will be most effective.

The question is, how easily can we develop the right combination of logic, curiosity, and perseverance in those who would analyze social media? Is it teachable, and how much of it depends on existing inclinations in future analysts? Or is there really, as someone at MSMBC suggested, a business opportunity in crafting complex queries as a service?

I wonder how many on-topic posts include exclusion keywords: "I called Orange to complain about my phone while eating a piece of fruit."

At last week's Monitoring Social Media Bootcamp, several speakers highlighted the importance of the human analyst. Having several of us stress the continued requirement for a knowledgable user to do something useful with the tool was apparently a surprise and a disappointment to some. So it must have been a shock when some later speakers showed examples of just how complicated this stuff can be. For those who aren't shocked, that complexity will be a lingering source of competitive advantage as the tools continue to improve.

Software still doesn't replace people
Jason Falls describes the lack of strategic services as where social media monitoring services fail, but that's not a fair summary. The market has both software companies and service companies, and clients can decide how much of the work they want to outsource. The software model requires that the user do something useful with the tools. The services model adds highly skilled consultants to the mix, at a higher price.

It's a familiar decision: build or buy. In this case, it refers to analytical skills and the ability to link analysis to business insights. In the journey from raw data to insight and strategy, some of the distance can be traveled by software. The rest requires people, no matter how sophisticated the tools. The available choice is whether the people who complete the conversion of data into insight are employees or consultants.

This response from Sysomos nicely summarizes the importance of both the software and the analyst:

When you think about it, neither side can be successful or effective without the other. The technology is interesting but not useful or valuable without people to do something with it, and people are only able to do a limited amount of monitoring without the assistance of technology to sift through millions of conversations.

All of this is in the pursuit of insight—the monitoring and mining modes of listening. In the monitoring and response mode, the emphasis is less on insight than on action, but similar logic applies. A monitoring platform can automate the discovery of items requiring a response. Although a few automated response products are available, most companies will want a person in the customer-service loop.

I'm surprised that people are surprised by this.

As a person myself, I'm glad that computers haven't replaced us.

I've tried a lot of different platforms for monitoring and measuring social media in the last few months. Somewhere in the middle of the project, I realized that new users of these systems face not one, but two learning curves. The obvious one—learning the software itself—is the easy part.

Get new software, and you'll spend some time figuring out the features and user interface. That much is obvious. Some vendors try harder than others to make their social media analysis platforms easy to use, but none of them are all that hard to understand.

Compared to other software products most of us have learned, these systems don't really have many features. Refuse to be intimidated, and you can figure it out.

Getting analytical
The more challenging learning curve for the new user is in using the tool appropriately—developing the analytical mindset to find meaning in the data. It's easy to fixate on the default charts that virtually every tool provides, but the interesting stuff only starts to happen when you go deeper into the data. On a checklist level, that means developing an understanding of:

• Queries and topics
  Most platforms use Boolean logic to find relevant content for analysis. Some use the same method to define subtopics, such as competitors or industry trends. Know what the topic does—and doesn't—cover.

• Analysis and metrics
  The usual list of available analyses is generally short. but it's important to think about how the numbers are calculated and what they mean in a given context.

• Filters and more queries
  Most platforms allow users to focus on subsets of the data, either by clicking on charts or by selecting filters based on metrics or new queries. Filters and queries can lead to useful information, but it's important to understand what applying a particular set of filters to the data tells you.
  

How many competing products do you have time to evaluate before you need to make a decision? I have some good stuff in my draft folder for next week, but first, let me tell you about the project I just completed. If your company is looking at software options for monitoring or analyzing social media, I can save you a lot of time and effort.

Over the last few months, I've reviewed about 30 companies on the way toward writing a comparison report on 21 social media analysis platforms that are built for workgroup environments. The report, Social Media Analysis Platforms for Workgroups, is now available at the Social Target web site. It has information on all of these companies:

It's possible to get an idea of what's on the market by visiting vendor web sites and reading reviews, but that's not what I do. If you want to know about more than the usual suspects, or if you want answers to questions the vendors don't answer in their marketing materials, it takes a bit more effort—actually, a lot more effort.

Here's my process:

• Invite everyone
  I've been tracking the companies in this space since 2006, and I published my first buyer's guide in 2007. I have a database of well over 200 companies who offer tools or services for social media analysis, and most of them are on my vendor mailing list. When I started the project, I invited everyone to participate. I made a special point to contact companies that I knew should be in it.

• Written RFI
  I sent a 36-question request for information, asking for details that sometimes make the vendors squirm. Want to know about sentiment analysis? I asked about the degree of automation and its granularity (document-level vs. entity-level). I got prices and pricing models. I found out how long it takes to get them up and running (from seconds to weeks). I got details—lots of details.

• Briefings and product demos
  30 companies responded to the RFI, and I took briefings and demos from each, running about 90 minutes each. We talked about their products, their customers, and their businesses. In the process, I learned that some of the companies didn't belong in the report, either because their software was single-user or because their consulting services were an essential piece of the platform. For some reason, the briefings and demos last just as long for the companies that don't end up in the report.

• Live testing
  Most of the companies gave me access to their platforms for hands-on testing as I wrote about them. There's nothing quite like trying to reproduce the cool demonstration to show how much work went into building it. Switching the user interface to a language I can't read was fun, too. I observed that learning the software itself isn't going to be the major challenge for most companies; the challenge is in understanding how to use the data.

Add hundreds of emails and a bazillion hours of writer's block writing (as counted by my junior associate) to get to 60 pages of finished report, and you have the complete process. It's not a project you want to duplicate.

I'll share some of what I learned in the coming weeks, but this post is getting (getting?) too long already. Please take a look at Social Media Analysis Platforms for Workgroups. If your company is actively searching for the right tool(s) for monitoring, measuring, or mining social media, I think you'll find it's worth the investment.

In the past week, both Radian6 and Sentiment Metrics have announced lightweight clients for their social media analysis platforms. (I'm calling it a slim client because thin client means something else.) The new applications are for users who need to work with SMA platforms but who don't need access to all of their features. It's not hard to imagine how this might lead to slim clients tailored to different needs, since the starting point is a dashboard environment that's already built from widgets.

Take a look; two announcements in one week suggests the start of a trend. Call it "SaaS meets client-server." :-)

Radian6: Engagement Console

Sentiment Metrics: SM Live

Both applications run on Adobe Air, like the popular Twitter client, TweetDeck (note the visual similarity with the Radian6 client). The Radian6 Engagement Console arrives in April; SM Live is due at the end of March.

Is it a problem of overpromising/underdelivering, or are people developing these unrealistic expectations on their own? Either way, I'm seeing more examples of people who seem surprised that software doesn't do all of the work in social media analysis. I really don't think this is controversial: regardless of your choice of tool, there's a necessary human contribution to the process.

This notion that the software isn't good enough because it requires a person to do something with it seems to be picking up speed. The first post that really kicked off a conversation was probably Asi Sharabi's. I saw a couple this morning, including one from Mark Schaefer that focuses on the graphics:

I’ve been spending time studying the trends in social media monitoring and have been impressed with the rapid progress. But there is still a lot of noise like this chart that really tells us nothing. The fact is, the most meaningful keyword and sentiment analysis is all still being done MANUALLY.

Building the social media spreadsheet
Think about spreadsheet software. When you first open a new spreadsheet, the software gives you a blank page. In the right hands, the software is a power tool for running financials, forecasting results, analyzing historical data... I've seen some impressive examples, but the most powerful spreadsheet software is useless without someone who knows how to use it. Which, if you think about it, is true of most software.

Social media analysis tools are software; they do some of the work, but to get the most out of them, you need someone who knows how to use them. The more you expect from your tool, the more the user needs to know. Anything that's fully automated either isn't doing much, or it has a lot of human effort baked into it.

Regardless of the tools used, at some point people take over. It may be earlier in the process (manual content analysis) or later (analysis and reporting), but eventually, a person takes what the computer produces and does something with it. All of those agencies that sell services based on the same SMA platforms presumably think this is where they add value.

If you want it done for you...
There is an answer for the company that wants the insights without putting in the effort, of course. Have someone else do it. You can't bypass the requirement for a human analyst, but you don't have to do it yourself. When you're shopping for social media analysis, just be sure to include analyst services among your requirements. That eliminates some of the best-known software companies, but it opens the door to an entirely different set of service providers.

If you want to make spreadsheets, you buy Excel. If you want financial projections, you roll up your sleeves or hire a financial analyst. It's up to you to decide whether that analyst will be an employee or work for an external service provider.

Where's the disconnect? Are unrealistic customer expectations coming from vendor hype, or is it just hope that things will be easy?

I try not to be too obnoxious with my iPhone, but it's hard to avoid being impressed with how it's changed my expectations about mobile computing. I've been wondering when social media analysis apps would start showing up, and now I've found one to play with: iCrossing's new Say What?

Compared to the tools I usually look at, Say What? (iTunes link) is pretty basic. It runs searches across Twitter, Digg, forums and blogs, returning a sampling of the results from each. It's not much, but it might be enough to get a quick sense of what's going on with a topic (especially if it's currently controversial).

Why are people talking about that?
Say What? is best used for getting a clue about a current topic—especially if it's controversial or newsworthy. Looking at results, four per screen, you're looking for someone to provide a hint about what's going on.

Looking up (Rush) Limbaugh, I immediately found mentions of his interest in buying into an NFL franchise. Ford returned items about the latest product recall. But when the company isn't making news, the results are far less interesting—people are apparently having breakfast at Dunkin' Donuts.

From Search to Analysis
I've been looking into web-based collaboration and project management tools for my own company, and as soon as I realized that some of these tools have iPhone apps, that became a requirement. We have so many web-based tools for monitoring and analyzing social media; who's going to be the first to offer a simple dashboard that delivers clients' data to smart phones?