Language Support in Social Media Analysis

Translator boothsDoes your social media program include foreign language requirements? Even if your company does business in only one country, you might need multiple languages. The question is, how much capability do you need to check off the language box?

An email from a vendor contact in Tokyo reminded me of a conversation at the Tech@State conference a few weeks ago. We were talking about monitoring Arabic-language social media, and someone pointed out that their analysts know Arabic. They don't need translation; they just need to collect the content for their analysts to read.

It's an important distinction, and it led us into a conversation about what a monitoring platform needs to do to support different types of users. The short answer is, language capability is more than a one-box checklist item. You have to know your needs in order to evaluate tools.

International support in the software
Let's start with the scenario from that initial conversation: an organization is looking for a software platform for monitoring or analyzing social media content in a specific set of languages. Here are some tool capabilities that might back up a claim about language support:

  1. Find content written in a language.
    Theoretically, all you need is support for the required character set, search terms in the desired language, and a broad range of sources. In practice, it's harder to collect content in some countries and languages than others. Ask about source coverage in the countries you need to include.

  2. Translate foreign-language content.
    In the age of Google Translate and other machine-translation programs, it's easy to add a translate button to a tool. If your needs are simple, machine translation could be good enough.

  3. Filter content by language.
    The most basic level of language support involves identifying the language used in a text. Based on some of my testing, that's harder than it looks. Tools that can identify source languages usually offer filters based on the language, which is useful for directing items to analysts who can read them, as well as for analysis of content by language.

  4. Apply text analytics to content in the language.
    Adding more languages to the analytics engine of a social media platform is hard work. I've heard from several sources that adding sentiment analysis in another language, for example, is equivalent to starting over. If you want your tool to do text analytics in a specific language—sentiment, topics, entity extraction, and the rest—ask specifically if those features are supported in the languages you need to analyze.

  5. Provide a user interface in a language.
    So far, this has all been about the content. If you're working with native-speaker analysts, though, you may also want to support them with a user interface in their language. I've talked with people at companies that monitor social media in multiple languages using teams in multiple countries. Giving them a UI in their own language(s) is a nice touch, and one that probably pays off in increased productivity.
International support in services
Now, let's look at the other side of the business: the services market. One easy way to add coverage of additional markets is to send the work to an agency that has those capabilities. The first question is, can they support the languages you need? The follow-up question is, how do they do it?

  1. Multilingual analysts
    Is it adequate to have an analyst who knows the language? Depending on your circumstances, it could be.

  2. Native language analysts
    Anyone who's studied a foreign language knows that it's easier to learn as a small child. Native fluency makes the analyst more likely to catch subtleties that a non-native speaker might miss.

  3. Native analysts located in foreign market
    If your native-fluent analysts are current residents of the foreign country of interest, they may be better attuned to current events and cultural trends than their peers working in another country.

  4. Vendor based in foreign market
    Social media analysis firms are virtually everywhere (try searching a country name in the directory). You can find native analysts who work abroad for international firms, and you can find them working for smaller firms based in their country. Working with foreign vendors adds complexity, but it could be the right answer in some circumstances.
It's easy to make up a list of languages and mark them yes or no. When I did my first report on companies in social media analysis in 2007, I didn't go much farther than that (I did ask about native fluency among analysts). If you're building a capability with international scope, be clear about the level of language support you need, and you'll be a big step closer to finding the right partners for your program.

I've been thinking lately about nuances in product requirements. More to follow.

Photo by David Weekly.


About Nathan Gilliatt

  • ng.jpg
  • Voracious learner and explorer. Analyst tracking technologies and markets in intelligence, analytics and social media. Studying complexity and futures.
  • Principal, Social Target

Subscribe