Use Tailrank's blog spider - Spinn3r

| 4 Comments

I'll post about my developing model of the social media analysis value chain soon, but today, the first link in the chain—content aggregation—is making news. Tailrank launched Spinn3r, a blog aggregator that anyone can use (via Matt Hurst).

Here's the short version from the Spinn3r blog:

Spinn3r is a web service that companies can use to index the blogosphere instead of having to write their own spider.

Instead of spending months designing a scalable backend infrastructure and fighting spam you can just start using our spider tomorrow.

Spinn3r is quicker and probably cheaper than building and running your own spider, but it's not free. There's a monthly license based on the scope of your search, with discounts for non-profit, research, and education.

From talking with companies who are doing their own data collection, I was about ready to start a pool on when people think spider and crawler traffic will become a majority of Internet traffic (kidding, people, really). If the aggregators do a good enough job, maybe the analysts won't feel the need to compete on aggregation.

Tags:


4 Comments

Nathan -

You say: "From talking with companies who are doing their own data collection, I was about ready to start a pool on when people think spider and crawler traffic will become a majority of Internet traffic (kidding, people, really)."

Can you point us to any published data that describes what volume of web (or blog) traffic is accounted for by bots, spiders, and indexres?

Whenever I scan my logs I am always amazed at how large a proportion of visits are accounted for by these necessary visitors, so I didn't think you were kidding!

Dennis McDonald
Alexandria, Virginia, USA
http://www.ddmcd.com

I was kidding, but apparently I was kidding about something that's already happened. Sort of ruins the joke, doesn't it?

I don't have the data to support it, but you're not the only one who sees a lot of bot traffic. I'm asking a couple of search/analytics experts if they know of data and will pass along what I learn.

It really depends on the site, with popular sites and blogs being crawled more often than others. I've seen stats anywhere from 7-40% of all site traffic comes from spiders/crawlers.

Thanks, Andy. I don't suppose all that spider traffic converts, does it?

Hey, another idea for Josh's April Fools list!

PayPerCrawl (paypercrawl.com)
Web spider that reads your ads and buys from your affiliate links as it indexes your site!

Comments are now closed for this entry.


About Nathan Gilliatt

  • ng.jpg
  • Voracious learner and explorer. Analyst tracking technologies and markets in intelligence, analytics and social media. Studying complexity and futures.
  • Principal, Social Target

Subscribe

  • Subscribe by email