AdSense as Route to Google Index + Google's Jugular?
Jennifer Slegg's been trying for a long time to prove that the AdSense bot's index finds its way into the main Google index. On her Jenstar blog she says she's "had several discussions with Matt Cutts over the past few years about this issue, and I have always been assured that they are completely separate and they are always careful the two never cross contaminate each other."

Greg Boser caught Google's index cross contamination in the act though: "We had setup some 301's for Googlebot, but had neglected to redirect the AdSense bot. The end result was a whole bunch of duplicate content due to the fact that we were serving the AdSense bot the old url, and Googlebot the new one. Both were getting indexed and added to the cache."

I spoke with MSI's Jon Revill about the issue, and he concluded, based on the evidence and the desire to avoid duplicate content issues, that publishers with session ids should be sure to include the Mediapartner bot in the user agent detection list. And be sure to 301 the Mediapartner bot as well.

Jon's going to be digging in to some logfile data we have here at MSI to explore what's happening and why.

Slegg concurs with Jon's advice regarding bot blocking: "This could have severe consequences to webmasters, such as Greg who suddenly had a duplicate content issue to clean up. Webmasters usually wouldn't think to include the mediabot in any special headers or robots.txt instructions they have for the regular googlebot."

On a related note, AdSense is, according to Greg Linden, one way that MSN could begin to put the hurt on. "AdSense is now about half of Google's revenue and their future growth. Microsoft should strangle Google's air supply, their revenue stream."

Check out his recent post, Kill Google, Vol. 3.

And be sure to keep reading Matt Cutt's blog. The Boston PubCon's coming up soon and I suspect he'll hear something about this there ;)

MSI's Mike Grehan and Adam Schultz will be rocking PubCon too. Let me know and I'll put you in touch with either or both. 919-433-3139.

Also check out Google's Mediapartner Bot Index Creep Indicate AdSense Publisher SERPs Advantage?

UPDATE: Matt Cutts confirms AdSense media bot in natural search index
Garrett French - MarketSmart Interactive | permalink | +marktd |