Recently I’ve switched over to using CloudFlare, for those unfamiliar it’s a proxy caching service. It caches resources from your site in several data centers. When looking over Google Webmaster Tools, I noticed every page it crawled was a 403 error. Not very conducive to having good search rankings.
After some digging the answer finally appeared on the WordPress.org forums. The Bad Behavior plugin was rejecting Google since the crawler was being proxied thru CloudFlare and had the user-agent of Googlebot but was not in the right IP address range. The solution, disable Bad Behavior. Although the plugin had always worked great in reducing the number of spam request, since CloudFlare increases security by blocking bad requests before ever reaching the server, no reason to have Bad Behavior taking up resources as well.
By Edge Cases & SEO Consequences: Bad Behavior vs. Cloudflare May 22, 2012 - 10:49 am
[…] little digging by our friend led her to this TechNet post by Jeremy Clark. Like bleach and ammonia (never mix them!), we have in this case two useful […]