[okfn-help] iptables on us1 blocking Googlebot

Wed Apr 6 00:31:38 BST 2011

On 29 March 2011 10:41, Chris Mear <chrismear at gmail.com> wrote:
> On 21 Mar 2011, at 20:21, Chris Mear <chrismear at gmail.com> wrote:
>
>> We (One Click Orgs) have been trying to figure out for the last few
>> days why Google wasn't listing anything from our site
>> http://www.oneclickorgs.com/ in its results.
>>
>> I had a look on the server we're on (us1.okfn.org) and it seems that
>> there's a couple of iptables rules in the INPUT chain that are
>> blocking 66.249.65.0/24 and 66.249.0.0/16, which (currently) include
>> (some of?) Google's web crawlers (e.g. 66.249.65.1). Hence nothing's
>> getting indexed.
>>
>> Is this intentional?
>
> Just checking again if anyone knows why this firewall rule is in place. Is there another list (sysadmins?) I should ping as well?
>
> If there are no objections within the next week, I will adjust the iptables rules to carve out access for the Googlebot servers to us1.

I have now removed two iptables rules from us1 that were blocking
connections from 66.249.0.0/16, and Googlebot is now able to crawl our
website.

Investigating the remaining rules, it seems they correspond to
Microsoft and Yahoo!'s crawlers. I have left these in place for now.

Chris