[openspending-dev] Fwd: [Webmaster Tools] http://openspending.org/: Googlebot can't access your site

[Nick Stenning]

For the love of all that is holy...

I've now fixed the problem. See http://i.imgur.com/ixtXvIs.png.

Why on earth anyone thought it was appropriate to add a rule into the
s095 firewall that blocked fully 1/256'th of the internet from accessing
OpenSpending is beyond me:

    Chain INPUT (policy ACCEPT)
    num  target     prot opt source               destination
    1    DROP       all  --  66.0.0.0/8           anywhere

Clearly, most of GoogleBot lives in that /8, and has been unable to
access OpenSpending since roughly the 11th of December 2013. The
robots.txt syntax issues are real, but weren't the cause of the problem.

Please, in future, if you have problems with GoogleBot, update your
robots.txt to include a crawl-delay. Do not block an appreciable
fraction of the internet.

-N

-- 
Nick Stenning
Technical Director, Open Knowledge Foundation