[openspending-dev] Fwd: [Webmaster Tools] https://openspending.org/: Googlebot can't access your site

Tue May 20 10:26:21 UTC 2014

Hmmm - its happening again ...

---------- Forwarded message ----------
From: <wmt-noreply at google.com>
Date: 19 May 2014 23:07
Subject: [Webmaster Tools] https://openspending.org/: Googlebot can't
access your site
To: rufus.pollock at okfn.org

 [image: Google Logo]
https://openspending.org/: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 1781 errors while attempting
to access your robots.txt. To ensure that we didn't crawl any pages listed
in that file, we postponed our crawl. Your site's overall robots.txt error
rate is 73.9%.

You can see more details about these errors in Webmaster
Tools<https://www.google.com/webmasters/tools/crawl-errors?siteUrl=https://openspending.org/&utm_source=wnc_94051&utm_term=link_2&utm_content=uns_af2bc68c4a000000&utm_campaign=t_1400482798655000&utm_medium=email#t1=2>
.

------------------------------

 *Recommended action*
If the site error rate is 100%:

   - Using a web browser, attempt to access
   https://openspending.org/robots.txt. If you are able to access it from
   your browser, then your site may be configured to deny access to googlebot.
   Check the configuration of your firewall and site to ensure that you are
   not denying access to googlebot.
   - If your robots.txt is a static page, verify that your web service has
   proper permissions to access the file.
   - If your robots.txt is dynamically generated, verify that the scripts
   that generate the robots.txt are properly configured and have permission to
   run. Check the logs for your website to see if your scripts are failing,
   and if so attempt to diagnose the cause of the failure.

If the site error rate is less than 100%:

   - Using Webmaster
Tools<https://www.google.com/webmasters/tools/crawl-errors?siteUrl=https://openspending.org/&utm_source=wnc_94051&utm_term=link_4&utm_content=uns_af2bc68c4a000000&utm_campaign=t_1400482798655000&utm_medium=email#t1=2>,
   find a day with a high error rate and examine the logs for your web server
   for that day. Look for errors accessing robots.txt in the logs for that day
   and fix the causes of those errors.
   - The most likely explanation is that your site is overloaded. Contact
   your hosting provider and discuss reconfiguring your web server or adding
   more resources to your website.
   - If your site redirects to another hostname, another possible
   explanation is that a URL on your site is redirecting to a hostname whose
   serving of its robots.txt file is exhibiting one or more of these issues.

 After you think you've fixed the problem, use Fetch as
Google<https://www.google.com/webmasters/tools/googlebot-fetch?hl=en-GB&siteUrl=https://openspending.org/&utm_source=wnc_94051&utm_term=link_4&utm_content=uns_af2bc68c4a000000&utm_campaign=t_1400482798655000&utm_medium=email>to
fetch
https://openspending.org/robots.txt to verify that Googlebot can properly
access your site.

Learn more in our Help
Center<http://support.google.com/webmasters/bin/answer.py?answer=2409682&hl=en-GB&utm_source=wnc_94051&utm_term=link_4&utm_content=uns_af2bc68c4a000000&utm_campaign=t_1400482798655000&utm_medium=email>.

------------------------------
 Got feedback? Leave it
here<http://productforums.google.com/forum/#!categories/webmasters/webmaster-tools>.
Be sure to include this message ID: [WMT-94051]
*Google Inc.* 1600 Amphitheatre Parkway Mountain View, CA 94043 |
Unsubscribe<https://www.google.com/webmasters/tools/preferences?hl=en-GB&utm_medium=email>.

-- 

*Rufus PollockFounder and President | skype: rufuspollock | @rufuspollock
<https://twitter.com/rufuspollock>Open Knowledge <http://okfn.org/> - see
how data can change the world**http://okfn.org/ <http://okfn.org/> | @okfn
<http://twitter.com/OKFN> | Open Knowledge on Facebook
<https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>*

The Open Knowledge Foundation is a not-for-profit organisation.  It is
incorporated in England & Wales as a company limited by guarantee, with
company number 05133759.  VAT Registration № GB 984404989. Registered
office address: Open Knowledge Foundation, St John’s Innovation Centre,
Cowley Road, Cambridge, CB4 0WS, UK.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/openspending-dev/attachments/20140520/032f0011/attachment.html>