[okfn-labs] csv/xls files for ONS/weather/etc data

Fri Feb 15 12:51:45 UTC 2013

Perhaps querying data.gov.uk/api will have better results, but it
appears to be giving 503 right now.

$ python ckan.py http://data.gov.uk/api 'health' ''
init: http://data.gov.uk/api
tag= health
types= ['CSV', 'XLS']
Traceback (most recent call last):
  File "ckan.py", line 57, in <module>
    write_pkgs(ckan,tag)
  File "ckan.py", line 33, in write_pkgs
    for resource in get_pkg_tag(ckan,tag):
  File "ckan.py", line 9, in get_pkg_tag
    pkgs = ckan.tag_entity_get(tag)
  File "/usr/local/lib/python2.7/dist-packages/ckanclient/__init__.py",
line 367, in tag_entity_get
    self.open_url(url, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/ckanclient/__init__.py",
line 233, in open_url
    raise CkanApiError(self.last_message)
ckanclient.CkanApiError:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>503 Service Unavailable</title>
  </head>
  <body>
    <h1>Error 503 Service Unavailable</h1>
    <p>Service Unavailable</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 1218306508</p>
    <hr>
    <address>
       <a href="http://www.varnish-cache.org/">Varnish cache server</a>
    </address>
  </body>
</html>

On 15 February 2013 12:31, Andy Twigg <andy.twigg at gmail.com> wrote:
> Hi all,
>
> I'm new to the OKFN list, but am working on a project that wants to
> use lots of public data. I wrote a simple script to go to a CKAN repo
> (right now datahub.io) and make a list of all the CSV/XLS resources
> with a given tag - https://github.com/andytwigg/ckan-tools
>
> I would like to initially get weather, ONS, crime and other simple
> indicator statistics for the UK, but that appears optimistic:
>
> $ python ckan.py 'weather' $api_key
> $ cat ckan-weather.json
> {
>     "d46c09e9-3140-4fec-b297-d50383ba80fe": "http://",
>     "89240a28-10ef-4760-8d93-ab53729b1623":
> "http://scraperwiki.com/scrapers/export/indiawaterportal-temperature/",
>     "0ec0bc23-165a-4292-bd55-41669df3bb26":
> "http://cliflo.niwa.co.nz/pls/niwp/wgenf.genform1"
> }
>
> This contains no CSV/XLS data.
>
> even worse:
>
> $ python ckan.py 'uk' $api_key
> $ cat ckan-uk.json
> {
>     "e1be0967-742c-48c7-8e0e-f11691ed3b90":
> "http://www.communities.gov.uk/documents/housing/xls/1769942.xls",
>     "a471aedd-43e0-4be1-9fff-abc2dc605e1a":
> "http://www.culture.gov.uk/images/research/NI_9-10-11_Baseline_data_-_Revised_July_2009.xls",
>     "b1354346-bbb7-4bd9-b945-dbcebdf2a93e":
> "http://www.defra.gov.uk/environment/business/reporting/conversion-factors.htm",
>     "092971cb-8675-43f9-8c1f-ad4b9632cc67":
> "http://www.dft.gov.uk/NaPTAN/snapshot/NaPTANcsv.zip",
>     "65ea9770-5f6f-4ff0-83ef-88189080434f":
> "http://www.openeconomics.net/store/uk-house-prices/data",
>     "5daf0b44-e2f1-4b18-b726-bd9a10d468b9":
> "http://www.opsi.gov.uk/advice/poi/edubase300608.zip",
>     "45aad6f2-8d81-4c42-a323-3df960883a5f":
> "https://docs.google.com/leaf?id=0B-mY7K7wzvaSNzNlOWViNjMtNjU0YS00NTMyLWI4ZDUtMzNhMjk0YWVmZTc5&hl=en_GB",
>     "79d4aca8-3bbf-4491-b137-8d635f2d5ea2":
> "https://docs.google.com/spreadsheet/pub?hl=en&hl=en&key=0AoGtQrY9JEACdF9MN2JDUXgwSmJZN3lRQTZCWEVnSHc&single=true&gid=1&output=csv",
>     "7df7b65e-807c-4500-a052-1ff80e3911d9":
> "http://mk.ucant.org/info/data/ukgov-25k-spending.csv",
>     "c67639b1-0acf-4d6b-bb27-38ac85c48961":
> "http://www.lloydsbankinggroup.com/media/excel/2010/010710historicdata.xls"
> }
>
> Have I missed something?
>
> Thanks,
> Andy
>
>
> --
> Dr Andy Twigg
> Junior Research Fellow, St Johns College, Oxford
> Room 351, Department of Computer Science
> http://www.cs.ox.ac.uk/people/andy.twigg/
> andy.twigg at cs.ox.ac.uk | +447799647538

--
Dr Andy Twigg
Junior Research Fellow, St Johns College, Oxford
Room 351, Department of Computer Science
http://www.cs.ox.ac.uk/people/andy.twigg/
andy.twigg at cs.ox.ac.uk | +447799647538