[okfn-labs] csv/xls files for ONS/weather/etc data

Fri Feb 15 12:31:45 UTC 2013

Hi all,

I'm new to the OKFN list, but am working on a project that wants to
use lots of public data. I wrote a simple script to go to a CKAN repo
(right now datahub.io) and make a list of all the CSV/XLS resources
with a given tag - https://github.com/andytwigg/ckan-tools

I would like to initially get weather, ONS, crime and other simple
indicator statistics for the UK, but that appears optimistic:

$ python ckan.py 'weather' $api_key
$ cat ckan-weather.json
{
    "d46c09e9-3140-4fec-b297-d50383ba80fe": "http://",
    "89240a28-10ef-4760-8d93-ab53729b1623":
"http://scraperwiki.com/scrapers/export/indiawaterportal-temperature/",
    "0ec0bc23-165a-4292-bd55-41669df3bb26":
"http://cliflo.niwa.co.nz/pls/niwp/wgenf.genform1"
}

This contains no CSV/XLS data.

even worse:

$ python ckan.py 'uk' $api_key
$ cat ckan-uk.json
{
    "e1be0967-742c-48c7-8e0e-f11691ed3b90":
"http://www.communities.gov.uk/documents/housing/xls/1769942.xls",
    "a471aedd-43e0-4be1-9fff-abc2dc605e1a":
"http://www.culture.gov.uk/images/research/NI_9-10-11_Baseline_data_-_Revised_July_2009.xls",
    "b1354346-bbb7-4bd9-b945-dbcebdf2a93e":
"http://www.defra.gov.uk/environment/business/reporting/conversion-factors.htm",
    "092971cb-8675-43f9-8c1f-ad4b9632cc67":
"http://www.dft.gov.uk/NaPTAN/snapshot/NaPTANcsv.zip",
    "65ea9770-5f6f-4ff0-83ef-88189080434f":
"http://www.openeconomics.net/store/uk-house-prices/data",
    "5daf0b44-e2f1-4b18-b726-bd9a10d468b9":
"http://www.opsi.gov.uk/advice/poi/edubase300608.zip",
    "45aad6f2-8d81-4c42-a323-3df960883a5f":
"https://docs.google.com/leaf?id=0B-mY7K7wzvaSNzNlOWViNjMtNjU0YS00NTMyLWI4ZDUtMzNhMjk0YWVmZTc5&hl=en_GB",
    "79d4aca8-3bbf-4491-b137-8d635f2d5ea2":
"https://docs.google.com/spreadsheet/pub?hl=en&hl=en&key=0AoGtQrY9JEACdF9MN2JDUXgwSmJZN3lRQTZCWEVnSHc&single=true&gid=1&output=csv",
    "7df7b65e-807c-4500-a052-1ff80e3911d9":
"http://mk.ucant.org/info/data/ukgov-25k-spending.csv",
    "c67639b1-0acf-4d6b-bb27-38ac85c48961":
"http://www.lloydsbankinggroup.com/media/excel/2010/010710historicdata.xls"
}

Have I missed something?

Thanks,
Andy

--
Dr Andy Twigg
Junior Research Fellow, St Johns College, Oxford
Room 351, Department of Computer Science
http://www.cs.ox.ac.uk/people/andy.twigg/
andy.twigg at cs.ox.ac.uk | +447799647538