[okfn-labs] csv/xls files for ONS/weather/etc data

Mon Feb 18 13:44:53 UTC 2013

Gday Andy,

good to see you on-list.

I wasn't aware ckanclient had been deprecated(!). What has replaced it?

Mk

On Fri, Feb 15, 2013 at 2:14 PM, Stefan Wehrmeyer
<stefan.wehrmeyer at okfn.org> wrote:
> Hi Andy,
>
> I believe the ckanclient Python package is deprecated (doesn't mean it shouldn't work with data.gov.uk though).
>
> The basic HTTP Get search request works:
> http://data.gov.uk/api/search/dataset?q=weather
>
> Maybe this would be better suited for the ckan-discuss/ckan-dev lists?
>
> Cheers
> Stefan
>
> On 15.02.2013, at 13:51 , Andy Twigg <andy.twigg at gmail.com> wrote:
>
>> Perhaps querying data.gov.uk/api will have better results, but it
>> appears to be giving 503 right now.
>>
>> $ python ckan.py http://data.gov.uk/api 'health' ''
>> init: http://data.gov.uk/api
>> tag= health
>> types= ['CSV', 'XLS']
>> Traceback (most recent call last):
>>  File "ckan.py", line 57, in <module>
>>    write_pkgs(ckan,tag)
>>  File "ckan.py", line 33, in write_pkgs
>>    for resource in get_pkg_tag(ckan,tag):
>>  File "ckan.py", line 9, in get_pkg_tag
>>    pkgs = ckan.tag_entity_get(tag)
>>  File "/usr/local/lib/python2.7/dist-packages/ckanclient/__init__.py",
>> line 367, in tag_entity_get
>>    self.open_url(url, headers=headers)
>>  File "/usr/local/lib/python2.7/dist-packages/ckanclient/__init__.py",
>> line 233, in open_url
>>    raise CkanApiError(self.last_message)
>> ckanclient.CkanApiError:
>> <?xml version="1.0" encoding="utf-8"?>
>> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
>> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
>> <html>
>>  <head>
>>    <title>503 Service Unavailable</title>
>>  </head>
>>  <body>
>>    <h1>Error 503 Service Unavailable</h1>
>>    <p>Service Unavailable</p>
>>    <h3>Guru Meditation:</h3>
>>    <p>XID: 1218306508</p>
>>    <hr>
>>    <address>
>>       <a href="http://www.varnish-cache.org/">Varnish cache server</a>
>>    </address>
>>  </body>
>> </html>
>>
>> On 15 February 2013 12:31, Andy Twigg <andy.twigg at gmail.com> wrote:
>>> Hi all,
>>>
>>> I'm new to the OKFN list, but am working on a project that wants to
>>> use lots of public data. I wrote a simple script to go to a CKAN repo
>>> (right now datahub.io) and make a list of all the CSV/XLS resources
>>> with a given tag - https://github.com/andytwigg/ckan-tools
>>>
>>> I would like to initially get weather, ONS, crime and other simple
>>> indicator statistics for the UK, but that appears optimistic:
>>>
>>> $ python ckan.py 'weather' $api_key
>>> $ cat ckan-weather.json
>>> {
>>>    "d46c09e9-3140-4fec-b297-d50383ba80fe": "http://",
>>>    "89240a28-10ef-4760-8d93-ab53729b1623":
>>> "http://scraperwiki.com/scrapers/export/indiawaterportal-temperature/",
>>>    "0ec0bc23-165a-4292-bd55-41669df3bb26":
>>> "http://cliflo.niwa.co.nz/pls/niwp/wgenf.genform1"
>>> }
>>>
>>> This contains no CSV/XLS data.
>>>
>>> even worse:
>>>
>>> $ python ckan.py 'uk' $api_key
>>> $ cat ckan-uk.json
>>> {
>>>    "e1be0967-742c-48c7-8e0e-f11691ed3b90":
>>> "http://www.communities.gov.uk/documents/housing/xls/1769942.xls",
>>>    "a471aedd-43e0-4be1-9fff-abc2dc605e1a":
>>> "http://www.culture.gov.uk/images/research/NI_9-10-11_Baseline_data_-_Revised_July_2009.xls",
>>>    "b1354346-bbb7-4bd9-b945-dbcebdf2a93e":
>>> "http://www.defra.gov.uk/environment/business/reporting/conversion-factors.htm",
>>>    "092971cb-8675-43f9-8c1f-ad4b9632cc67":
>>> "http://www.dft.gov.uk/NaPTAN/snapshot/NaPTANcsv.zip",
>>>    "65ea9770-5f6f-4ff0-83ef-88189080434f":
>>> "http://www.openeconomics.net/store/uk-house-prices/data",
>>>    "5daf0b44-e2f1-4b18-b726-bd9a10d468b9":
>>> "http://www.opsi.gov.uk/advice/poi/edubase300608.zip",
>>>    "45aad6f2-8d81-4c42-a323-3df960883a5f":
>>> "https://docs.google.com/leaf?id=0B-mY7K7wzvaSNzNlOWViNjMtNjU0YS00NTMyLWI4ZDUtMzNhMjk0YWVmZTc5&hl=en_GB",
>>>    "79d4aca8-3bbf-4491-b137-8d635f2d5ea2":
>>> "https://docs.google.com/spreadsheet/pub?hl=en&hl=en&key=0AoGtQrY9JEACdF9MN2JDUXgwSmJZN3lRQTZCWEVnSHc&single=true&gid=1&output=csv",
>>>    "7df7b65e-807c-4500-a052-1ff80e3911d9":
>>> "http://mk.ucant.org/info/data/ukgov-25k-spending.csv",
>>>    "c67639b1-0acf-4d6b-bb27-38ac85c48961":
>>> "http://www.lloydsbankinggroup.com/media/excel/2010/010710historicdata.xls"
>>> }
>>>
>>> Have I missed something?
>>>
>>> Thanks,
>>> Andy
>>>
>>>
>>> --
>>> Dr Andy Twigg
>>> Junior Research Fellow, St Johns College, Oxford
>>> Room 351, Department of Computer Science
>>> http://www.cs.ox.ac.uk/people/andy.twigg/
>>> andy.twigg at cs.ox.ac.uk | +447799647538
>>
>>
>>
>> --
>> Dr Andy Twigg
>> Junior Research Fellow, St Johns College, Oxford
>> Room 351, Department of Computer Science
>> http://www.cs.ox.ac.uk/people/andy.twigg/
>> andy.twigg at cs.ox.ac.uk | +447799647538
>>
>> _______________________________________________
>> okfn-labs mailing list
>> okfn-labs at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/okfn-labs
>> Unsubscribe: http://lists.okfn.org/mailman/options/okfn-labs
>
>
> _______________________________________________
> okfn-labs mailing list
> okfn-labs at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/okfn-labs
> Unsubscribe: http://lists.okfn.org/mailman/options/okfn-labs