[ckan-discuss] API of catalog.data.gov is not usable at all

Mon Jul 29 18:08:28 BST 2013

Hi,

Using the search endpoint (/3/action/package_search) [1] will be
significantly faster, as will get data directly from the search index.
You can use the "rows" and "start" parameters to paginate.

Adrià

[1] http://docs.ckan.org/en/latest/api.html#ckan.logic.action.get.package_search

On 24 July 2013 07:35, Konrad Reiche <konrad.reiche at gmail.com> wrote:
> Hey Dominik,
>
> thanks a lot for answering. As a matter of fact, I did use pagination
> (limit and offset) to harvest catalog.data.gov (see my first remark).
>
> Your answer, however, gave me an idea anyway. I set the limit to 50,
> because I thought it's still reasonable, didn't work. Now I set it to
> 1 and it works. Progress is slow, but I am in for the long haul :-)
>
> Thanks,
> Konrad
>
> On 23.07.2013 22:56, Dominik Moritz wrote:
>> Hey,
>>
>> On 23 Jul 2013, at 17:41, Konrad Reiche <konrad.reiche at gmail.com> wrote:
>>
>>> Hello everyone,
>>>
>>> I am happy that catalog.data.gov was launched. Unfortunately the API is
>>> not usable at all. Any attempts to harvest the provided datasets fail.
>>>
>>>
>>> 1. Loading the datasets by using pagination with
>>> /3/action/current_package_list_with_resources
>>>
>>> works for the first 200 datasets before I run into timeouts or 503
>>> Service Unavailable.
>>
>> It's a server setting to avoid Dos. To get the data, use pagination (limit and offset). Docs are at [1].
>>
>>>
>>>
>>> 2. Loading the datasets through /3/action/package_list
>>>
>>> does not work at all. After 2min I get a 503 Service Unavailable, too.
>>>
>>> I have already issues a ticket regarding offering the datasets through a
>>> static JSON dump.
>>>
>>> Does anyone see other possibilities to harvest the data or at least use
>>> the API in one way or the other?
>>>
>>> Best regards,
>>> Konrad
>>
>> [1] http://docs.ckan.org/en/latest/api.html?highlight=current_package_list_with_resources#ckan.logic.action.get.current_package_list_with_resources
>>
>>>
>>> _______________________________________________
>>> ckan-discuss mailing list
>>> ckan-discuss at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>>
>> Dominik Moritz
>> CKAN developer  |  skype: d.moritz  |  @doobly_doo
>> The Open Knowledge Foundation
>> Empowering through Open Knowledge
>> http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject
>>
>
>
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss