[ckan-discuss] API of catalog.data.gov is not usable at all

Wed Jul 24 07:35:55 BST 2013

Hey Dominik,

thanks a lot for answering. As a matter of fact, I did use pagination
(limit and offset) to harvest catalog.data.gov (see my first remark).

Your answer, however, gave me an idea anyway. I set the limit to 50,
because I thought it's still reasonable, didn't work. Now I set it to
1 and it works. Progress is slow, but I am in for the long haul :-)

Thanks,
Konrad

On 23.07.2013 22:56, Dominik Moritz wrote:
> Hey,
> 
> On 23 Jul 2013, at 17:41, Konrad Reiche <konrad.reiche at gmail.com> wrote:
> 
>> Hello everyone,
>>
>> I am happy that catalog.data.gov was launched. Unfortunately the API is
>> not usable at all. Any attempts to harvest the provided datasets fail.
>>
>>
>> 1. Loading the datasets by using pagination with
>> /3/action/current_package_list_with_resources
>>
>> works for the first 200 datasets before I run into timeouts or 503
>> Service Unavailable.
> 
> It's a server setting to avoid Dos. To get the data, use pagination (limit and offset). Docs are at [1].
> 
>>
>>
>> 2. Loading the datasets through /3/action/package_list
>>
>> does not work at all. After 2min I get a 503 Service Unavailable, too.
>>
>> I have already issues a ticket regarding offering the datasets through a
>> static JSON dump.
>>
>> Does anyone see other possibilities to harvest the data or at least use
>> the API in one way or the other?
>>
>> Best regards,
>> Konrad
> 
> [1] http://docs.ckan.org/en/latest/api.html?highlight=current_package_list_with_resources#ckan.logic.action.get.current_package_list_with_resources
> 
>>
>> _______________________________________________
>> ckan-discuss mailing list
>> ckan-discuss at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
> 
> Dominik Moritz
> CKAN developer  |  skype: d.moritz  |  @doobly_doo
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject
> 