[ckan-discuss] dataproxy timeout

Fri Mar 15 14:43:09 GMT 2013

All sounds really good - i should also add I'm personally very excited to
see the innovation and work going on here from DGU and others :-)

Rufus

On 15 March 2013 14:39, Ross Jones <ross at servercode.co.uk> wrote:

> Hi Rufus,
>
> Sorry, I should have clarified that our preview is potentially only a
> stop-gap measure to resolve issues we are having now on our 1.x
> installation and the older dataproxy. We also use an older recline as well.
> For the 2.0 branch I think a solution that brings the dataproxy away from
> appengine would be fantastic, I suspect it wouldn't be a huge task - using
> the local archive where it exists gives, obviously, a huge performance
> boost.
>
> Using data converters definitely looks like a better path forward in the
> longer term. I'm unsure yet exactly when we'll upgrade to the newer recline
> etc (I quite like the idea of using the explorer for interacting with the
> data), I'll see if I can find out.
>
> I'll add moving datapreview to dataconverters to my list, should then at
> least provide a good starting point for a proper service.
>
> Ross.
>
>
> On Fri, Mar 15, 2013 at 2:18 PM, Rufus Pollock <rufus.pollock at okfn.org>wrote:
>
>> On 15 March 2013 13:45, Ross Jones <ross at servercode.co.uk> wrote:
>>
>>> Hi Davide,
>>>
>>> On data.gov.uk we're now using a modified version of the jsonpdataproxy
>>> that will serve the same format response from /data/preview/<resource_id>
>>> which is working fairly well.  It's modified to run with the local archived
>>> version of the resource should you also be running ckanext-archiver.
>>>
>>
>> This is great stuff Ross :-) - see comments below ...
>>
>> @Davide: if you want the plain dataproxy that runs
>> jsonpdataproxy.appspot.com you can get it here
>> https://github.com/okfn/dataproxy
>>
>> I should point out that the main reason jsonpdataproxy is slow is that it
>> runs on google app engines free tier (it is after all a free service).
>> Responsiveness and performance could no doubt be dramatically improved by
>> dedicating more resources to it. Alternatively if you use the datastore you
>> no longer need the dataproxy at all!
>>
>>
>>> It requires a change to recline.js, and it might be more useful to
>>> modify the current recline backend to accept any url as long as the target
>>> understands the 'protocol'.  As we're using an old version of recline (for
>>> now) we haven't yet done that.
>>>
>>> I am currently in the process of modifying it so that it uses
>>> github.com/okfn/messytables, and so is dependant on a branch within
>>> that.
>>>
>>
>> Strong recommendation: use https://github.com/okfn/dataconverters rather
>> than plain messytables. It's easier to use and provides a good wrapper.
>> It's what dataproxy now uses.
>>
>>
>>> Now that I've got the disclaimers out of the way, it is at
>>> https://github.com/datagovuk/ckanext-datapreview if you want to give it
>>> a try. If you have any problems, do let me know (or add tickets to github).
>>>
>>
>> Would it be worth trying to collaborate a bit here - perhaps produce a
>> common conversion app (and possibly a common service?)
>>
>> Rufus
>>
>>
>>> Ross
>>>
>>>
>>>
>>> On Fri, Mar 15, 2013 at 1:17 PM, Davide <davide.savazzi at gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm configuring 3 new CKAN 1.8 instances. I have an issue about CSV
>>>> file preview, at first it didn't work because DataProxy doesn't answer
>>>> after 5 seconds:
>>>>
>>>> Request Error: Backend did not respond after 5 seconds
>>>>
>>>> To solve this I've edited
>>>> ckan/public/scripts/vendor/recline/recline.js and increased the
>>>> timeout.
>>>>
>>>> Is there a way to avoid DataProxy usage or is there a DataProxy
>>>> version that I can install on my own server?
>>>>
>>>> Thanks,
>>>> --
>>>> ==
>>>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>>>> more information.
>>>> ==
>>>>
>>>> Davide Savazzi
>>>> @svzdvd
>>>> Senior Software Engineer
>>>>
>>>> GeoSolutions S.A.S.
>>>> Via Poggio alle Viti 1187
>>>> 55054  Massarosa (LU)
>>>> Italy
>>>> phone: +39 0584 962313
>>>> fax:      +39 0584 1660272
>>>>
>>>> http://www.geo-solutions.it
>>>> http://twitter.com/geosolutions_it
>>>>
>>>> _______________________________________________
>>>> ckan-discuss mailing list
>>>> ckan-discuss at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-discuss mailing list
>>> ckan-discuss at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-discuss/attachments/20130315/7972f4c6/attachment.htm>