[ckan-discuss] dataproxy timeout

Fri Mar 15 14:39:16 GMT 2013

Hi Rufus,

Sorry, I should have clarified that our preview is potentially only a
stop-gap measure to resolve issues we are having now on our 1.x
installation and the older dataproxy. We also use an older recline as well.
For the 2.0 branch I think a solution that brings the dataproxy away from
appengine would be fantastic, I suspect it wouldn't be a huge task - using
the local archive where it exists gives, obviously, a huge performance
boost.

Using data converters definitely looks like a better path forward in the
longer term. I'm unsure yet exactly when we'll upgrade to the newer recline
etc (I quite like the idea of using the explorer for interacting with the
data), I'll see if I can find out.

I'll add moving datapreview to dataconverters to my list, should then at
least provide a good starting point for a proper service.

Ross.

On Fri, Mar 15, 2013 at 2:18 PM, Rufus Pollock <rufus.pollock at okfn.org>wrote:

> On 15 March 2013 13:45, Ross Jones <ross at servercode.co.uk> wrote:
>
>> Hi Davide,
>>
>> On data.gov.uk we're now using a modified version of the jsonpdataproxy
>> that will serve the same format response from /data/preview/<resource_id>
>> which is working fairly well.  It's modified to run with the local archived
>> version of the resource should you also be running ckanext-archiver.
>>
>
> This is great stuff Ross :-) - see comments below ...
>
> @Davide: if you want the plain dataproxy that runs
> jsonpdataproxy.appspot.com you can get it here
> https://github.com/okfn/dataproxy
>
> I should point out that the main reason jsonpdataproxy is slow is that it
> runs on google app engines free tier (it is after all a free service).
> Responsiveness and performance could no doubt be dramatically improved by
> dedicating more resources to it. Alternatively if you use the datastore you
> no longer need the dataproxy at all!
>
>
>> It requires a change to recline.js, and it might be more useful to modify
>> the current recline backend to accept any url as long as the target
>> understands the 'protocol'.  As we're using an old version of recline (for
>> now) we haven't yet done that.
>>
>> I am currently in the process of modifying it so that it uses
>> github.com/okfn/messytables, and so is dependant on a branch within that.
>>
>
> Strong recommendation: use https://github.com/okfn/dataconverters rather
> than plain messytables. It's easier to use and provides a good wrapper.
> It's what dataproxy now uses.
>
>
>> Now that I've got the disclaimers out of the way, it is at
>> https://github.com/datagovuk/ckanext-datapreview if you want to give it
>> a try. If you have any problems, do let me know (or add tickets to github).
>>
>
> Would it be worth trying to collaborate a bit here - perhaps produce a
> common conversion app (and possibly a common service?)
>
> Rufus
>
>
>> Ross
>>
>>
>>
>> On Fri, Mar 15, 2013 at 1:17 PM, Davide <davide.savazzi at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm configuring 3 new CKAN 1.8 instances. I have an issue about CSV
>>> file preview, at first it didn't work because DataProxy doesn't answer
>>> after 5 seconds:
>>>
>>> Request Error: Backend did not respond after 5 seconds
>>>
>>> To solve this I've edited
>>> ckan/public/scripts/vendor/recline/recline.js and increased the
>>> timeout.
>>>
>>> Is there a way to avoid DataProxy usage or is there a DataProxy
>>> version that I can install on my own server?
>>>
>>> Thanks,
>>> --
>>> ==
>>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>>> more information.
>>> ==
>>>
>>> Davide Savazzi
>>> @svzdvd
>>> Senior Software Engineer
>>>
>>> GeoSolutions S.A.S.
>>> Via Poggio alle Viti 1187
>>> 55054  Massarosa (LU)
>>> Italy
>>> phone: +39 0584 962313
>>> fax:      +39 0584 1660272
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> _______________________________________________
>>> ckan-discuss mailing list
>>> ckan-discuss at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>>>
>>
>>
>> _______________________________________________
>> ckan-discuss mailing list
>> ckan-discuss at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-discuss/attachments/20130315/016b3935/attachment-0001.htm>