[ckan-dev] Future, flask, breaking things, funding.

Mon Sep 14 12:57:59 UTC 2015

Right now CKAN is great for static sources of data, which is really all
that existed from government sources when CKAN was first written.

Now that government is (slowly) catching on, more stream, API, and even
real-time data is being published. CKAN doesn't do a great job here. The
biggest obstacle to creating nice extensions to CKAN for non-file data is
that Pylons is still firmly stuck within the HTTP request-response
lifecycle.

This worked well for CRUD apps, but now is really showing it's limitations.
It's hard to do anything in CKAN that doesn't take place within the context
of a user's HTTP request. If you want to do some extra data processing on
the side, you have to use celery queues or worse, cron. Worse yet, some
people do try to put extra processing inside the request-response
lifecycle, causing problems.

Even core CKAN is guilty of this. For example, CKAN will call datapusher to
send upload jobs and retrieve job results, and those requests to datapusher
happen while the user is waiting for the request to return. This is kind of
terrible. Not even because somebody did it this way, but because CKAN
doesn't give you a sane alternative to do it properly.

Porting CKAN to flask is no small feat, so let's make sure we do it right.
Now that we're not using CKAN to just host static files anymore, we need to
have better, built-in async support in CKAN. Perhaps this means moving to
Python 3 where we'll have asyncio (and hopefully a future version of flask
will work well with it). Other frameworks, like tornado, are also quite
lightweight and support this out of the box for python 2.x.

- Denis

On Mon, Sep 14, 2015 at 3:56 AM, Angelos Tzotsos <gcpp.kalxas at gmail.com>
wrote:

> On 09/14/2015 10:24 AM, Ross Jones wrote:
>
>> Hi,
>>
>> I’ve recently been playing about with implementing parts of CKAN in Flask
>> side-by-side with the current Pylons implementation. I’m doing it like this
>> so that it isn’t immediately obvious that there’s a migration happening
>> towards using Flask (aka nothing breaks).  I don’t think this branch should
>> ever be merged, it’s more exploratory but it has raised some questions that
>> I think it would be good to discuss.
>>
>> WARNING:anecdata
>> It’s pretty clear that the vast majority of people asked would like to
>> move to Flask as a replacement for some layers of the system (leaving
>> things like logic and plugins alone).
>> ENDWARNING
>>
>> We’ve discussed at the tech-team meetings, but I think a longer, more
>> accessible conversation would be beneficial.
>>
>> 1. What version of CKAN should be targeted? Common sense suggests 3.0,
>> but that being the case, exactly how far can we go in breaking some
>> backward compatibility?  This isn’t really a technical question - would be
>> good to hear what the community would accept …
>>
>> 2. Does it *really* need to be side-by-side?  Running Flask and Pylons
>> side-by-side means staying on Python 2 for another few years (because
>> Pylons).  A reasonably deep incision and removal of non-logic/non-plugin
>> code would make a move to Py3 easier, but with some level of breakage in
>> external plugins. Staying on 2 would mean a move to 3 at a later date and
>> more pain.
>>
>> 3. Would the CKAN Association like to fund someone to do some of this
>> work? This is just one of several ideas mentioned on
>> https://github.com/ckan/ideas-and-roadmap/issues/152 that really needs
>> to be done if CKAN is going to thrive instead of just survive.
>>
>> Any feedback welcome…
>>
>> Cheers
>>
>> Ross.
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
> Hi Ross,
>
> I believe that a Flask port (or rewrite) is an excellent idea for CKAN 3.0
> in order to support Python 3.x
> The alternative would be to port Pylons to Python 3.x, which perhaps is a
> more difficult task...
>
> Given that Python 2.x will EOL relatively soon, CKAN should move forward.
>
> Just my 2 cents.
>
> Best,
> Angelos
>
> --
> Angelos Tzotsos, PhD
> OSGeo Charter Member
> http://users.ntua.gr/tzotsos
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150914/0a96379e/attachment-0003.html>