[ckan-dev] Recommended practice for supporting both CKAN 1.8 and 2.0 in extensions

David Read david.read at hackneyworkshop.com
Thu Jan 31 09:53:15 UTC 2013 

Ian, thanks for expanding on how it might work sharing code across two
repositories for an extension. However it seems rather contrived to me
and I remain unconvinced by these points. Branches exist for exactly
this sort of challenge and seems to me to be the best way for
developers to keep track, compare, copy between the two similar
threads of work.

Regarding users installing the extensions, I really think we're
venturing into cloud cuckoo land if we consider the jump from 1.x to
2.0 as the only compatibility jump we're going to see. I've managed
most of the CKAN releases from 1.2 when extensions were introduced,
and every release there has been at least one extension that needed
updates due to changes in interfaces. So we need to have a system of
branching and referencing compatibility, whatever.

David

On 30 January 2013 23:07, Ian Ward <ian at excess.org> wrote:
> Here are my arguments for adding a ckanext2 namespace package (extra
> directory in existing extensions), instead of using branching:
>
>
> - Branching is already used for feature work and bug fixes for the
> extension.  When a bug fix or feature applies to both versions
> developers will have to manually cherry pick or rebase branches for
> each version.  As David mentions this cost is low for single commits,
> but the same is not true for large branches.
>
> - Testing an extension against multiple CKAN versions would be more
> expensive: you now have to switch branches between tests (and stash
> pending changes?)
>
> - With a ckanext2 namespace package it is easy to import code from
> either version "from ckanext2.foo.form import ..." so that there is no
> code duplication.  Branching has no way to share code, other than what
> existed before the branch started.
>
> - Fixes can be a single commit.  With branching, bug fix comments
> would have to link to the commit for each version.
>
> - Features can exist in a single branch.  With branching even features
> that are compatible with both versions need to duplicated.
>
> - There is no need to start from scratch and copy all the code into a
> new repository, or rename the extension itself.  ckanext-foo would
> still be ckanext-foo.  It would just add a ckanext2 directory to the
> top level *if* the extension has code/templates that only work with
> 2.x.
>
> - For end users extensions continue to just work with no extra effort.
>  CKAN 1.x and 2.x will find the right version of the code.
>
> - CKAN 1.x users can keep getting the latest ckanext-foo-X.Y.tar.gz.
> Extension developers won't have to implement some new version scheme
> for tarballs that includes the CKAN version.
>
>
> Ian
>
>
> On Wed, Jan 30, 2013 at 5:15 PM, David Read
> <david.read at hackneyworkshop.com> wrote:
>> I'm with Adria - my preference is for branching extensions when
>> developing for both CKAN 1.x vs 2.x. Perhaps when work starts on
>> compatibility for 2.x, anyone wanting to develop for 1.x can branch
>> from the appropriate point, since master should be for the latest
>> version. (I've been pretty haphazard on branching so far, as my
>> opinion changes.)
>>
>> Of course features & bugs may appear on one or other CKAN version
>> branch, which is fine, and if it's wanted on the other branch then it
>> is easy to cherry-pick. This would be harder if the namespace was
>> changed to ckanext2. And think about when CKAN 2.1 has a slightly
>> different interface (it will happen!) - you're back to branching again
>> for that anyway.
>>
>> I'm against starting a repo afresh for CKAN 2.0, since as Toby says,
>> most of the 2.0 effort is in the templates, so you'll be copying
>> plenty of the rest of the code into the new repo, which is annoying
>> when you want to cherry-pick features between them. Even in cases of
>> ckanext-archiver and ckanext-spatial which have seen serious forking
>> of how big things work, if there is a chance there will be
>> cherry-picking between the forks (and we want to encourage the forks
>> keeping as close together as possible) then it is useful them being in
>> the same repo.
>>
>> I'm tempted to say there needs to be something in the code which says
>> what versions of CKAN an extension works against, rather than relying
>> on docs as we have done until now. In this case, could we just use the
>> line in setup.py?
>> e.g.
>> install_requires=[
>>     ckan >=1.5, <2.0
>> ]
>>
>> David
>>
>> On 28 January 2013 14:30, Ian Ward <ian at excess.org> wrote:
>>> +1 for the ckanext2 idea. Branching is already used for features and
>>> bug fixes, adding version compatibility only multiplies the branches.
>>>
>>> However, the python packages themselves could still be named ckanext-*.
>>>
>>> ckanext2 could just be another namespace directory in the python
>>> package, and CKAN 2.0 would select the extension in that directory if
>>> it exists.  Extensions that need no changes for 2.0 can remain
>>> unchanged.
>>>
>>> On Fri, Jan 25, 2013 at 9:21 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:
>>>> I'm not sure creating a brand new extension (ckanext2) is the right
>>>> way to go, for me it feels more complicated having to maintain twice
>>>> as much repos just for the convenience of having a new namespace. For
>>>> users installing an extension choosing ckanext/ckanext2 is at least as
>>>> annoying (I would say a bit more) than pointing to a particular branch
>>>> or tag, and for us it means that we can maintain a single repo.
>>>>
>>>> I don't think we need to create a new branch for each CKAN core
>>>> release if there haven't been any changes needed on the extension.
>>>> Perhaps the easiest way is to go is using tags that match ckan core
>>>> release, so eg in the release-2.0 branch there could be ckan-2.0,
>>>> ckan-2.1 tags, even pointing to the same commit if the extension is
>>>> indeed compatible between them. The problem with tags is what happens
>>>> if the extension gets updated with new features (still compatible),
>>>> you still want users to get the latest code, but that would mean
>>>> moving the tag which I'm not sure if it a good idea.
>>>>
>>>> Whether if on the master/release-v.2.0 branch we keep compatibility
>>>> for stuff like the genshi templates is a different matter. I'm also +1
>>>> on clean start, but this may need to be reviewed for particular cases.
>>>>
>>>>
>>>> I agree it would be good to get feedback from users using extensions
>>>> on their instances.
>>>>
>>>>
>>>> Adrià
>>>>
>>>>
>>>> On 25 January 2013 12:21, Toby Dacre <toby.okfn at gmail.com> wrote:
>>>>> Revisiting this issue - now that I am going to be updating the archiver and
>>>>> qa extensions for 2.0 and I am keen to clean them up to be example
>>>>> extensions if possible.
>>>>>
>>>>> For this I want to break the 1.x compatibility and make a clean start for
>>>>> 2.x eg remove all IGenshiFilters etc use the plugins.toolkit try to not
>>>>> reach into ckan
>>>>>
>>>>> so the options I see are
>>>>>
>>>>> a) branch release-v2.x (or whatever) this feels a bit complicated and hacky
>>>>> but is the easiest in the short term but OI think has long term costs.
>>>>>
>>>>> b) create a new extension ckanext2-qa, ckanext2-archiver so we get a whole
>>>>> new namespace.  We could make ckan 2.0 accept both namespaces and it would
>>>>> maybe make things less confused for ckan 1.x
>>>>>
>>>>> c) some other solution I don't know yet
>>>>>
>>>>> Personally I think ckanext2 is my preferred option but we need a definite
>>>>> decision that we are happy with and can communicate to our users
>>>>>
>>>>> I'm keen for feedback on this idea from ckan devs and the wider ckan
>>>>> community dread, ross etc
>>>>>
>>>>> Toby
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 18 January 2013 15:36, Toby Dacre <toby.okfn at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 18 January 2013 09:30, Ross Jones <ross at servercode.co.uk> wrote:
>>>>>>>
>>>>>>> Toby,
>>>>>>>
>>>>>>> Do you have any suggestions on the best way to keep the extensions
>>>>>>> working with 1.x, and do you have objections to branching the current code
>>>>>>> into a 1.x branch?  There are still a lot of sites on 1.x for whom the
>>>>>>> upgrade to 2.0 would be very difficult, or at least very time consuming, at
>>>>>>> least in terms of templating - maybe not immediately, but at some point in
>>>>>>> the future.
>>>>>>>
>>>>>>
>>>>>> Ross,
>>>>>>
>>>>>> Hi, I agree it is important to keep 1.x extensions running.  We should at
>>>>>> the minimum branch at a working version.
>>>>>>
>>>>>> Personally I'm very much in to maintaining backwards compatibility where
>>>>>> we can.  Some extensions can support both ckan 1.0 and 2.0 together easily.
>>>>>> But for others it is more difficult.
>>>>>>
>>>>>> But we should keep the existing stuff working.
>>>>>>
>>>>>> I'd really like us to get some good extension writing docs at some point
>>>>>> but it would require quite a bit of work and some decisions being made.
>>>>>>
>>>>>> basically 1.x we need to use genshi templates, 2.0 we can use both but
>>>>>> genshi is due to die at some point so they should use jinja2 so to support
>>>>>> both we need both template types.  With this we could postpone killing
>>>>>> genshi and just keep genshi templates (this may be the best approach)
>>>>>> switching to template helpers so the extension adds a helper function that
>>>>>> renders a snippet means we don't need IGenshiStreamFilter which is the main
>>>>>> problem.
>>>>>>
>>>>>> There are more issues with including stuff from ckan outside of
>>>>>> ckan.plugins that cause problems.  I think we need to look at each extension
>>>>>> individually.
>>>>>>
>>>>>> Hope that helps.  let me know if you need any specific info.  Hope all is
>>>>>> good with you.  I'm in a bar on a beach in Thailand so can't complain too
>>>>>> much
>>>>>>
>>>>>> Toby
>>>>>>
>>>>>>> Ross
>>>>>>>
>>>>>>> On 18 Jan 2013, at 04:36, Toby Dacre wrote:
>>>>>>>
>>>>>>> My thoughts on this is that it would be nice to have extensions that just
>>>>>>> work and do not need to be updated just because ckan has changed.
>>>>>>>
>>>>>>> Once an extension is ckan 2.0 compatible (and correctly written) then it
>>>>>>> should work for all future versions of ckan. Therefore there would be no
>>>>>>> need to release a 2.0, 2.1, 2.x of the extension.  This is mainly about
>>>>>>> using plugins.toolkit as the only method of integration with ckan.  Now that
>>>>>>> we can call get_action() without a context the last main hurdle has been
>>>>>>> removed as extensions needed to import ckan.model before this.
>>>>>>>
>>>>>>> 2.0 also moves to Jinja2 so it is a good cut off point or else we need
>>>>>>> multiple templates or keep genshi ones (but it would be nice to remove these
>>>>>>> in 2.1 or 2.2)
>>>>>>>
>>>>>>> Hmm
>>>>>>>
>>>>>>> This is all a bit messy.  My main desire with extensions is to isolate
>>>>>>> them from core as much as possible to allow refactoring without breakage.
>>>>>>>
>>>>>>> As a reminder if we need new things added to the toolkit then this is
>>>>>>> possible if they make sense and we are happy to support them long term.  But
>>>>>>> much of the issue feels is about how extensions are written so for exam[le I
>>>>>>> don't think they should ever include lib.helpers but the can access any
>>>>>>> helper functions within their templates for example - again this gives us
>>>>>>> flexibility that we need to maintain the code and give power to the end
>>>>>>> user.
>>>>>>>
>>>>>>> It would be nice to have better extension writing guidelines at some
>>>>>>> point.
>>>>>>>
>>>>>>> tobes
>>>>>>>
>>>>>>>
>>>>>>> On 17 January 2013 11:29, Sean Hammond <sean.hammond at okfn.org> wrote:
>>>>>>>>
>>>>>>>> I agree. I think to keep it simple and consistent our recommendation
>>>>>>>> should just be that extension branches track the CKAN master and release
>>>>>>>> branches, even though in reality, very new or simple extensions will
>>>>>>>> probably only have the one master branch (but by the time they have
>>>>>>>> conditional behaviour like googleanalytics does, we should just
>>>>>>>> recommend them to use the branches).
>>>>>>>>
>>>>>>>> We'll have to change the install instructions in the README file of each
>>>>>>>> extension, and probably mention it for extension authors in the 'writing
>>>>>>>> extensions' sphinx docs too.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ckan-dev mailing list
>>>>>>> ckan-dev at lists.okfn.org
>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ckan-dev mailing list
>>>>>>> ckan-dev at lists.okfn.org
>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

More information about the ckan-dev mailing list