[ckan-dev] Recommended practice for supporting both CKAN 1.8 and 2.0 in extensions

Thu Jan 31 10:03:32 UTC 2013

Hi,

If anybody is interested, I have put a bounty of 100 points of my hard
earned reputation on StackOverflow to answer the question on uploading a
file to datahub.io through a Java client.
No response yet... common Java developers, that's an easy one!
http://stackoverflow.com/questions/14535523/uploading-files-to-a-dataset-in-ckan-datahub-io-through-a-java-client
Best,

Clement

--------------------------------------------
Clement Levallois, PhD
Erasmus University Rotterdam
The Netherlands

pro website<http://www.erim.eur.nl/ERIM/People/Person_Details?p_aff_id=4321>
 / personal website <http://www.clementlevallois.net/>

twitter and skype: @seinecle
Discover the NESSHI project: http://www.nesshi.eu

On Thu, Jan 31, 2013 at 10:53 AM, David Read <david.read at hackneyworkshop.com
> wrote:

> Ian, thanks for expanding on how it might work sharing code across two
> repositories for an extension. However it seems rather contrived to me
> and I remain unconvinced by these points. Branches exist for exactly
> this sort of challenge and seems to me to be the best way for
> developers to keep track, compare, copy between the two similar
> threads of work.
>
> Regarding users installing the extensions, I really think we're
> venturing into cloud cuckoo land if we consider the jump from 1.x to
> 2.0 as the only compatibility jump we're going to see. I've managed
> most of the CKAN releases from 1.2 when extensions were introduced,
> and every release there has been at least one extension that needed
> updates due to changes in interfaces. So we need to have a system of
> branching and referencing compatibility, whatever.
>
> David
>
> On 30 January 2013 23:07, Ian Ward <ian at excess.org> wrote:
> > Here are my arguments for adding a ckanext2 namespace package (extra
> > directory in existing extensions), instead of using branching:
> >
> >
> > - Branching is already used for feature work and bug fixes for the
> > extension.  When a bug fix or feature applies to both versions
> > developers will have to manually cherry pick or rebase branches for
> > each version.  As David mentions this cost is low for single commits,
> > but the same is not true for large branches.
> >
> > - Testing an extension against multiple CKAN versions would be more
> > expensive: you now have to switch branches between tests (and stash
> > pending changes?)
> >
> > - With a ckanext2 namespace package it is easy to import code from
> > either version "from ckanext2.foo.form import ..." so that there is no
> > code duplication.  Branching has no way to share code, other than what
> > existed before the branch started.
> >
> > - Fixes can be a single commit.  With branching, bug fix comments
> > would have to link to the commit for each version.
> >
> > - Features can exist in a single branch.  With branching even features
> > that are compatible with both versions need to duplicated.
> >
> > - There is no need to start from scratch and copy all the code into a
> > new repository, or rename the extension itself.  ckanext-foo would
> > still be ckanext-foo.  It would just add a ckanext2 directory to the
> > top level *if* the extension has code/templates that only work with
> > 2.x.
> >
> > - For end users extensions continue to just work with no extra effort.
> >  CKAN 1.x and 2.x will find the right version of the code.
> >
> > - CKAN 1.x users can keep getting the latest ckanext-foo-X.Y.tar.gz.
> > Extension developers won't have to implement some new version scheme
> > for tarballs that includes the CKAN version.
> >
> >
> > Ian
> >
> >
> > On Wed, Jan 30, 2013 at 5:15 PM, David Read
> > <david.read at hackneyworkshop.com> wrote:
> >> I'm with Adria - my preference is for branching extensions when
> >> developing for both CKAN 1.x vs 2.x. Perhaps when work starts on
> >> compatibility for 2.x, anyone wanting to develop for 1.x can branch
> >> from the appropriate point, since master should be for the latest
> >> version. (I've been pretty haphazard on branching so far, as my
> >> opinion changes.)
> >>
> >> Of course features & bugs may appear on one or other CKAN version
> >> branch, which is fine, and if it's wanted on the other branch then it
> >> is easy to cherry-pick. This would be harder if the namespace was
> >> changed to ckanext2. And think about when CKAN 2.1 has a slightly
> >> different interface (it will happen!) - you're back to branching again
> >> for that anyway.
> >>
> >> I'm against starting a repo afresh for CKAN 2.0, since as Toby says,
> >> most of the 2.0 effort is in the templates, so you'll be copying
> >> plenty of the rest of the code into the new repo, which is annoying
> >> when you want to cherry-pick features between them. Even in cases of
> >> ckanext-archiver and ckanext-spatial which have seen serious forking
> >> of how big things work, if there is a chance there will be
> >> cherry-picking between the forks (and we want to encourage the forks
> >> keeping as close together as possible) then it is useful them being in
> >> the same repo.
> >>
> >> I'm tempted to say there needs to be something in the code which says
> >> what versions of CKAN an extension works against, rather than relying
> >> on docs as we have done until now. In this case, could we just use the
> >> line in setup.py?
> >> e.g.
> >> install_requires=[
> >>     ckan >=1.5, <2.0
> >> ]
> >>
> >> David
> >>
> >> On 28 January 2013 14:30, Ian Ward <ian at excess.org> wrote:
> >>> +1 for the ckanext2 idea. Branching is already used for features and
> >>> bug fixes, adding version compatibility only multiplies the branches.
> >>>
> >>> However, the python packages themselves could still be named ckanext-*.
> >>>
> >>> ckanext2 could just be another namespace directory in the python
> >>> package, and CKAN 2.0 would select the extension in that directory if
> >>> it exists.  Extensions that need no changes for 2.0 can remain
> >>> unchanged.
> >>>
> >>> On Fri, Jan 25, 2013 at 9:21 AM, Adrià Mercader <
> adria.mercader at okfn.org> wrote:
> >>>> I'm not sure creating a brand new extension (ckanext2) is the right
> >>>> way to go, for me it feels more complicated having to maintain twice
> >>>> as much repos just for the convenience of having a new namespace. For
> >>>> users installing an extension choosing ckanext/ckanext2 is at least as
> >>>> annoying (I would say a bit more) than pointing to a particular branch
> >>>> or tag, and for us it means that we can maintain a single repo.
> >>>>
> >>>> I don't think we need to create a new branch for each CKAN core
> >>>> release if there haven't been any changes needed on the extension.
> >>>> Perhaps the easiest way is to go is using tags that match ckan core
> >>>> release, so eg in the release-2.0 branch there could be ckan-2.0,
> >>>> ckan-2.1 tags, even pointing to the same commit if the extension is
> >>>> indeed compatible between them. The problem with tags is what happens
> >>>> if the extension gets updated with new features (still compatible),
> >>>> you still want users to get the latest code, but that would mean
> >>>> moving the tag which I'm not sure if it a good idea.
> >>>>
> >>>> Whether if on the master/release-v.2.0 branch we keep compatibility
> >>>> for stuff like the genshi templates is a different matter. I'm also +1
> >>>> on clean start, but this may need to be reviewed for particular cases.
> >>>>
> >>>>
> >>>> I agree it would be good to get feedback from users using extensions
> >>>> on their instances.
> >>>>
> >>>>
> >>>> Adrià
> >>>>
> >>>>
> >>>> On 25 January 2013 12:21, Toby Dacre <toby.okfn at gmail.com> wrote:
> >>>>> Revisiting this issue - now that I am going to be updating the
> archiver and
> >>>>> qa extensions for 2.0 and I am keen to clean them up to be example
> >>>>> extensions if possible.
> >>>>>
> >>>>> For this I want to break the 1.x compatibility and make a clean
> start for
> >>>>> 2.x eg remove all IGenshiFilters etc use the plugins.toolkit try to
> not
> >>>>> reach into ckan
> >>>>>
> >>>>> so the options I see are
> >>>>>
> >>>>> a) branch release-v2.x (or whatever) this feels a bit complicated
> and hacky
> >>>>> but is the easiest in the short term but OI think has long term
> costs.
> >>>>>
> >>>>> b) create a new extension ckanext2-qa, ckanext2-archiver so we get a
> whole
> >>>>> new namespace.  We could make ckan 2.0 accept both namespaces and it
> would
> >>>>> maybe make things less confused for ckan 1.x
> >>>>>
> >>>>> c) some other solution I don't know yet
> >>>>>
> >>>>> Personally I think ckanext2 is my preferred option but we need a
> definite
> >>>>> decision that we are happy with and can communicate to our users
> >>>>>
> >>>>> I'm keen for feedback on this idea from ckan devs and the wider ckan
> >>>>> community dread, ross etc
> >>>>>
> >>>>> Toby
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 18 January 2013 15:36, Toby Dacre <toby.okfn at gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 18 January 2013 09:30, Ross Jones <ross at servercode.co.uk> wrote:
> >>>>>>>
> >>>>>>> Toby,
> >>>>>>>
> >>>>>>> Do you have any suggestions on the best way to keep the extensions
> >>>>>>> working with 1.x, and do you have objections to branching the
> current code
> >>>>>>> into a 1.x branch?  There are still a lot of sites on 1.x for whom
> the
> >>>>>>> upgrade to 2.0 would be very difficult, or at least very time
> consuming, at
> >>>>>>> least in terms of templating - maybe not immediately, but at some
> point in
> >>>>>>> the future.
> >>>>>>>
> >>>>>>
> >>>>>> Ross,
> >>>>>>
> >>>>>> Hi, I agree it is important to keep 1.x extensions running.  We
> should at
> >>>>>> the minimum branch at a working version.
> >>>>>>
> >>>>>> Personally I'm very much in to maintaining backwards compatibility
> where
> >>>>>> we can.  Some extensions can support both ckan 1.0 and 2.0 together
> easily.
> >>>>>> But for others it is more difficult.
> >>>>>>
> >>>>>> But we should keep the existing stuff working.
> >>>>>>
> >>>>>> I'd really like us to get some good extension writing docs at some
> point
> >>>>>> but it would require quite a bit of work and some decisions being
> made.
> >>>>>>
> >>>>>> basically 1.x we need to use genshi templates, 2.0 we can use both
> but
> >>>>>> genshi is due to die at some point so they should use jinja2 so to
> support
> >>>>>> both we need both template types.  With this we could postpone
> killing
> >>>>>> genshi and just keep genshi templates (this may be the best
> approach)
> >>>>>> switching to template helpers so the extension adds a helper
> function that
> >>>>>> renders a snippet means we don't need IGenshiStreamFilter which is
> the main
> >>>>>> problem.
> >>>>>>
> >>>>>> There are more issues with including stuff from ckan outside of
> >>>>>> ckan.plugins that cause problems.  I think we need to look at each
> extension
> >>>>>> individually.
> >>>>>>
> >>>>>> Hope that helps.  let me know if you need any specific info.  Hope
> all is
> >>>>>> good with you.  I'm in a bar on a beach in Thailand so can't
> complain too
> >>>>>> much
> >>>>>>
> >>>>>> Toby
> >>>>>>
> >>>>>>> Ross
> >>>>>>>
> >>>>>>> On 18 Jan 2013, at 04:36, Toby Dacre wrote:
> >>>>>>>
> >>>>>>> My thoughts on this is that it would be nice to have extensions
> that just
> >>>>>>> work and do not need to be updated just because ckan has changed.
> >>>>>>>
> >>>>>>> Once an extension is ckan 2.0 compatible (and correctly written)
> then it
> >>>>>>> should work for all future versions of ckan. Therefore there would
> be no
> >>>>>>> need to release a 2.0, 2.1, 2.x of the extension.  This is mainly
> about
> >>>>>>> using plugins.toolkit as the only method of integration with ckan.
>  Now that
> >>>>>>> we can call get_action() without a context the last main hurdle
> has been
> >>>>>>> removed as extensions needed to import ckan.model before this.
> >>>>>>>
> >>>>>>> 2.0 also moves to Jinja2 so it is a good cut off point or else we
> need
> >>>>>>> multiple templates or keep genshi ones (but it would be nice to
> remove these
> >>>>>>> in 2.1 or 2.2)
> >>>>>>>
> >>>>>>> Hmm
> >>>>>>>
> >>>>>>> This is all a bit messy.  My main desire with extensions is to
> isolate
> >>>>>>> them from core as much as possible to allow refactoring without
> breakage.
> >>>>>>>
> >>>>>>> As a reminder if we need new things added to the toolkit then this
> is
> >>>>>>> possible if they make sense and we are happy to support them long
> term.  But
> >>>>>>> much of the issue feels is about how extensions are written so for
> exam[le I
> >>>>>>> don't think they should ever include lib.helpers but the can
> access any
> >>>>>>> helper functions within their templates for example - again this
> gives us
> >>>>>>> flexibility that we need to maintain the code and give power to
> the end
> >>>>>>> user.
> >>>>>>>
> >>>>>>> It would be nice to have better extension writing guidelines at
> some
> >>>>>>> point.
> >>>>>>>
> >>>>>>> tobes
> >>>>>>>
> >>>>>>>
> >>>>>>> On 17 January 2013 11:29, Sean Hammond <sean.hammond at okfn.org>
> wrote:
> >>>>>>>>
> >>>>>>>> I agree. I think to keep it simple and consistent our
> recommendation
> >>>>>>>> should just be that extension branches track the CKAN master and
> release
> >>>>>>>> branches, even though in reality, very new or simple extensions
> will
> >>>>>>>> probably only have the one master branch (but by the time they
> have
> >>>>>>>> conditional behaviour like googleanalytics does, we should just
> >>>>>>>> recommend them to use the branches).
> >>>>>>>>
> >>>>>>>> We'll have to change the install instructions in the README file
> of each
> >>>>>>>> extension, and probably mention it for extension authors in the
> 'writing
> >>>>>>>> extensions' sphinx docs too.
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> ckan-dev mailing list
> >>>>>>>> ckan-dev at lists.okfn.org
> >>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> ckan-dev mailing list
> >>>>>>> ckan-dev at lists.okfn.org
> >>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> ckan-dev mailing list
> >>>>>>> ckan-dev at lists.okfn.org
> >>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> ckan-dev mailing list
> >>>>> ckan-dev at lists.okfn.org
> >>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> ckan-dev mailing list
> >>>> ckan-dev at lists.okfn.org
> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130131/6a944fcb/attachment-0001.html>