[ckan-dev] CREPs - important proposal!

[Seb Bacon]

Thanks for all the feedback everyone!

On 6 May 2011 13:44, Friedrich Lindenberg <friedrich.lindenberg at okfn.org> wrote:
> Hi all,
>
> On Fri, May 6, 2011 at 12:17 PM, David Raznick <kindly at gmail.com> wrote:
>> +1 for Cep.   Actually I think they should be call "Sebs" ...
>
> +1 on "Sebs"

But it seems that you don't really like the proposal, so I'm not sure
how to take the compliment ;-)

>>> How about this: we could suggest that as a first step, people fire off
>>> an email to ckan-dev with a one paragraph of their idea, and see if
>>> the consensus is that they should write up a CEP for it?  This would
>>> cover you and Will's concerns quite nicely.  The guideline would then
>>> be that *every* idea should start with an email of any length at all
>>> to ckan-dev; and CEPs should only be written if requested (or the
>>> proposer thinks it's needed anyway)?
>
> Just to chime in a bit: I'm pretty neutral on the idea - I think the
> "enhancement protocol" concept applies much better to API projects and
> that we should be honest to say that this is a waterfallish
> requirements document that we're proposing, but maybe this is needed.
> I don't think this fixes many of the core problems we have with CKAN
> development, though. Let me explain the challenges that I see:

We could call it a CKAN Requirements Understanding Document if you preferred ;-)

I don't understand this "only applies to API projects" point that you
and Will have made.  We're talking about software that many different
parties rely on, and use in many different contexts.  Each time we add
a new feature, we are potentially "breaking" it for other users (e.g.
confusing them with a UI that works against their own particular
workflow).  Conceptually, in the broadest sense, DEPs, PEPs and CEPs
are all about attempting to have conversations about change, between
members a large, distributed "committee".  That's the important
similarity.

> 1) We have many clients with very specific requirements and these are
> added as needed.

There are also loads of bits of functionality that aren't being added
to meet any specific client request, e.g. the admin interface that
John's been working on, or the workbench questions we've been
discussing lately.

Also, remember there are non-programmers with opinions about where
things should go.  It would be good to give them a structured way of
proposing ideas.  People are intimidated by the idea of writing a
free-form email to a list of people they don't know.  Giving them
structure should help with this.

> Putting up a CEP for one of those after you're often
> in practice under some contractual or personal obligation to do it
> anyway seems a bit dishonest to me. If the question is "Should we add
> X" and the UK government wants X - we should just have a quick
> implementation discussion, not a "consensus-driven" CEP (which implies
> some openness wrt. to the requirements IMO).

If you phrase the question as "should we have X", then it would be
dishonest.  If you phrase it "we need X, how should we implement it",
then it seems fine to me.  Then the consensus would be around the
implementation, not the requirement.

The phrase "just have a quick implementation discussion" is key.  How
can we have a sensible implementation discussion as a group?  I'd say,
like we're doing right now, by email, with reference to a proposal.

> 2) We need a way for the larger community to stay up to date with CKAN
> development. The goal here IMO should be short blog posts and good
> documentation of the actual implementation. This is not to say we
> should not plan new functionality in an open way, just that the
> specific problem of keeping people up to date just won't be fixed by
> this.

Absolutely.  I believe this is orthogonal to the requirement that CEPs
are supposed to address.  The problem of getting everyone to write
blog posts and good documentation is a whole thorny issue of its own -
ask Lucy or Jonathan or Rufus about pestering people to do so :)  I
think it's harder than the CEP proposal.

> 3) We need to recognize and support the emerging workflows in and
> around CKAN. These are small, unstable and initially not well defined
> practices. Writing a 5 page document on "how people should do
> cooperative work in CKAN" at this stage will in its best case miss the
> point, in the worst case it may actively harm the use by imposing
> rules based on a model that's not likely to be sufficiently inspired
> by practice (see TimBLs 5 stars: academic best practices that hurt
> reality).

If I understand the objections right, it seems that there's a concern
that my proposal would be a massive barrier to doing anything.  I
don't think it needs to be; after all, even an hour isn't a very long
time compared with how long we've all spent trying to understand
something that's not documented, or trying to hunt some elusive bug.

I am just worried that without some structure, we will fail to gain
cohesiveness as a team, which is in itself a barrier; in fact, a *big*
risk to our collective success.  Right now we are too often like a
bunch of separate people all working on our own CKAN sub projects.

Re. being inspired by practice: that's subjective.  I could argue that
the CEP proposition *is* inspired by practice: currently there are
various overlapping conversations.  The proposal is to capture these
conversations in a standard format so that everyone can view and
comment on them.

> Overall, I think that we might be better served by keeping a more
> structured six-month backlog in which larger stories can be added and
> are then broken down as their implementation approaches and which will
> be priorised by some steering committee, rather than focusing on these
> documents. They might look similar, but I'd like to stick with
> describing desired functionality in a concise format rather than
> formulating "CKAN theories".

I don't get the difference.  Apart from the "implementation plan" part
of the CEP proposal, which could be dropped and dealt with elsewhere,
I think the proposed headings should lead to a pretty concise format.
If they're in Trac with metadata that allows simple listing of current
"stories" by status, then a steering committee can prioritise them if
we like.  Notwithstanding your allergy to "theories", could you
explain the difference a bit more? :)

Thanks,

Seb