[okfn-labs] Next generation data catalogues.

[Rufus Pollock]

On 26 August 2013 15:45, Ross Jones <ross at servercode.co.uk> wrote:

> Hi,
>
> There was a bit of discussion on the #ckan channel this morning between
> some developers
> building data catalogs around what a next-generation data catalog would
> look like.  It was
> noted that it might _not_ look like CKAN, and so we booted up the
> catalog-ng organisation
> on github (https://github.com/catalog-ng) as a place to work on some
> ideas and proof-of-concepts
> around what we think would be included in a new data catalog using the
> knowledge gained
> working with CKAN.
>

Very interesting.

I would observe that http://data.okfn.org/ is in some ways a bit of an
experiment in precisely this regard - its a data catalog but a radically
stripped down one with a strong orientation to a) data packages b) small
data c) data stored in git (and github) [d) being written in js!]

We've also had experiments with pure JS data catalogs (e.g.
https://github.com/okfn/datacatalog.js) individual data stores, Friedrich's
datahub, lists stored in wikis, extensions to CMS'es and more over the
years ;-)

I think, as already being discussed later in this thread, the key question
is:

* What user stories do you have (what features do you want)?
* What kind of architecture do you want to adopt (e.g. small services
loosely joined, small core with plugins, one consolidated system)

Related to both of these here are some diagrams indicating thoughts at the
time about what a "DataHub" might offer (and also suggesting a fairly
separated architecture):

http://notebook.okfn.org/2012/06/22/datahub-small-pieces-loosely-joined/
http://notebook.okfn.org/2011/04/27/data-hubs-data-management-systems-and-ckan/
 (older)


> There really isn't a plan yet, certainly the intention isn't to damage
> CKAN in any way, but more
> of a way to experiment with building catalogs without any of the baggage
> involved with a
> mature product - what would you build today with the new tech available
> and the experience
> of working with CKAN.  Rather than go through the process of doing this
> outside, I thought it might
> be useful to do this as part of labs and keep it inside OKF, and possibly
> keep it on a OKF mailing list.
>

+1 on all of this.


> Does this belong in OKFN-labs? Does the idea of building something new in
> that area sound
> vaguely interesting to anyone (other than those of us with work with CKAN
> day-to-day)?
>

I definitely think this "belongs" in Labs :-) and I'm certainly a big +1 on
having this discussion (and I would imagine there are def folks beyond
CKAN-ers who find this interesting).

Rufus

PS: you've already mentioned it below but I'll mention it again: "beware
second-system syndrome" ;-) This does not mean one should not build new
(and certainly the discussion of problems and possibilities re new is
*very* useful) but before actually starting on something (completely) new
you want to have a very strong benefit/cost in favour of that versus
fix/extend/enhance of existing system (as a side note: originally version
of CKAN started life as various wikis on exactly this logic - it was only
after we got *really* sick of trying to mod-MoinMoin that CKAN in its
current pythonic form began ...)
*

*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/okfn-labs/attachments/20130827/538f6ec8/attachment-0002.html>