[okfn-labs] Next generation data catalogues.

Tue Aug 27 08:37:59 UTC 2013

Hi Rufus,

On 27 Aug 2013, at 00:12, Rufus Pollock <rufus.pollock at okfn.org> wrote:

> Very interesting.
> 
> I would observe that http://data.okfn.org/ is in some ways a bit of an experiment in precisely this regard - its a data catalog but a radically stripped down one with a strong orientation to a) data packages b) small data c) data stored in git (and github) [d) being written in js!]
> 
> We've also had experiments with pure JS data catalogs (e.g. https://github.com/okfn/datacatalog.js) individual data stores, Friedrich's datahub, lists stored in wikis, extensions to CMS'es and more over the years ;-)

I think they're all cool ideas (and more importantly, implementations) and I don't doubt we'll be stealing as many ideas as possible, and possibly even code.  

> I think, as already being discussed later in this thread, the key question is:
> 
> * What user stories do you have (what features do you want)?

Initially, built in multi-lingual datasets (this is important), DCAT metadata, datatank style serialisation. There's some discussion about API or Services, I don't think there's a solid consensus yet though. 

> Related to both of these here are some diagrams indicating thoughts at the time about what a "DataHub" might offer (and also suggesting a fairly separated architecture):
> 
> http://notebook.okfn.org/2012/06/22/datahub-small-pieces-loosely-joined/
> http://notebook.okfn.org/2011/04/27/data-hubs-data-management-systems-and-ckan/ (older)

Think I've seen this before, but very useful, thanks.

> PS: you've already mentioned it below but I'll mention it again: "beware second-system syndrome" ;-) This does not mean one should not build new (and certainly the discussion of problems and possibilities re new is *very* useful) but before actually starting on something (completely) new you want to have a very strong benefit/cost in favour of that versus fix/extend/enhance of existing system (as a side note: originally version of CKAN started life as various wikis on exactly this logic - it was only after we got *really* sick of trying to mod-MoinMoin that CKAN in its current pythonic form began …)

Yeah, it was something I mulled over, but I'm not really any form of lead on it, I'm just sticking my oar in (as usual ;) ). I think DKAN is a fairly good example of why just trying to re-write something isn't always a very good idea (primarily because it adds nothing).

Ross

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/okfn-labs/attachments/20130827/5d076190/attachment-0002.html>