[ckan-discuss] distributing changes and syncing instances

Wed Mar 17 13:20:08 GMT 2010

On 17 March 2010 12:22, Friedrich Lindenberg <friedrich at pudo.org> wrote:
>>> Our current analysis is here:
>>> http://knowledgeforge.net/ckan/trac/wiki/DistributingChanges
>
> Two thoughts:
>
> - I don't know if this has been discussed previously, but after
> researching OAI-PMH (http://www.openarchives.org/) for a bit, I feel
> like there is a lot of functional overlap here. The protocol is
> non-REST, overengineered and it does not have native support for
> versioning, yet it is widely accepted and supported (DSpace and most
> other OA servers, Google Scholar, etc.).

[...]

I think having a mapping into OAI-PMH would definitely be nice and
shouldn't be that hard to do.

As Benoit notes though it is orthogonal though to the question of
supporting revisioning/versioning.

> - There seems to be a lot of interesting work going on wrt replication
> in CouchDB. While adopting their technology for package data storage
> (instead of SQL) is probably overkill, maybe there are some abstract
> ideas we could ... inherit?

Good idea. I have used CouchDB a bit (and MongoDB) but haven't looked
at their replication mechanism as yet.

My impression is that this is similar to the general distributed
databases problem and synchronising distributed writes (if people can
write in different places they can write to the same document at the
same time ...). This is a problem we partly have too but we're also
rather different in that we wish to stores revisions *permanetly*
whereas in distributed db land (as i understand it) you only need to
store revisions temporarily while you resolve any write conflicts that
occurred.

I think CouchDB replication mechanism fits more into this latter
one-head-but-possible-conflict scenario rather than full distributed
revision control but I'm sure their experience with diffs and wire
protocol for disseminating them would be useful.

>> I've started a more formal RFC-like document aimed at generalising the
>> protocol beyond the CKAN here:
>>
>>  <http://knowledgeforge.net/ckan/hg/file/tip/doc/distributed.rst>

Looks like someone pushed onto a separate branch straight after (why I
should use 'tip' when referring to changesets i guess).

It's back now: <http://knowledgeforge.net/ckan/hg/file/tip/doc/distributed.rst>

Rufus