[ckan-discuss] Data Registry Aggregator Experiment

Wed Mar 30 04:52:31 BST 2011

Actually rather successful. There is lots more that can be done. I
must admit I was quite surprised at the discouragin and critical
reception this received when I let it slip that I was working on it on
IRC, before I even had a chance to get it into a shape where it could
be looked at and explain a bit about what it does.

In any event here I explain somewhat:

   http://eris.okfn.org/ww/2011/03/gockan

Briefly, it is an API aggregator and protocol translator for CKAN
instances. This means (1) you can talk the CKAN JSON protocol with it
to retrieve information about datasets from many CKAN instances and
(2) you can ask it for RDF representations of these same datasets.  It
is 25-50x faster than CKAN (caveat: it doesn't do as much) and at
least 5-10x more memory efficient (caveat: same). It also has some
very interesting scaling properties and is designed with working as a
distributed system in mind.

I would be very interested in people's impressions of the architecture
and strategy. On implementation details I'm very interested in
feedback as well, only keep in mind that this was written very quickly
over the course of two days.

Cheers,
-w
-- 
William Waites                <mailto:ww at styx.org>
http://river.styx.org/ww/        <sip:ww at styx.org>
F4B3 39BF E775 CF42 0BAB  3DF0 BE40 A6DF B06F FD45