[ckan-discuss] Data Registry Aggregator Experiment

Wed Mar 30 10:06:45 BST 2011

Will,

Great to see another way of getting and serving package RDF. I agree
that using Go or other lower level language suits this particular use
case when optimising for speed.

Are your customers asking you to put the aggregated RDF in a store and
providing a sparql service?

How does this fit with the rest of the team's existing efforts in CKAN
data aggregation? I'm thinking of the dcat stuff, repo syncing and
aggregated search?

And looking at it from the reverse direction, (since we know data
communities love their independent identity too) could the RDF
function be added into a CKAN extension, say?

Dave

On 30 March 2011 04:52, William Waites <ww at styx.org> wrote:
> Actually rather successful. There is lots more that can be done. I
> must admit I was quite surprised at the discouragin and critical
> reception this received when I let it slip that I was working on it on
> IRC, before I even had a chance to get it into a shape where it could
> be looked at and explain a bit about what it does.
>
> In any event here I explain somewhat:
>
>   http://eris.okfn.org/ww/2011/03/gockan
>
> Briefly, it is an API aggregator and protocol translator for CKAN
> instances. This means (1) you can talk the CKAN JSON protocol with it
> to retrieve information about datasets from many CKAN instances and
> (2) you can ask it for RDF representations of these same datasets.  It
> is 25-50x faster than CKAN (caveat: it doesn't do as much) and at
> least 5-10x more memory efficient (caveat: same). It also has some
> very interesting scaling properties and is designed with working as a
> distributed system in mind.
>
> I would be very interested in people's impressions of the architecture
> and strategy. On implementation details I'm very interested in
> feedback as well, only keep in mind that this was written very quickly
> over the course of two days.
>
> Cheers,
> -w
> --
> William Waites                <mailto:ww at styx.org>
> http://river.styx.org/ww/        <sip:ww at styx.org>
> F4B3 39BF E775 CF42 0BAB  3DF0 BE40 A6DF B06F FD45
>
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>