[ckan-dev] Thoughts on realtime data

Sun Oct 27 12:55:39 UTC 2013

Hi all.

We are actually working quite a lot with real time data in relation to
CKAN... Actually, it's more or less the only thing we use it for at the
moment.

@Henrik: The municipality of Aarhus (Denmark) is already putting real time
data on their platform - I did it. They are a little in front of you guys
in Copenhagen (on that part at least) :-)

We are also working with real time data in a project called Gatesense
(where CKAN is the core). We have already had quite some luck in
registering accelerometer data from smart phones (through html5), feeding
the data to the ckan datastore, and extracting it to a web page (if any of
you come to the IoT world forum in Barcelona you can see a running
prototype - a simple game). At the moment we are experiencing a few seconds
of delay, but that is sufficient in most cases. So we consider CKAN being
usable for real time data (not in the hardcore milli second understanding,
but more on the average end user perception), but not the optimal solution
(ex. Xively is way better for that part, at the moment). We have a couple
of ideas, on the drawing board, on how to optimise CKAN for having better
real time performance. Unfortunately, we haven't time for developing these
features right now. If anyone is interested, then let's try and do some
collaboration on if, how and why real time data streaming (in and out)
should be implemented.

Best regards

Lasse Vestergaard

2013/10/21 Mark Wainwright <mark.wainwright at okfn.org>

> Hi Henrik,
>
> Others may have more informed advice, but some thoughts:
>
> 1. The Orbital project [1] did some work with an engineering project
> with a dataset of sensor data, using the Datastore API to add rows at
> particular time intervals.
>
> 2. This might be a better bet than adding a new CSV resource every 5
> minutes, which will mean the URL for the data keeps changing.
>
> I don't think much work has been done on using CKAN with real-time
> data (perhaps others will correct me) so it would be great if you
> updated the list with your thoughts/findings.
>
> Regards,
>
> Mark
>
>
> [1] orbital.blogs.lincoln.ac.uk
>
>
>
> On 16/10/2013, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk> wrote:
> > I'm trying to implement realtime data in CKAN from a SQL server.
> >
> > I've thought of various strategies, such as creating a CSV-file every 5
> > minute or so and upload it to CKAN or perhaps using the DataStore to add
> new
> > records every 5 minutes.
> >
> > Does anyone else have experience with such "challenge" as every thought,
> > idea etc. would be very welcomed.
> >
> >
>
>
> --
> Business development and user engagement manager
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131027/ec73d09a/attachment-0001.html>