[ckan-dev] Datastreaming possible?

Sun Mar 24 16:22:55 UTC 2013

Hallo Stephan,

I have to admit that I haven't used any of the pages you mention but you can definitely use the upsert method [1] to append data to the datastore. It also supports array types which you can use for the acceleration [2]. The graph comes from recline [3] which, because of the way it is integrated in CKAN, cannot be used to combine multiple resources. So you would need to store everything in one datastore resource. Alternatively, you can set up your own recline and use the sql search of the datastore [4] to join multiple resources. 

The datastore uses a postgres database which should be able to handle a few requests per second. However, it won't work if you want to add hundreds of new datasets per second.

Having said that, I'm not sure whether the recline preview is the feature you want since it is intended to let you explore tabular data and not a tool to create generic visualizations. I'd suggest that you use the datastore to store the data and write your queries and use the json output to build a visualization using one of the many js plotting libraries out there.

Best wishes,
Dominik

[1] http://docs.ckan.org/en/latest/datastore-api.html#ckanext.datastore.logic.action.datastore_upsert
[2] http://docs.ckan.org/en/latest/datastore-api.html#field-types
[3] http://reclinejs.com/
[4] http://docs.ckan.org/en/latest/datastore-api.html#ckanext.datastore.logic.action.datastore_search_sql

On 24 Mar 2013, at 16:46, Stephan Opfer <opfer at vs.uni-kassel.de> wrote:

> Hi CKANer,
> 
> I wonder if it is possible to use CKAN for the same purpose as thingspeak.com or cosm.com are made for.
> 
> The basic idea is to collect data from several sensor nodes every second and integrate them into the datastorage of ckan. The data can contain temperature, sonar-based distances, accelerations (x,y,z).
> 
> At least I saw, that it is possible to visualize data in a graph. Especially, it is possible to combine several data sources in one graph, in order to compare them. This, for example, is not possible with thingspeak.com.
> 
> A crucial question is, how fast can new data be integrated automatically.
> 
> Best Regards,
>  Stephan
> -- 
> Distributed Systems Research Group
> Stephan Opfer T. +49 561 804-6283 F. +49 561 804-6277
> Univ. Kassel, FB 16, Wilhelmshöher Allee 73, D-34121 Kassel
> WWW: http://www.vs.uni-kassel.de/mitarbeiter/wiss-mitarbeiter/stephan-opfer/
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev