[CKAN-support] Real-time data

Wed Mar 12 23:07:57 UTC 2014

Hi Akhilesh,

You’re right, there is a harvesting extension, but unfortunately, you seem to have found some rather old documentation for it. The current documentation/home is https://github.com/ckan/ckanext-harvest — hopefully the README document explains things a lot better: it’s significantly more detailed.

Automating data updates is, a great aim. For providing real-time data, using the DataStore extension may be a nice approach: http://docs.ckan.org/en/latest/maintaining/datastore.html explains the features of the DataStore.

Do you think that those appraoches might be useful approaches to you? It could be possible to write a custom harvester, that hits, periodically a set (list) of URI end-points to retrieve and then process the data found, otherwise.

Best wishes,

Adam

> ---------- Forwarded message ----------
> From: Harsh, Akhilesh (OCIO) <Akhilesh.Harsh at sa.gov.au>
> Date: 7 March 2014 00:46
> Subject: Real-time data [DLM=For-Official-Use-Only]
> To: Darwin Peltan <darwin.peltan at okfn.org>
> Cc: "Nigel Babu (nigel.babu at okfn.org)" <nigel.babu at okfn.org>
> 
> 
> For Official Use Only
>  
> Hi Darwin,
>  
> There is a feature in CKAN documentation about the Harvesting extension http://docs.ckan.org/en/ckan-1.7.2/loading-data.html#load-data-harvester that mentions having a web interface for running harvesting jobs.
>  
> Would it be possible for you to explain how this works? And also is this feature available, or will it be available in the future releases?
> Basically, we are trying to identify processes for agencies to automate the update of data. Currently, users can access static spreadsheets or shp files, but how do agencies provide real-time data? Please let us know if you have any ideas or suggestions.
>  
> More than happy to talk to you over the phone next week in regards to this.

-- 
Adam McGreggor
General Manager  |  skype: adamamyl  |  @adamamyl
The Open Knowledge Foundation
Empowering through Open Knowledge
http://okfn.org/  |  @okfn