[ckan-dev] Work on the ckanext-datapreview plugin

Thu Jul 14 01:51:30 UTC 2011

On 13 July 2011 12:07, Aron Carroll <aron.carroll.lists at gmail.com> wrote:
>
> Hi all,
> I'm just posting a quick update on some work I'm doing on the data preview CKAN plugin (https://bitbucket.org/okfn/ckanext-datapreview).
>
> I've fixed a couple of minor issues with the plugin not handling failed requests to the data proxy API (as illustrated here http://thedatahub.org/package/fukushima-radiation-monitoring-greenpeace) as well as the formatting of the error messages. I've added these fixes in a separate fork of the project which can be found on Bitbucket, https://bitbucket.org/aron_/ckanext-datapreview.
>
> I'm now looking into implementing the ability to create graphs from CSV data and could do with some advice on various issues.

Just pushed up a new mini-lib encapsulating data stuff and suggest we
put generic table / graphing stuff in there (and then tie in to
specific CKAN use case in data-preview plugin).

<https://github.com/okfn/datajs>

> 1. How to generate the graphs? It seems to me the best way to create useful graphs is to get users to generate them via a simple creation tool similar to the one on socrata.com.

I've written a basic version of line charting stuff before (first time 4y ago):

<http://old.openeconomics.net/store/>
<http://old.openeconomics.net/store/view/8d7d4770-e1d1-11db-9f7e-00145101c316>

A newer version here:

<http://eurostat.dev.okfn.org/>

Suggest using flot and a small bit of ui to select fields. At the
start let's just aim for time series or similar. Basic story of
graphing is that almost immediately you need the user to specific
stuff so let's do absolute simplest possible. (We can also think about
integrating with google chart stuff and/or any other lib you think
of).

> 2. How to store the graphs? Can they be saved alongside the resource data in CKAN?

I'd suggest we store either a) the relevant config info (e.g. this
column is x, this is y etc etc) or b) some javascript that gets run
(probably better medium term). In short-term we can store this on
resource object metadata. medium-term we probably want something more
like a proper view object but that can wait.

> 3. Where to find test data? Currently most of the CSV data is archived in zip or tar files and currently inaccessible so I'm having trouble finding useful data that can be actually be plotted on a graph. If anyone has any suggestions for test datasets I'd be extremely grateful.

I suggest using csv's from here: <http://old.openeconomics.net/store/>
(i've been meaning to transfer these to thedatahub.org for last year
but now will really do it. Have already transferred some e.g.
<http://thedatahub.org/package/gold-price>

> So from here I'm going to try and add a widget to the data preview lightbox that creates a simple line graph from selected columns and move from there. If anyone has any suggestions for features that they'd like to see implemented I'd love to hear them.

Sounds good.

Rufus

PS: I'll aim to be up and online at 8am my time (1pm your time) so we
can catch up in person ...