[ckan-dev] Views on data through the web

Wed Jun 15 18:28:08 UTC 2011

Hi Seb,

Thanks for the pointer. We've been talking to the Scraperwiki folks a
lot and recently were discussing if there was a possibility of sharing
some code libs even :-)

For some time (e.g. 1y+!) we have known that we want to integrate some
kind of datastore / data processing system with CKAN. As you may
recall we've had a CREP in progress on this for some months:

<http://bitbucket.org/okfn/ceps/src/tip/cep/cep-0004.txt>

I think we are now close to having an agreed way ahead. I think we can
distinguish 3 modules that are needed:

1. A datastore with dataapi.

Suggestion is this would be sqlite based with a simple sql based API.
http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

2. A data processing system which utilizes this datastore. One could
get a long way with simple javascript running in the browser for
development with this javascript then run offline using something like
nodejs. Alternatively one could allow one to specify a url to e.g. a
python file which would then be run in a sandbox (with access to some
specified set of python modules)

3. A bootstrap process to convert CKAN resources into new resources
stored in our datastore so that they are accessible via the API.

Rufus

On 15 June 2011 14:40, Seb Bacon <seb.bacon at gmail.com> wrote:
> Hi,
>
> There's been some discussion about through-the-web data wrangling.
>
> I'd not looked at scraperwiki for a while and just found out about
> their views, e.g.
>
>   http://scraperwiki.com/docs/python/python_view_guide/
>
> Here's an example view done with HTML/Javascript:
>
>   http://scraperwikiviews.com/run/hmrc_spending_pie_chart_date_slider/
>
> Seb
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>

-- 
Co-Founder, Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/