[ckan-dev] Views on data through the web

Wed Jun 15 23:25:19 UTC 2011

Yep, I hope we can share the SQL data server.

I think the core of it is the same for each of us - and one day we'll
both want multiple server scaling of some sort, and no doubt lots of
other things.

It would make sense for it to be a separate open source project.

Quite surprising that we've both settled on SQLite for this...
(Everybody else busy talking about CouchDB and MongoDB...)

Francis

On Wed, Jun 15, 2011 at 07:28:08PM +0100, Rufus Pollock wrote:
> Hi Seb,
> 
> Thanks for the pointer. We've been talking to the Scraperwiki folks a
> lot and recently were discussing if there was a possibility of sharing
> some code libs even :-)
> 
> For some time (e.g. 1y+!) we have known that we want to integrate some
> kind of datastore / data processing system with CKAN. As you may
> recall we've had a CREP in progress on this for some months:
> 
> <http://bitbucket.org/okfn/ceps/src/tip/cep/cep-0004.txt>
> 
> I think we are now close to having an agreed way ahead. I think we can
> distinguish 3 modules that are needed:
> 
> 1. A datastore with dataapi.
> 
> Suggestion is this would be sqlite based with a simple sql based API.
> http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}
> 
> 2. A data processing system which utilizes this datastore. One could
> get a long way with simple javascript running in the browser for
> development with this javascript then run offline using something like
> nodejs. Alternatively one could allow one to specify a url to e.g. a
> python file which would then be run in a sandbox (with access to some
> specified set of python modules)
> 
> 3. A bootstrap process to convert CKAN resources into new resources
> stored in our datastore so that they are accessible via the API.
> 
> Rufus
> 
> On 15 June 2011 14:40, Seb Bacon <seb.bacon at gmail.com> wrote:
> > Hi,
> >
> > There's been some discussion about through-the-web data wrangling.
> >
> > I'd not looked at scraperwiki for a while and just found out about
> > their views, e.g.
> >
> >   http://scraperwiki.com/docs/python/python_view_guide/
> >
> > Here's an example view done with HTML/Javascript:
> >
> >   http://scraperwikiviews.com/run/hmrc_spending_pie_chart_date_slider/
> >
> > Seb
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >
> 
> 
> 
> -- 
> Co-Founder, Open Knowledge Foundation
> Promoting Open Knowledge in a Digital Age
> http://www.okfn.org/ - http://blog.okfn.org/
> 