[okfn-labs] [idea-rfc]: DataPipes - Streaming Online Data Transformation!

[Rufus Pollock]

On 7 May 2013 09:05, Michael Bauer <michael.bauer at okfn.org> wrote:
> Hi,
>
> Currently datapipes.okfn.org leads me to okfn.org/register

It was http://datapipes.okfnlabs.org (okfnlabs.org) right ;-)

> I do like the idea of unix pipes - however it's hard for me to imagine it
> working on a web level. Wouldn't this be just a transforming version of the
> data proxy? Where I have a source, several steps and an output?

DataProxy = data format conversion is indeed very similar and you
could definitely merge these two ideas into one general transformation
service.

> Will you allow me to use my own scripting to define new elements in
> pipes? e.g. a new transformation in the pipeline?

See https://github.com/okfn/datapipes/issues/3 (which I believe you
have already discovered). Basic point: you can do something basic but
if you really want arbitrary stuff I recommend your own server, or
scraperwiki X or ... :-)

> At one point we will reach the issue of running proprietary untrusted code
> on our server (with map, reduce and filter functions) - I'd propose to use
> a language that supports proper sandboxing underneath. (Not sure node.js
> does so). Also I don't like javascript as a data handling language: it's
> simply not designed for it (yes I'm a hopeless lamdahead).
>
> Right now the pipe is simply one block, combining multiple pipes seems
> painfull - can we make this easier?

Good point - see this comment and subsequent discussion (this probably
merits a separate issue):

https://github.com/okfn/datapipes/issues/2#issuecomment-17533202

Regards,

Rufus