[ckan-dev] Fwd: Data processing in the browser: codenode-ng

Mon Jun 20 12:22:44 UTC 2011

On 20 June 2011 11:45, David Read <david.read at okfn.org> wrote:

> The screenshot suggests some cool markdown extension for equations and
> python code, with some javascript that renders the output and runs the
> code. Sort of like live Sphinx, or the Scraperwiki code editor. Sounds
> great, but where do you see the connection with CKAN?

Data processing/viewing, i.e:
http://lists.okfn.org/pipermail/ckan-dev/2011-June/000893.html

The python code is getting executed by an engine server, which has a
long running engine process.  These engine processes can be any
executable, but what I imagine is that we would provide a python
environment with CKAN libs. So I might fire up my browser, munge two
datasets and then publish the result back to CKAN - doing it on
codenode would allow: transparency (you can see what I have done),
repeatability (you can rerun my code and test it), data-local
computation (run the code near the data).  You could use some kind of
queue, but the engine process allows interaction with the data - since
you can interact with the engine while running.