[wdmmg-dev] [ckan-dev] Started work on the 'webstore' (a datastore with web API)

Thu Jul 7 17:26:01 UTC 2011

Hi all,

here's a quick update on the webstore: the basic service is now
functional and should appear in a DNS near you at
webstore.thedatahub.org soon. Too see a simple example, here are the
first 10 lines of a budget dataset:

http://webstore.thedatahub.org/pudo/tests/italy_sample

You can do some nice tricks on this (https://github.com/okfn/webstore
has more), e.g.:

http://webstore.thedatahub.org/pudo/tests/italy_sample/distinct/function
http://webstore.thedatahub.org/pudo/tests/italy_sample/row/5

If you don't like the ugly HTML, use CSV or JSON:

http://webstore.thedatahub.org/pudo/tests/italy_sample.json

The service allows users to upload to their own namespace,
authentication is delegated to ckan.net (erm, thedatahub.org), so your
normal credentials should work if put into an HTTP Basic header.

Would appreciate any feedback!

- Friedrich

p.s. For people who think 10 lines is crap, here is 1.8 million:

http://webstore.thedatahub.org/pudo/uk25k/spending.csv

Please only use CSV, as streaming JSON is todo. ?_limit=100 is your friend.

On Tue, Jul 5, 2011 at 7:56 PM, Friedrich Lindenberg
<friedrich.lindenberg at okfn.org> wrote:
> Hi,
>
> just to keep everyone here updated: I've posted a set of use cases
> related to the datastore to http://wiki.ckan.net/Data_Wrangling (they
> are a bit informal but intended as reminders).
>
> I've also done a rewrite of the webstore that is based on
> Flask/SQLalchemy and offers a RESTful interface including support for
> CSV, JSON and HTML representations.
>
> How this relates to ckanext-qa and the wider CKAN system will be
> discussed tomorrow.
>
> - Fr.
>
> On Mon, Jul 4, 2011 at 8:54 PM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>> On 4 July 2011 19:50, James Casbon <casbon at gmail.com> wrote:
>>> Rufus,
>>>
>>> see inline...
>>>
>>> On 4 July 2011 19:39, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>> Following up earlier discussion [1] of CREP 0004 Data API and Data
>>>> Processing System:
>>>>
>>>> <http://trac.ckan.org/ticket/1190>
>>>
>>> Not CREP! ;)
>>
>> i agreed but was outvoted ;-)
>>
>>> You should probably add sleepy mongoose to your prior art list:
>>> https://github.com/kchodorow/sleepy.mongoose/wiki/
>>
>> That's prior art for the webstore for but not the whole thing I think
>> ... (and i've used it a little bit).
>>
>>> I want to ask about the specific use case you have there and how it
>>> relates to the datastore.
>>> You talk about converting a set of prices using interest rates: is
>>> that happening inside the datastore or in the client javscript in your
>>> vision?
>>
>> In client javascript initially. Idea then being that script could be
>> saved and executed offline (in bulk) using e.g. nodejs. So we use
>> client js for 'editor' and then ship to backend engine for large scale
>> processing.
>>
>> Rufus
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>
>
>
>
> --
> Open Knowledge Foundation
> Promoting Open Knowledge in a Digital Age
> http://www.okfn.org/ - http://blog.okfn.org/
>
> http://twitter.com/pudo
> http://pudo.org
>

-- 
Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/

http://twitter.com/pudo
http://pudo.org