[ckan-dev] got chance to talk about webstore?

Mon Oct 3 08:08:10 UTC 2011

Hi Julian,

first off: let me apologize - I was on a weird conference marathon
last week and could not get back to you sooner. As you probably know,
the CKAN team now has also started working more intensely on webstore
(they may have gotten in touch with you). So as a general plan I would
propose we coordinate mainly through the github issues tracker on
okfn/webstore in the future in order not to duplicate effort or to
diverge to much. Perhaps we could all fill this up and then have a
Skype call to discuss it later this week?

On Thu, Sep 29, 2011 at 9:29 PM, Julian Todd <julian at goatchurch.org.uk> wrote:
> Friedrich,
> I've now done the protocols to implement save() and execute() working from
> all 3 scraperwiki languages connecting to webstore.  See for example:
>
>   https://bitbucket.org/ScraperWiki/scraperwiki/src/334c1a59764b/scraperlibs/ruby/scraperwiki/lib/scraperwiki.rb#cl-196

Brilliant -)

> Clearly we would like to have it so we install a standard client for each
> language (like the webstore-client-0.3.2), but backwards compatibility with
> the current scraperlibs library as well as working out a way of getting the
> configurations working slows this down.
> Recall we were thinking of inserting the authentication values into the
> headers at the httpproxy level, which makes it even more complicated (unless
> we think of a more standard way of doing it).

I'm not actually sure this makes it harder - seems to me it removes a
concern from webstore-client to deal with and that seems like a nice
idea.

> One of the main motivations for using the webstore-client that comes with
> the package is that it you can control both ends of the protocol and
> gradually improve its efficiency without needing us to do
> re-implementations.  As you know, I don't think much of the protocol, with
> its PUT commands and "Accept: json" things in the headers, so I would like
> to make it as easy as possible to evolve into something more sane.
>
> For the record, a very good place to begin is at line 356
> of webstore-client-0.3.2/webstore/client/__init__.py, in the function
> writerows() where it says:
> if self.exists():
>     res = self._request("PUT", query, rows)
> else:
>     res = self._request("POST", query, rows)
> [self is type Table]
> Now, the exists() function is implemented with a GET ?_limit=0, which makes
> an http request of its own to the webstore and has to reinterpret
> WebstoreClientException into boolean False when the table doesn't exist.
> The receiving function of this request is in webstore/webstore/views.py line
> 186 upsert(), which accepts both "POST" and "PUT" anyway, and doesn't even
> notice if it is creating a new table in the line:
>    _table = db[table]
> due to the implementation of __getitem__ in DatabaseHandler, which
> automatically creates a table if one doesn't exist.

All of this is gone in the github version of webstore-client and the
okfn branch of webstore. But it was a problem, yes. And we should
really push out a client 0.3.3 to PyPI!

> When you pull the code I have committed onto git under scraperwiki you will
> see the problem I had with this lazy creation of tables -- it makes it
> impossible to pass parameters into the constructor:
>
>   https://github.com/scraperwiki/webstore/commit/ed99d4db866d51398e225c1d55fa46ac94b4a0b7#L3R202
>
> In other words, this implementation has made a special case for creating a
> new table where it doesn't matter, and no special case in the one place
> where it does matter!
>
> Anyways, we've got a few things to tick off, like a better way of passing in
> the jargtypes argument, and an explanation for the conversion to strings of
> all the values.

Yes, looking at this commit I think there are a few things that I'd comment on:

* So far, webstore has really good test coverage. I hate tests but it
works well to ensure quality. Can we test this feature?
* I have no idea what "jargtypes" means - I assume it is some
combination of "javascript argument types". Why not call it
column_types?
* The jargtypelookup will KeyError if I use an unknown type. Maybe we
can make this fall back or at least catch the error and emit an HTTP
400 (Bad Request) exception.
* Can we uppercase constants? jargtypelookup is what I mean.
* Can you add documentation on this to the docs (doc/index.rst)?
* not everyone has "/home/julian/scraperwiki/" on their computer ;-)

> But as a higher priority, we're going to try and convert over to
> webstore very soon on account of my current dataproxy breaking down under
> the load (because of its socket implementation).
> We're not using the webstore API yet, but that bit we can survive for the
> moment until this bit is done.
> Would it be appropriate to add various feature requests into the following
> page, or do feature requests get pulled back if I added them to the
> scraperwiki fork?
>     https://github.com/okfn/webstore/issues

As I said: lets manage features/issues together and push/pull frequently.

- Friedrich

-- 
Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/

http://twitter.com/pudo
http://pudo.org