[ckan-dev] Hello

[Seb Bacon]

Hi all,

Thanks for all the feedback.

On 8 December 2010 12:42, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> On 8 December 2010 12:08, David Read <david.read at okfn.org> wrote:
>> Seb,
>>
>> Welcome!
>>
>> +1 to ploughing on getting vdm working in sqlalchemy 0.5.
>
> vdm works with sqlalchemy 0.5 and 0.6 (and has done for a few months) :)

Yup, I found this out shortly after posting my original message.

The problem was in CKAN, not VDM.  Am currently ploughing through all
the tests to make them work against sqlalchemy 0.6 + sqlite :memory.
Will report back in due course -- as ever with fixing lots of tests at
once, it's a bit painful :)

>> Yes there is a warning to do with sqla querying polymorphism that
>> crops up in the CKAN tests. I had a look at this quickly before and
>> couldn't work out how to solve it, so if another pair of eyes on that
>> would be great.
>>
>> The reason CKAN specifies postgres, as opposed to being db neutral, is
>> because of use of the pg's full text searching. There is talk of
>> ditching pg search in favour of SOLR. But requiring SOLR to be
>> installed seems a big hurdle for anyone wanting to install CKAN
>> (indeed I think it could be a faff for every developer running CKAN
>> locally too). I don't think we've gone down this path yet. Maybe it is
>> worth having an option to use SQLite with basic search (as opposed to
>> full text) and skip tests that use full text search.
>
> I definitely think a basic versus complex search is very useful. I'd
> earlier proposed we could get most of this via abstracting the
> SearchQuery interface. I also propose we converge on a search
> parameters that are modelled on solr.

I've not hit these tests yet; for the time being will probably just
put in a flag to skip them if postgres isn't the backend and then
perhaps look at implementing a simple search next, pending further
discussion.

>> I agree the tests take too long. Last time I took a look at it (June)
>> I got the time down from 475s to 275s - I'll forward you the info. FYI
>> tests take 793s (13 minutes) on my machine now. If you're getting 40
>> minutes then maybe you are hitting the ubuntu version bug James hits.

Yes, I think so.  I suspect it's a crappy psycopg2 build (or related)
in this version of ubuntu.

>> Or maybe the profiling you have setup is taking the extra time?
>>
>> I think there are still areas where we use setup when setup_class
>> would be possible and faster. Also I wonder if we can reduce the
>> number of times new ckan instances are started ( _start_ckan_server )
>> as these are very slow.
>
> Those _start_ckan_server come from the Changesets
> (model/changesets.py) tests I believe. Changesets are currently not
> being used in CKAN system at all and I would propose these moved out
> of core (if we want them back in later we can reintegrate).

That's good news :)

This kind of thing would be the next step in optimising the tests so
they run really quickly: consider what isn't core, and also consider
splitting the tests into "complete" and "basic" sets, to enable a
fast-test-cycle during very active development.

Anyway, will keep everyone informed.

This is a short week for me as I get started of only 2 days, so expect
a proper update some time during my next week (i.e. 2 weeks time).

Seb

-- 
skype: seb.bacon
mobile: 07790 939224
land: 0207 183 9618
web: http://baconconsulting.co.uk