[ckan-dev] SQLite removal?

Fri Jun 14 14:59:36 UTC 2013

The time taken on postgres is nothing to do with CKAN core or the
specific tests. It is a result of postgres dropping dozens of tables
and recreating them. I think when David did the current postgres test
setup it was in haste and it is clear to anyone reading logging during
tests that it could be improved somewhat. So please do improve it! :-)

But just in case that takes another 3 years before that is complete,
may I suggest keeping sqlite in the meantime? ;-)

Interesting idea about 'defaulting' developers to using a test mode
that uses sqlite for some tests and postgres for others. Sounds like a
can of worms though, judging by the existing debate on sqlite.

David

On 14 June 2013 15:15, Sean Hammond <sean.hammond at okfn.org> wrote:
>> > Right, I'll often run the specific test modules that the code I'm
>> working on affects frequently as I'm developing, but only run the whole
>> test suite at the end before merging to master (if at all, Travis will
>> often have run it for me).
>>
>> Great, you're finally addressing the situation that I'm talking about
>> (I'm in agreement about not running the full tests - leave it to
>> Travis).
>>
>> As per my example, postgres takes 12s versus sqlite 3s for running a
>> couple of simple tests - the ones relevant to the code currently being
>> written. As you add a couple of lines of code and add a new assert to
>> the test in parallel with the feature, you'll find yourself running
>> the relevant tests several times a minute. Waiting 12s for them to run
>> each time and you simply break your flow.
>
> I'm not convinced about this. You're telling me that the unit tests for
> one module takes 12s to run with postgres and 3s with sqlite? And this
> isn't because either the tests or the ckan module is badly designed?
>
> Anyway, from my point of view, there should only be one way of running
> the tests and it should be fast. As discussed introducing "run the tests
> with postgres" and "run the same tests, but with sqlite" creates various
> problems. I think the right way to do this might be to write more
> testable code that's easier and faster to test in the first place, and
> then write better and faster tests for it.
>
> For example, rather than either running all the tests with postgres and
> then having a different command to run all the tests with sqlite
> instead, a particular CKAN module that uses the database should accept a
> database object as parameter, in normal operation it would be passed a
> postgres database object but the tests could pass it an in-memory sqlite
> database object instead.
>
> If we used this approach throughout then you would have just one command
> "run the tests" and each test would use either a postgres database
> (when really necessary) or a faster sqlite one whenever possible.
> Hopefully the tests overall would be pretty fast, and running the tests
> for most individual modules would be very fast and not touch the disk at
> all. And no problem with having two separate ways of running the tests.
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev