[ckan-dev] SQLite removal?

David Raznick david.raznick at okfn.org
Fri Jun 14 21:26:42 UTC 2013


On Fri, Jun 14, 2013 at 3:59 PM, David Read
<david.read at hackneyworkshop.com>wrote:

> The time taken on postgres is nothing to do with CKAN core or the
> specific tests. It is a result of postgres dropping dozens of tables
> and recreating them. I think when David did the current postgres test
> setup it was in haste and it is clear to anyone reading logging during
> tests that it could be improved somewhat. So please do improve it! :-)
>

I also did the current sqlite test setup if that is the case and there was
a lot of pain in doing it and is why I still want it removed.

>From what I remember there where compromises in making the postgres setup
as fast as it could be due to using sqlite as well.
I distinctly remember making the postgres tests start up faster but not
being able to due to the complication of having to support sqlite as well.



> But just in case that takes another 3 years before that is complete,
> may I suggest keeping sqlite in the meantime? ;-)
>
> Interesting idea about 'defaulting' developers to using a test mode
> that uses sqlite for some tests and postgres for others. Sounds like a
> can of worms though, judging by the existing debate on sqlite.
>
> David
>
>
> On 14 June 2013 15:15, Sean Hammond <sean.hammond at okfn.org> wrote:
> >> > Right, I'll often run the specific test modules that the code I'm
> >> working on affects frequently as I'm developing, but only run the whole
> >> test suite at the end before merging to master (if at all, Travis will
> >> often have run it for me).
> >>
> >> Great, you're finally addressing the situation that I'm talking about
> >> (I'm in agreement about not running the full tests - leave it to
> >> Travis).
> >>
> >> As per my example, postgres takes 12s versus sqlite 3s for running a
> >> couple of simple tests - the ones relevant to the code currently being
> >> written. As you add a couple of lines of code and add a new assert to
> >> the test in parallel with the feature, you'll find yourself running
> >> the relevant tests several times a minute. Waiting 12s for them to run
> >> each time and you simply break your flow.
> >
> > I'm not convinced about this. You're telling me that the unit tests for
> > one module takes 12s to run with postgres and 3s with sqlite? And this
> > isn't because either the tests or the ckan module is badly designed?
> >
> > Anyway, from my point of view, there should only be one way of running
> > the tests and it should be fast. As discussed introducing "run the tests
> > with postgres" and "run the same tests, but with sqlite" creates various
> > problems. I think the right way to do this might be to write more
> > testable code that's easier and faster to test in the first place, and
> > then write better and faster tests for it.
> >
> > For example, rather than either running all the tests with postgres and
> > then having a different command to run all the tests with sqlite
> > instead, a particular CKAN module that uses the database should accept a
> > database object as parameter, in normal operation it would be passed a
> > postgres database object but the tests could pass it an in-memory sqlite
> > database object instead.
> >
> > If we used this approach throughout then you would have just one command
> > "run the tests" and each test would use either a postgres database
> > (when really necessary) or a faster sqlite one whenever possible.
> > Hopefully the tests overall would be pretty fast, and running the tests
> > for most individual modules would be very fast and not touch the disk at
> > all. And no problem with having two separate ways of running the tests.
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130614/78e7e767/attachment-0001.html>


More information about the ckan-dev mailing list