[kforge-dev] failing tests

[Rufus Pollock]

John Bywater wrote:
> Just to say that with Matthew Brett's trial KForge deployment yesterday, 
> two of the tests in the kforge.test.core test suite failed.
> 
> One was a failure to call 'apachectl'. Another was a failure to send 
> email. I don't have the details, but I'm sure it will repeat when we 
> reinstall with today's release. I'll get the details then so we can sort 
> it out.
> 
> Both calling apachectl and sending email are strange things for unit 
> tests to do.

I agree and the apachectl was a recent addition aimed at addressing the 
difficulty of providing adequate test coverage for the apacheconfig module.

On the mail sending front I think we should address that by providing a 
stub for the mail sending facility (perhaps we could override/disable 
the mail sending facility -- kforge.utils.mailer -- for the tests?)

> I'm sure you agree that we must maintain the distinction between 
> "developer tests" which are "sub-cutaneous" and run fairly independently 
> of the operating system environment and the configuration of third party 
> applications (in this case whether or not you have apachectl and email), 
> and "customer test" a.k.a. "black box" tests which invoke the 
> application through it's external interface and therefore very much rely 
> on configuration of web servers, email servers, and whatever else needs 
> to be tested for acceptance (for general public release, in this case).

I completely agree and think we could perhaps move all the apacheconfig 
and mailer stuff (and perhaps all plugin stuff) into customer tests.

> In the case of the KForge test suite, we also distinguish between tests 
> on the core system, and tests on plugins (assumed to be the core plugins 
> developed by the KForge project, not any third party plugins).

I think we could go further and distinguish between truly core plugins 
that are required for the system (e.g. apacheconfig) and 
'service/subsystem application' plugins (trac, svn, moinmoin etc etc).

> But I feel these two distinctions are becoming lost. Can I propose we 
> restore the original conception, so that we have:
> 
> kforge-test kforge.test.all.developer (all unit tests)
> kforge-test kforge.test.all.customer (all acceptance tests)
> 
> kforge-test kforge.test.core.developer (kforge unit tests)
> kforge-test kforge.test.core.customer (kforge acceptance tests)
> kforge-test kforge.test.plugins.developer (plugin unit tests)
> kforge-test kforge.test.plugins.customer (plugin acceptance tests)
> 
> and not too much else?

This seems like a great idea.

> The script kforge-test could then either default to running the suite 
> kforge.test.core.developer or kforge.test.all.developer depending on 
> whether or not the plugins are considered to be inside the system boundary.
> 
> We might also wish to distinguish between "core plugins" which are 
> really part of the core and "non-core plugins" and which shouldn't be 
> involved in testing the core. But I would hope not.

See my comments above. In my opinion all service plugins are dodgy from 
a developer point of view since they won't work unless trac/svn/moinmoin 
/wordpress is actually installed. At the same time leaving them out 
makes it much more likely that changes break them without noticing. The 
solution here I think is to move them out of the developer tests but 
ensure there is at least one machine (e.g. the dedicated test machine at 
test.kforgeproject.com) which has a full deployment setup and can 
therefore test everything.

> But we must try to maintain the distinction between unit tests and 
> blackbox tests.

Yes, though we also here have a distinction between tests that require 
external services to run (apache, svn, moinmoin an email system ...) and 
those that don't (or is all of this type of stuff blackbox/functional 
testing?).

Regards,

Rufus