[kforge-dev] failing tests

[John Bywater]

Just to say that with Matthew Brett's trial KForge deployment yesterday, 
two of the tests in the kforge.test.core test suite failed.

One was a failure to call 'apachectl'. Another was a failure to send 
email. I don't have the details, but I'm sure it will repeat when we 
reinstall with today's release. I'll get the details then so we can sort 
it out.

Both calling apachectl and sending email are strange things for unit 
tests to do.

I'm sure you agree that we must maintain the distinction between 
"developer tests" which are "sub-cutaneous" and run fairly independently 
of the operating system environment and the configuration of third party 
applications (in this case whether or not you have apachectl and email), 
and "customer test" a.k.a. "black box" tests which invoke the 
application through it's external interface and therefore very much rely 
on configuration of web servers, email servers, and whatever else needs 
to be tested for acceptance (for general public release, in this case).

In the case of the KForge test suite, we also distinguish between tests 
on the core system, and tests on plugins (assumed to be the core plugins 
developed by the KForge project, not any third party plugins).

But I feel these two distinctions are becoming lost. Can I propose we 
restore the original conception, so that we have:

kforge-test kforge.test.all.developer (all unit tests)
kforge-test kforge.test.all.customer (all acceptance tests)

kforge-test kforge.test.core.developer (kforge unit tests)
kforge-test kforge.test.core.customer (kforge acceptance tests)
kforge-test kforge.test.plugins.developer (plugin unit tests)
kforge-test kforge.test.plugins.customer (plugin acceptance tests)

and not too much else?

The script kforge-test could then either default to running the suite 
kforge.test.core.developer or kforge.test.all.developer depending on 
whether or not the plugins are considered to be inside the system boundary.

We might also wish to distinguish between "core plugins" which are 
really part of the core and "non-core plugins" and which shouldn't be 
involved in testing the core. But I would hope not.

But we must try to maintain the distinction between unit tests and 
blackbox tests.

Best wishes,

John.



-- 
Appropriate Software Foundation
Registered in England and Wales
17 Chapel Street, Hyde Cheshire
Company number: 04977110
W: appropriatesoftware.net
T: 0870 720 2944
M: 0781 139 2292