[kforge-dev] failing tests

[John Bywater]

More for the list than anything..........

Rufus Pollock wrote:
> John Bywater wrote:
>> Just to say that with Matthew Brett's trial KForge deployment 
>> yesterday, two of the tests in the kforge.test.core test suite failed.
>>
>> One was a failure to call 'apachectl'. Another was a failure to send 
>> email. I don't have the details, but I'm sure it will repeat when we 
>> reinstall with today's release. I'll get the details then so we can 
>> sort it out.
>>
>> Both calling apachectl and sending email are strange things for unit 
>> tests to do.
>
> I agree and the apachectl was a recent addition aimed at addressing 
> the difficulty of providing adequate test coverage for the 
> apacheconfig module.

No worries. As we discussed a few days ago now, I've separated all the 
tests into developer and customer, and put the test with the apachectl 
call in the customer tests.

>
> On the mail sending front I think we should address that by providing 
> a stub for the mail sending facility (perhaps we could 
> override/disable the mail sending facility -- kforge.utils.mailer -- 
> for the tests?)

I refactored the class that was calling kforge.utils.mailer so that the 
call was made inside a dedicated method, then sub-classed to create a 
class with that method stubbed out, and then substituted that new class 
in the test for the real class. Test runs OK.

I would have preferred to have stubbed the mailer utility: the best way 
may have been to register the mailer as an application feature, then 
register a stub mailer with the TestApplicationBuilder instead of the 
real one, so the whole application-under-test sends any and all email to 
a stub service. But since kforge.utils.mailer isn't really an object 
oriented boundary object, I couldn't and so I wrote a to-do instead. :-)

>> I'm sure you agree that we must maintain the distinction between 
>> "developer tests" which are "sub-cutaneous" and run fairly 
>> independently of the operating system environment and the 
>> configuration of third party applications (in this case whether or 
>> not you have apachectl and email), and "customer test" a.k.a. "black 
>> box" tests which invoke the application through it's external 
>> interface and therefore very much rely on configuration of web 
>> servers, email servers, and whatever else needs to be tested for 
>> acceptance (for general public release, in this case).
>
> I completely agree and think we could perhaps move all the 
> apacheconfig and mailer stuff (and perhaps all plugin stuff) into 
> customer tests.

Cool. That's what we've got now.

>> In the case of the KForge test suite, we also distinguish between 
>> tests on the core system, and tests on plugins (assumed to be the 
>> core plugins developed by the KForge project, not any third party 
>> plugins).
>
> I think we could go further and distinguish between truly core plugins 
> that are required for the system (e.g. apacheconfig) and 
> 'service/subsystem application' plugins (trac, svn, moinmoin etc etc).

Yes, we should definitely make sure we're considering the former as part 
of the "core" system.

>> But I feel these two distinctions are becoming lost. Can I propose we 
>> restore the original conception, so that we have:
>>
>> kforge-test kforge.test.all.developer (all unit tests)
>> kforge-test kforge.test.all.customer (all acceptance tests)
>>
>> kforge-test kforge.test.core.developer (kforge unit tests)
>> kforge-test kforge.test.core.customer (kforge acceptance tests)
>> kforge-test kforge.test.plugins.developer (plugin unit tests)
>> kforge-test kforge.test.plugins.customer (plugin acceptance tests)
>>
>> and not too much else?
>
> This seems like a great idea.

As we discussed a little while ago, I did something along those lines a 
little while ago.

>> The script kforge-test could then either default to running the suite 
>> kforge.test.core.developer or kforge.test.all.developer depending on 
>> whether or not the plugins are considered to be inside the system 
>> boundary.
>>
>> We might also wish to distinguish between "core plugins" which are 
>> really part of the core and "non-core plugins" and which shouldn't be 
>> involved in testing the core. But I would hope not.
>
> See my comments above. In my opinion all service plugins are dodgy 
> from a developer point of view since they won't work unless 
> trac/svn/moinmoin /wordpress is actually installed. At the same time 
> leaving them out makes it much more likely that changes break them 
> without noticing. The solution here I think is to move them out of the 
> developer tests but ensure there is at least one machine (e.g. the 
> dedicated test machine at test.kforgeproject.com) which has a full 
> deployment setup and can therefore test everything.

Indeed, and that sounds sensible.

>> But we must try to maintain the distinction between unit tests and 
>> blackbox tests.
>
> Yes, though we also here have a distinction between tests that require 
> external services to run (apache, svn, moinmoin an email system ...) 
> and those that don't (or is all of this type of stuff 
> blackbox/functional testing?).

Ideally.... all the system would be exercised both with developer tests 
and with customer tests. The customer shouldn't "trust" the developer 
tests to verify what the customer requires, and the developers shouldn't 
be stuck with whatever the customer happens to feel comfortable with 
knowing from outside the system.

So each of the plugins should be unit tested with mock or stub objects 
(depending on one's testing style) such that they in fact don't depend 
on external systems. Then they should also be black box tested as 
normally running objects that potentially require external systems to be 
available, even if those external systems are themselves provide only 
mocked up, or stubbed out services.

Having said all that, we should also try to keep it simple! :-)

Bye for now,

John.



>
> Regards,
>
> Rufus
>
>
>


-- 
Appropriate Software Foundation
Registered in England and Wales
17 Chapel Street, Hyde Cheshire
Company number: 04977110
W: appropriatesoftware.net
T: 0870 720 2944
M: 0781 139 2292