[ckan-dev] RFC: use oktest for ckan

[Dominik Moritz]

On Sep 19, 2012, at 13:12 , Sean Hammond <sean.hammond at okfn.org> wrote:

>> I'm currently working on integration tests for CKAN. Before that I
>> worked on the datastore and wrote quite a few tests. Testing takes a
>> lot of time and the syntax and fixture injection is not simple. In my
>> Summer of Code Projekt and other project at university, I used Oktest
>> and would like to show you the advantages.
> 
> Would using oktest mean we would have to replace nosetest? Or can it be
> integrated with our existing nose tests?

It is fully backwards compatible. Notetest runs all oktests as normal tests, even though the oktest test runner makes nicer output. 

> 
>> * Shorter syntax and easier to read syntax
> 
> Can you give some examples about what you don't like about nosetests?
> I'm not sure I'm convinced that oktest is a lot better. To me it looks
> like:
> 
> oktest
> ------
> 
> @test("1 + 1 should be 2")
> def _(self):
>    ok(1+1) == 2
> 
> nosetests
> --------
> 
> @test("1 + 1 should be 2")
> def test_one_plus_one_is_two(self):
>    "1 + 1 should be 2"
>    assert 1 + 1 == 2
> 
> I don't see a huge difference there, and the rest of the ok() examples
> also seem to map to assert statements in the same way. The @test
> decorator seems to do a similar thing to what docstrings on test methods
> do in nose.

As I said, it's not oktest or nose but more nose + oktest. I just think that oktest wraps assertions nicely and in a more semantic way that enables better assertion errors. Also it is more readable and can be written faster. In this 1+1 example the difference is not that significant but it becomes more interesting when you have larger tests.

> 
>> * Better errors with diffs[1]
> 
> nosetests will try to automatically generate more informative error
> messages if you pass it the --detail-errors option, example:
> 
> ======================================================================
> FAIL: test_foo_is_two (ckan.tests.test_plugins.TestInterface)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/home/seanh/Projects/ckan/ckan/ckan/tests/test_plugins.py", line 68, in test_foo_is_two
>    assert FOO == 2
> AssertionError: 
>    "Test that FOO is 2"
>>> assert 3 == 2
> 
> It shows me what the value of FOO really was (3). Not sure how well it
> does with more complex examples.

I didn't know that but that is exactly what oktest does automatically. It evaluates expressions and shows the actual values.

> You can also put something after a comma at the end of the assert to
> tell nose what to print out when the test fails, e.g.:
> 
>    def test_that_foo_is_two(self):
>        "Test that FOO is 2"
>        assert FOO == 2, "FOO should be 2, but was {0}".format(FOO)
> 
> but most of the time people don't bother to do this with their asserts
> and to be honest I don't really like it, it's extra work and is error
> prone. Maybe just --detailed-errors is better.

--detailed-errors should be the default ;-)

> Anyway, it could be that oktests diff output is better than nosetest's
> for complex cases.
> 
> The fixture injection might be useful..
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev