[ckan-dev] RFC: use oktest for ckan

[Sean Hammond]

> I'm currently working on integration tests for CKAN. Before that I
> worked on the datastore and wrote quite a few tests. Testing takes a
> lot of time and the syntax and fixture injection is not simple. In my
> Summer of Code Projekt and other project at university, I used Oktest
> and would like to show you the advantages.

Would using oktest mean we would have to replace nosetest? Or can it be
integrated with our existing nose tests?

> * Shorter syntax and easier to read syntax

Can you give some examples about what you don't like about nosetests?
I'm not sure I'm convinced that oktest is a lot better. To me it looks
like:

oktest
------

@test("1 + 1 should be 2")
def _(self):
    ok(1+1) == 2

nosetests
--------

@test("1 + 1 should be 2")
def test_one_plus_one_is_two(self):
    "1 + 1 should be 2"
    assert 1 + 1 == 2

I don't see a huge difference there, and the rest of the ok() examples
also seem to map to assert statements in the same way. The @test
decorator seems to do a similar thing to what docstrings on test methods
do in nose.

> * Better errors with diffs[1]

nosetests will try to automatically generate more informative error
messages if you pass it the --detail-errors option, example:

======================================================================
FAIL: test_foo_is_two (ckan.tests.test_plugins.TestInterface)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/Projects/ckan/ckan/ckan/tests/test_plugins.py", line 68, in test_foo_is_two
    assert FOO == 2
AssertionError: 
    "Test that FOO is 2"
>>  assert 3 == 2

It shows me what the value of FOO really was (3). Not sure how well it
does with more complex examples.

You can also put something after a comma at the end of the assert to
tell nose what to print out when the test fails, e.g.:

    def test_that_foo_is_two(self):
        "Test that FOO is 2"
        assert FOO == 2, "FOO should be 2, but was {0}".format(FOO)

but most of the time people don't bother to do this with their asserts
and to be honest I don't really like it, it's extra work and is error
prone. Maybe just --detailed-errors is better.

Anyway, it could be that oktests diff output is better than nosetest's
for complex cases.

The fixture injection might be useful..