[ckan-dev] Notes from CKAN test suite discussion (OKF summit)

[Sean Hammond]

Here are my notes from our discussion at the OKF summit about what to do
with CKAN's test suite.

There's still a lot of work to be done on this: writing example tests of
different kinds (logic tests, controller tests, model and lib tests..)
and writing up new testing guidelines/coding standards.

Toby, Vitor, Dominik and I started work on some logic tests at the
summit but we didn't have time to finish them. I've worked on them a bit
more since, but unfortunately progress has been hindered by CKAN
freezing whenever I try to clean the db, I'll have to get to the bottom
of that first before I can continue.

Anyway, here are my notes on what we discussed and agreed at the summit:

Todo
----

1. Finish off writing the example tests for each package and testing
   coding standards (Toby, Vitor, Sean - I'm happy to lead on this as
   I've already done some work at the summit and on my way home)

2. Write tests to cover the frontend - this wasn't assigned to anyone,
   but we agreed it was important to block some time in the short term
   for someone to write a whole bunch of tests to cover the frontend.

   Based on the discussion below, I think this would mean functional
   tests, using the new templates, for every controller method in
   ckan.controllers.

   Point 1 above really needs to be done before 2 can be done.


Strategy for moving to a new test suite
---------------------------------------

We agreed that the existing test suite is a lost cause and we just want
to start fresh with a new test suite, and incrementally transition from
the old one to the new one:

1. Rename ckan/tests -> ckan/legacy_tests
2. Create new dir ckan/tests
3. Publish coding standards and examples for new-style tests
4. Going forward, all new or changed code should have new-style tests.
   As changes to code break old tests, replace them with new-style
   tests.

New tests
---------

The questions we tried to answer about the new test suite were:

1. What should be tested? i.e. which parts of the source code, and which
   functions and methods, should have tests? See below for the answers
   to this.

2. How should the tests be organized?

   We just decided to mirror the organization of the source code in the
   tests dir, so all the tests for ckan/logic/action/update.py will go
   in ckan/tests/logic/action/update.py and so on.

   There will be some exceptions to this, e.g. PEP8 tests and coding
   standards tests, but there should only be a few of these exceptional
   test modules and they can just go in the top-level ckan/tests/ dir.

3. How detailed should the tests for a given function or method be?
   We didn't discuss this.

4. How should the tests be written?
   The answer is different for different tests, e.g. controller tests
   will work differently from logic tests. The notes below touch on this
   question slightly but the answers really need to be fleshed out by
   writing example tests for each package within CKAN.

Main points that we agreed on were:

* Kill sqlite, have just one way of running the tests and make it fast.
  We didn't assign this to anyone - who wants to do it?

* Answering question 1 (what should be tesed?), we went over each source
  directory in CKAN:

  - ckan.logic.action: all action functions should have tests, we'll test by
    calling the action functions directly, passing ignore_auth=True in the
    context because we want to test the auth functions separately

    - There need to be some functional tests (test action functions by HTTP
      posting) as well, will this be covered by the API controller's tests?

  - ckan.logic.auth: all auth functions should have tests, we'll call the auth
    functions directly

    - There will be a test that every action function does call its auth
      function (or can we just do this in get_action()?)

  - converters.py, validators.py: all funcitons in these should have tests,
    we'll test by calling the functions directly

  - schema.py: validate() itself should have its own unit tests, but other than
    that the schemas will be tested indirectly when we're testing the action
    functions. It would be nice if we could do ignore_validation=True and then
    test the schemas separately, but since schemas also convert data this may
    not be possible.

  - ckan.controllers: all controller methods should have tests.

    Controller tests will be "functional" tests making HTTP posts and
    getting the HTML back. So they will be testing the template
    rendering as well as the code in the controllers themselves.

    Controller tests shouldn't need to be too detailed: test that the
    page renders without crashing, maybe that basic page contents are correct,
    thst submitting forms has the right effect.

    When asserting side effects in controller tests, do so by calling
    logic functions. (For example, if you've just tested submitting the
    create package form and now want to assert that your package was
    created correctly, use package_show to assert it.)

    Unfortunately some controllers contain a lot of logic, which shouldn't be
    there because it can't be easily unit-tested. Try to move this into logic
    and lib.

    API controller is a bit of a special case, maybe, because it doesn't have
    any templates.

  - ckan.model and ckan.lib: unit-tests for all "non-trivial" public functions
    and methods. Here "public" means a function/method that the
    module/class exports to be used by other parts of CKAN. Private
    helper functions (names beginning with _) can also have unit tests
    but don't have to.

    If a function or method is exported to third-party code, e.g. if
    it's part of the plugins toolkit or the template helpers, then it
    should have unit tests even if you think it's trivial.

    Note: I think we might be able to use the mock module a lot here, to
    really test each method in isolation and to make the tests fast and
    simple.

  - ckan.i18n: check_po_files.py should have tests

  - ckan.plugins:

    The plugin interfaces themselves are not really testable as they
    don't contain any code, but there should be example plugins with
    functional tests for each one, and also the tests for the code in
    logic etc. that actually calls the plugin interface methods should
    test this.

    Everything in toolkit.py should have tests, but most of it is
    functions imported from elsewhere in CKAN so the tests would be
    elsewhere as well.

  - ckan.migration: would be good to have tests of the migration
    scripts.

  - ckanext: the same rules apply for testing an extension as for CKAN core.
    Also extension code structure should mirror CKAN core code structure.

There was some more specific work from Toby, Vitor, Dominik and I about
test helper code and logic tests, I'll send this in a pull request once
I've had time to finish it.