[ckan-dev] RFC: Webtests for CKAN

Thu Sep 20 14:51:46 UTC 2012

> As discussed in the last standup, 
> 
> There are quite a few testing frameworks available but I think only two major once make sense for ckan. The first is Twill and the second is WebTest.
> 
> # A few words about Twill:
> Docs: http://twill.idyll.org/
> 
> ## Pros:
> * more straight forward language
> * record sequences of actions
> 
> ## Cons:
> * uses only beautiful soup
> * poor docs
> * not actively maintained (however, there is a retwill fork on github)
> 
> 
> # And a few words about WebTest:
> Docs: http://webtest.pythonpaste.org/en/latest/index.html
> 
> ## Pros:
> * actively maintained
> * integrated into major python web frameworks, recommended for pylons
> * you can choose between lxml html, lxml xml, beautiful soup, pyquery and json
> * good documentation
> 
> ## Cons:
> * No real webtesting (in an actual browser) since js is ignored
> * sometimes a little bit difficult to understand how to select links/ forms.
> 
> Overall, I think WebTest is the way to go which is why I added a few quick examples tat demonstrate how to use forms, `click` and xpaths/ pyquery. Pull request: https://github.com/okfn/ckan/pull/130
> 
> The ticket for the whole thing is here: http://trac.ckan.org/ticket/2934
> 
> Sean suggested that we should talk about this in the next standup tomorrow.

So I was quite happy with the way I was writing UI tests for
ckanext-cmap: using paste.fixture.TestApp to request pages, then using
BeautifulSoup to parse the results. Note the most concise thing in terms
of saving on typing but simple enough.

WebTest is based on paste.fixture.TestApp but apparently parts of it
have been rewritten to use WebOb. As far as I can see the interface of
the TestApp and Response objects are pretty much the same as those from
paste.  The documentation looks better or at least easier to find. The
response object has builtin convenience methods for getting
BeautifulSoup, ElementTree, LXML, or PyQuery parsed copy of the body,
which as far as I know paste's Response object didn't have, but it was
only one line of code to get it yourself anyway.

We would have to add webtest and I guess one of BeautifulSoup,
ElementTree or LXML to pip-requirements-test.txt (anyone have a
preference?)

So WebTest looks like just the thing to me.

I think it would probably be worthwhile for Dominik to write WebTest
tests for a couple of parts of CKAN and get them reviewed and merged
into master, and then copy a couple of examples from them and paste them
into a new section in the CKAN Coding Standards, explaining what our
best practice is for doing UI tests.
I think that's important because using the right tool won't stop us from
writing terrible tests with it.

P.S. For testing JavaScript, which WebTest doesn't do, there is actually
a JavaScript test framework builtin into CKAN now, that came with the
big demo merge.

P.P.S. Dominik I see that you're calling these "integration" tests, in
CKAN currently these kind of UI tests (that use paste.fixture.TestApp)
are called "functional" tests (see ckan/tests/functional). So maybe you
just want to call yours functional tests, or maybe there's a distinction
to be made between functional tests that test the contents of individual
pages in detail and process or integration tests that test clicking
through multiple pages but without checking the contents of each page in
thorough detail.