[ckan-dev] Errors when Testing CKAN

Thu Jun 30 00:09:36 UTC 2016

Hello,

I have been setting up a pilot instance of CKAN 2.5.2 and following the CKAN documentation for Release 2.5.2.  I can run the Paste development server from the command line and see the CKAN front page.  The documentation then says to "run CKAN's tests to make sure that everything's working".  When I follow the instructions in the "Testing CKAN" section of the documentation, I get hundreds of errors.  It looks like it fails often on the line "with mock.patch('ckan.logic.get_action') as mock_get_action, \" in /usr/lib/ckan/default/src/ckan/ckan/tests/helpers.py.

My pilot environment includes:

centos 6.7
ckan 2.5.2
solr 6.0.0
java 1.8.0
python 2.6.6
postgres 8.4.20
jetty 9.2.15

At the end of this post is an excerpt from the output when I run the following command:

nosetests --ckan --reset-db --with-pylons=test-core.ini ckan

Sorry for the length of this post ... I wanted to make sure I included enough information for diagnostic purposes.

Thank you for any assistance you can offer regarding how to resolve these errors.

--- Jim Espeland
james.espeland at montana.edu

The output:

EEEEEEEEEEEEEE/usr/lib/ckan/default/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py:79: SAWarning: Usage of the 'related attribute set' operation is not currently supported within the execution stage of the flush process. Results may not be consistent.  Consider using alternative event listeners or connection-level operations instead.
  sess._flush_warning("related attribute set")
EEEEEEEEEEEEE.....E...EEEEEEEEEEE................................E...E.EEEEEEEEEESSEEEEEEEEEEEEEEEE.EEEEEE..EEEEEEEE..EEEEEEEEEEE.....EEEEE..EE.....EEE........EEEE.............E.............E................EEEE.EEEEEE...EEEEEE...EEEEFEFEEE.F.EEFEEEE......EE...E......................EE................EEE....EEEEEEEEEEEEEEEEEEEE.........EE.....................EE.E..EE...EEE.............EE.........EEEEEEEEEE...........EEEEEE........................................................EFE
======================================================================
ERROR: Failure: SyntaxError (invalid syntax (helpers.py, line 366))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/config/test_environment.py", line 6, in <module>
    import ckan.tests.helpers as h
SyntaxError: invalid syntax (helpers.py, line 366)
-------------------- >> begin captured logging << --------------------
pyutilib.component.core.pca: DEBUG: Creating PluginEnvironment 'pca'
pyutilib.component.core.<default>: DEBUG: Creating PluginEnvironment '<default>'
pyutilib.component.core.<default>: DEBUG: Pushing environment '<default>' on the PluginGlobals stack
pyutilib.component.core.pca: DEBUG: Pushing environment 'pca' on the PluginGlobals stack
pyutilib.component.core.pca: DEBUG: Pushing environment 'pca' on the PluginGlobals stack
pyutilib.component.core.pca: INFO: Adding service IgnorePluginPlugins to environment pca
pyutilib.component.core.pca: INFO: Adding service IgnorePluginPlugins to environment pca
pyutilib.component.core.pca: DEBUG: Popping environment 'pca' from the PluginGlobals stack
passlib.utils.compat: DEBUG: loaded lazy attr 'BytesIO': <built-in function StringIO>
passlib.registry: DEBUG: registered 'pbkdf2_sha512' handler: <class 'passlib.handlers.pbkdf2.pbkdf2_sha512'>
pyutilib.component.core.pca: INFO: Adding service SynchronousSearchPlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service SynchronousSearchPlugin to environment pca
pylons.configuration: DEBUG: Initializing configuration, package: 'ckan'
pylons.configuration: DEBUG: Pushing process configuration
pylons.configuration: DEBUG: Adding mako engine with alias None and {'myghty.data_dir': '/usr/lib/ckan/default/src/ckan/data/templates', 'mako.directories': [], 'myghty.component_root': [], 'kid.encoding': 'utf-8', 'kid.assume_encoding': 'utf-8', 'mako.module_directory': '/usr/lib/ckan/default/src/ckan/data/templates', 'myghty.allow_globals': ['c', 'config', 'g', 'h', 'render', 'request', 'session', 'translator', 'ungettext', '_', 'N_'], 'myghty.output_encoding': 'utf-8', 'myghty.raise_error': True, 'mako.output_encoding': 'utf-8', 'mako.filesystem_checks': True} options
pylons.configuration: DEBUG: Loaded mako template engine as the default template renderer
pyutilib.component.core.pca: INFO: Removing service IgnorePluginPlugins from environment pca
pyutilib.component.core.pca: INFO: Removing service IgnorePluginPlugins from environment pca
pyutilib.component.core.pca: INFO: Removing service SynchronousSearchPlugin from environment pca
pyutilib.component.core.pca: INFO: Removing service SynchronousSearchPlugin from environment pca
ckan.config.environment: WARNING: Support for Genshi templates is deprecated and will be removed in a future release
ckan.logic: DEBUG: check access OK - get_site_user user=None
ckan.lib.activity_streams_session_extension: DEBUG: session had no _object_cache or no revision, skipping this commit
Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/activity_streams_session_extension.py", line 52, in before_commit
    revision = session.revision
AttributeError: 'Session' object has no attribute 'revision'
pyutilib.component.core.pca: INFO: Adding service DomainObjectModificationExtension to environment pca
pyutilib.component.core.pca: INFO: Adding service DomainObjectModificationExtension to environment pca
ckan.plugins.core: DEBUG: Loading the synchronous search plugin
pyutilib.component.core.pca: INFO: Adding service StatsPlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service StatsPlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service StatsPlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service stats to environment pca
pyutilib.component.core.pca: INFO: Adding service stats to environment pca
pyutilib.component.core.pca: INFO: Adding service stats to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service DatastorePlugin to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service datastore to environment pca
pyutilib.component.core.pca: INFO: Adding service domain_object_mods to environment pca
pyutilib.component.core.pca: INFO: Adding service domain_object_mods to environment pca
pyutilib.component.core.pca: INFO: Adding service synchronous_search to environment pca
pyutilib.component.core.pca: INFO: Adding service synchronous_search to environment pca
ckan.config.environment: WARNING: Support for Genshi templates is deprecated and will be removed in a future release
ckanext.datastore.plugin: WARNING: Legacy mode active. The sql search will not be available.
ckan.logic: DEBUG: check access OK - get_site_user user=None
pylons.templating: DEBUG: Initialized Buffet object
pylons.templating: DEBUG: Adding mako template language for use with Buffet
ckan.lib.app_globals: DEBUG: config `ckan.site_title` set to `CKAN` from config
ckan.lib.app_globals: DEBUG: config `ckan.site_url` set to `http://test.ckan.net` from config
ckan.lib.app_globals: DEBUG: config `ckan.site_logo` set to `/images/ckan_logo_fullname_long.png` from config
routes.middleware: DEBUG: Initialized with method overriding = True, and path info altering = True
ckan.lib.uploader: CRITICAL: Please specify a ckan.storage_path in your config
                         for your uploads
ckan.tests: WARNING: ckan.new_tests has been renamed to ckan.tests.In the next release tests will only be available from ckan.tests.
ckan.tests: WARNING: ckan.new_tests has been renamed to ckan.tests.In the next release tests will only be available from ckan.tests.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (helpers.py, line 366))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/config/test_middleware.py", line 1, in <module>
    import ckan.tests.helpers as helpers
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/helpers.py", line 366
    with mock.patch('ckan.logic.get_action') as mock_get_action, \
                                                               ^
SyntaxError: invalid syntax

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (helpers.py, line 366))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/controllers/test_admin.py", line 8, in <module>
    import ckan.tests.helpers as helpers
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/helpers.py", line 366
    with mock.patch('ckan.logic.get_action') as mock_get_action, \
                                                               ^
SyntaxError: invalid syntax

.
.
.

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (helpers.py, line 366))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
 File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/test_factories.py", line 4, in <module>
    import ckan.tests.helpers as helpers
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/helpers.py", line 366
    with mock.patch('ckan.logic.get_action') as mock_get_action, \
                                                               ^
SyntaxError: invalid syntax

======================================================================
FAIL: ckan.tests.legacy.logic.test_auth.TestAuthGroups.test_01_create_groups
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 461, in test_01_create_groups
    self._call_api('group_create', group, 'sysadmin')
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 55, in _call_api
    (res.status, status, error))
AssertionError: Status was 409 but should be None. Error: {u'message': u'Search error: \'SOLR returned an error running query: {\\\'sort\\\': \\\'score desc, metadata_modified desc\\\', \\\'fq\\\': [u\\\'groups:"group_no_user" capacity:"public" +site_id:"test.ckan.net" +state:active\\\'], \\\'facet.mincount\\\': 1, \\\'rows\\\': 0, \\\'facet\\\': u\\\'false\\\', \\\'facet.limit\\\': \\\'50\\\', \\\'wt\\\': \\\'json\\\', \\\'q\\\': \\\'*:*\\\', \\\'fl\\\': \\\'id validated_data_dict\\\'} Error: \\\'Bad Request\\\'\'', u'__type': u'Search Error'}
-------------------- >> begin captured logging << --------------------
ckan.model: INFO: Database initialised
ckan.config.environment: WARNING: Support for Genshi templates is deprecated and will be removed in a future release
ckan.config.environment: WARNING: Support for Genshi templates is deprecated and will be removed in a future release
ckan.lib.base: INFO:  /api/action/group_create render time 0.031 seconds
ckan.lib.base: INFO:  /api/action/group_create render time 0.109 seconds
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: ckan.tests.legacy.logic.test_auth.TestAuthGroups.test_03_add_dataset_to_group
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 492, in test_03_add_dataset_to_group
    self._call_api('organization_create', org, 'sysadmin')
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 55, in _call_api
    (res.status, status, error))
AssertionError: Status was 409 but should be None. Error: {u'message': u'Search error: \'SOLR returned an error running query: {\\\'sort\\\': \\\'score desc, metadata_modified desc\\\', \\\'fq\\\': [u\\\'owner_org:"96c041dc-e1e8-401a-9f36-a8b1613fbb00" +site_id:"test.ckan.net" +state:active\\\'], \\\'facet.mincount\\\': 1, \\\'rows\\\': 0, \\\'facet\\\': u\\\'false\\\', \\\'facet.limit\\\': \\\'50\\\', \\\'wt\\\': \\\'json\\\', \\\'q\\\': \\\'*:*\\\', \\\'fl\\\': \\\'id validated_data_dict\\\'} Error: \\\'Bad Request\\\'\'', u'__type': u'Search Error'}
-------------------- >> begin captured logging << --------------------
ckan.lib.base: INFO:  /api/action/organization_create render time 0.245 seconds
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: ckan.tests.legacy.logic.test_auth.TestAuthOrgs.test_02_create_orgs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 115, in test_02_create_orgs
    self._call_api('organization_create', org, 'sysadmin')
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 55, in _call_api
    (res.status, status, error))
AssertionError: Status was 409 but should be None. Error: {u'message': u'Search error: \'SOLR returned an error running query: {\\\'sort\\\': \\\'score desc, metadata_modified desc\\\', \\\'fq\\\': [u\\\'owner_org:"25d59e46-c7c2-4b59-9e6c-dde13be52bb3" +site_id:"test.ckan.net" +state:active\\\'], \\\'facet.mincount\\\': 1, \\\'rows\\\': 0, \\\'facet\\\': u\\\'false\\\', \\\'facet.limit\\\': \\\'50\\\', \\\'wt\\\': \\\'json\\\', \\\'q\\\': \\\'*:*\\\', \\\'fl\\\': \\\'id validated_data_dict\\\'} Error: \\\'Bad Request\\\'\'', u'__type': u'Search Error'}
-------------------- >> begin captured logging << --------------------
ckan.lib.base: INFO:  /api/action/organization_create render time 0.005 seconds
ckan.lib.base: INFO:  /api/action/organization_create render time 0.098 seconds
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: ckan.tests.legacy.logic.test_auth.TestAuthOrgs.test_07_add_datasets
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 186, in test_07_add_datasets
    self._add_datasets('org_admin')
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 171, in _add_datasets
    self._call_api('package_create', dataset, user, 200)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/legacy/logic/test_auth.py", line 55, in _call_api
    (res.status, status, error))
AssertionError: Status was 403 but should be 200. Error: {u'message': u'Access denied: User org_admin not authorized to create packages', u'__type': u'Authorization Error'}
-------------------- >> begin captured logging << --------------------
ckan.lib.base: INFO:  /api/action/package_create render time 0.006 seconds
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: ckan.tests.test_coding_standards.test_building_the_docs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/ckan/default/src/ckan/ckan/tests/test_coding_standards.py", line 68, in test_building_the_docs
    "{warnings}".format(warnings=new_warnings))
AssertionError: Don't add any new warnings to the Sphinx build: ["/usr/lib/ckan/default/src/ckan/doc/contributing/testing.rst:268: WARNING: autodoc: failed to import module u'ckan.tests.factories'; the following exception was raised:", "/usr/lib/ckan/default/src/ckan/doc/contributing/testing.rst:275: WARNING: autodoc: failed to import module u'ckan.tests.helpers'; the following exception was raised:"]

----------------------------------------------------------------------
Ran 399 tests in 213.392s

FAILED (SKIP=2, errors=196, failures=5)

-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160630/79ce9411/attachment-0002.html>