[ckan-changes] [ckan/ckan] a5d3e4: [#3365] Dispose database engine before forking a w...

Thu Jan 19 19:48:23 UTC 2017

  Branch: refs/heads/master
  Home:   https://github.com/ckan/ckan
  Commit: a5d3e46318f412bf182d91eda9831ffd43052757
      https://github.com/ckan/ckan/commit/a5d3e46318f412bf182d91eda9831ffd43052757
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2017-01-19 (Thu, 19 Jan 2017)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/tests/lib/test_jobs.py

  Log Message:
  -----------
  [#3365] Dispose database engine before forking a worker

rq's workers perform their tasks in separate processes. When a worker is
forked, it inherits the global state of the parent, which in our case
included SQLAlchemy's database engine and its associated connections.
The new worker process then initializes its own engine, thereby losing
the reference to its copy of the parent's engine. The latter is now not
referenced anymore and therefore eventually garbage collected, which
includes closing the associated connections. However, these connections
have been copied from the parent and may still be in use there, so
closing them in the worker leads to errors in the main process.

This commit resolves the issue by explicitly disposing the engine and
all its associated connections before the fork. This rolls back all
non-committed changes. The engine automatically re-initializes itself
once it is used again in the parent after the fork (the worker
initializes its own separate engine directly after the fork).

While this solution is clean and deterministic it still implies that you
cannot re-use session instances and ORM instances from before the fork
after the fork. However, since workers are only started in separate
paster processes and in test cases this is not an issue in normal code.

  Commit: efd930acde1bbd6049a1cf28fa91d663a91fc09e
      https://github.com/ckan/ckan/commit/efd930acde1bbd6049a1cf28fa91d663a91fc09e
  Author: Ian Ward <ian at excess.org>
  Date:   2017-01-19 (Thu, 19 Jan 2017)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/tests/lib/test_jobs.py

  Log Message:
  -----------
  Merge pull request #3377 from torfsen/3365-background-job-test-failures

[#3365] Fix intermittent test failures in test_jobs

Compare: https://github.com/ckan/ckan/compare/73ba6ff46a8b...efd930acde1b