[ckan-changes] [ckan/ckan] cc46ef: [#2977] Basic implementation of background jobs vi...

Fri Sep 16 15:59:54 UTC 2016

  Branch: refs/heads/master
  Home:   https://github.com/ckan/ckan
  Commit: cc46efa33e5ee1fabe50ae734ce458073d508528
      https://github.com/ckan/ckan/commit/cc46efa33e5ee1fabe50ae734ce458073d508528
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/cli.py
    A ckan/lib/jobs.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M ckan/logic/auth/delete.py
    M ckan/logic/auth/get.py
    M ckan/websetup.py
    M requirements.in
    M requirements.txt
    M setup.py

  Log Message:
  -----------
  [#2977] Basic implementation of background jobs via python-rq.

Adds a basic support of background jobs using python-rq. Jobs can be
managed via the `ckan.lib.jobs` module, the `job_*` API functions and
the `jobs` paster command.

  Commit: 4f50fe38d6be080ed4de91ad91fc611ae904fabd
      https://github.com/ckan/ckan/commit/4f50fe38d6be080ed4de91ad91fc611ae904fabd
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/config/environment.py
    M ckan/lib/cli.py
    M ckan/lib/jobs.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M ckan/websetup.py

  Log Message:
  -----------
  [#2977] Lazy initialization of background job queue.

Check Redis availability on server start but initialize queue only once
it is actually required.

  Commit: dd2edc3fcb6ef9ddf0357847bebaa0c2d4315376
      https://github.com/ckan/ckan/commit/dd2edc3fcb6ef9ddf0357847bebaa0c2d4315376
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/celery_app.py
    M ckan/lib/cli.py
    M ckan/logic/action/get.py

  Log Message:
  -----------
  [#2977] Mark Celery features as deprecated.

  Commit: e0afc7cbd423377960f852067984ddd97350870a
      https://github.com/ckan/ckan/commit/e0afc7cbd423377960f852067984ddd97350870a
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py

  Log Message:
  -----------
  [#2977] Log exceptions raised by background jobs.

  Commit: 7ded14794cde0f34004d03b82d29f74ecb6cee16
      https://github.com/ckan/ckan/commit/7ded14794cde0f34004d03b82d29f74ecb6cee16
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M doc/maintaining/installing/install-from-package.rst
    M doc/maintaining/installing/install-from-source.rst

  Log Message:
  -----------
  [#2977] Add Redis to installation instructions.

  Commit: b264a8507c3c9469c2fbefd60a2e08b6fe0f406b
      https://github.com/ckan/ckan/commit/b264a8507c3c9469c2fbefd60a2e08b6fe0f406b
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckan/config/environment.py
    M ckan/lib/jobs.py
    M doc/maintaining/configuration.rst

  Log Message:
  -----------
  [#2977] Add `redis_url` configuration option.

  Commit: 541558a81b88cbeaf235d17113532d769efa3824
      https://github.com/ckan/ckan/commit/541558a81b88cbeaf235d17113532d769efa3824
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckan/lib/cli.py
    M ckan/lib/jobs.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M ckan/logic/schema.py
    M doc/maintaining/configuration.rst

  Log Message:
  -----------
  [#2977] Add support for multiple queues.

The background job system now supports multiple queues. By default, a
single queue is used.

In addition, RQ queue names are now prefixed with the CKAN site ID so
that multiple CKAN instances can share the same Redis database.

  Commit: b3be3208d0803cff0730c29e8e7cbd533c9c247f
      https://github.com/ckan/ckan/commit/b3be3208d0803cff0730c29e8e7cbd533c9c247f
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M doc/extensions/best-practices.rst
    M doc/extensions/custom-config-settings.rst
    M doc/theming/best-practices.rst
    M doc/theming/javascript.rst
    M doc/theming/templates.rst

  Log Message:
  -----------
  [#2977] Document requirement to namespace Redis keys.

Documents that Redis keys should be prefixed with the site ID and the
extension name.

  Commit: e9e7e3c7a74500afda9da58ec8ed1e0cb3632aeb
      https://github.com/ckan/ckan/commit/e9e7e3c7a74500afda9da58ec8ed1e0cb3632aeb
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/config/environment.py
    M ckan/lib/jobs.py
    A ckan/lib/redis.py
    M ckan/tests/helpers.py
    A ckan/tests/lib/test_jobs.py
    M doc/contributing/test.rst
    M test-core.ini

  Log Message:
  -----------
  [#2977] Add tests for `ckan.lib.jobs`.

  Commit: df7e61ff9fad17c9b4aecf146a20d4da3a69acdc
      https://github.com/ckan/ckan/commit/df7e61ff9fad17c9b4aecf146a20d4da3a69acdc
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M ckan/tests/helpers.py
    M ckan/tests/lib/test_jobs.py
    M ckan/tests/logic/action/test_delete.py
    M ckan/tests/logic/action/test_get.py

  Log Message:
  -----------
  [#2977] Tests for `job_*` API functions.

  Commit: 061ca311846462625dea0478a3ebf5131586dcfb
      https://github.com/ckan/ckan/commit/061ca311846462625dea0478a3ebf5131586dcfb
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/cli.py
    M ckan/tests/helpers.py
    M ckan/tests/lib/test_cli.py
    M ckan/tests/lib/test_jobs.py
    M ckan/tests/logic/action/test_delete.py

  Log Message:
  -----------
  [#2977] Tests for `paster jobs` command.

  Commit: c8a8553de1e976fa9113f03a2f8bc1da0d8187f6
      https://github.com/ckan/ckan/commit/c8a8553de1e976fa9113f03a2f8bc1da0d8187f6
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/cli.py
    M ckan/tests/lib/test_cli.py

  Log Message:
  -----------
  [#2977] Add `paster jobs show` command.

Useful for displaying details about a single job. Corresponds to the
job_show API function.

  Commit: a6788dbdc36a88d4201e6b3a67292adb6712c420
      https://github.com/ckan/ckan/commit/a6788dbdc36a88d4201e6b3a67292adb6712c420
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    R ckan/config/celery-supervisor.conf
    A ckan/config/supervisor-ckan-worker.conf
    M ckan/lib/cli.py
    M doc/api/index.rst
    M doc/extensions/best-practices.rst
    M doc/maintaining/background-tasks.rst
    M doc/maintaining/installing/deployment.rst
    M doc/maintaining/paster.rst

  Log Message:
  -----------
  [#2977] Document new background job system.

  Commit: d0b56a6896038cd475e26fd94581dc784e9a86f2
      https://github.com/ckan/ckan/commit/d0b56a6896038cd475e26fd94581dc784e9a86f2
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py

  Log Message:
  -----------
  [#2977] Increase verbosity of background job worker output.

This output doesn't go into the CKAN logs but into a separate worker log
(`/var/log/ckan-worker.log` if the default Supervisor configuration is
used).

  Commit: 958db3acc5180f4dafca873a366eb9aeb77ee335
      https://github.com/ckan/ckan/commit/958db3acc5180f4dafca873a366eb9aeb77ee335
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/lib/redis.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M doc/maintaining/configuration.rst
    M doc/maintaining/paster.rst

  Log Message:
  -----------
  [#2977] Document in which CKAN version new features were added.

  Commit: 6379585216e47de0c3a7eaff7f0efc7408c75d89
      https://github.com/ckan/ckan/commit/6379585216e47de0c3a7eaff7f0efc7408c75d89
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckan/config/environment.py
    M ckan/lib/redis.py
    M doc/contributing/test.rst
    M doc/extensions/best-practices.rst
    M doc/maintaining/configuration.rst
    M test-core.ini

  Log Message:
  -----------
  [#2977] Rename configuration option `redis_url` to `ckan.redis.url`.

  Commit: 4fcc8c1563844853769a96911df234bada8a6e6d
      https://github.com/ckan/ckan/commit/4fcc8c1563844853769a96911df234bada8a6e6d
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/tests/helpers.py
    M ckan/tests/lib/test_cli.py

  Log Message:
  -----------
  [#2977] Remove `ckan.tests.helpers.temp_file` context manager.

In most cases one can simply use `tempfile.NamedTemporaryFile`
instead.

  Commit: 4bd5521812fab111dfa8708cb629e96f27abc260
      https://github.com/ckan/ckan/commit/4bd5521812fab111dfa8708cb629e96f27abc260
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/tests/helpers.py

  Log Message:
  -----------
  [#2977] Improve docs of `ckan.tests.helpers.RecordingLogHandler`

Mention that it's usually best to simply use the `ckan.tests.recorded_logs`
context manager.

  Commit: 74aabf9d7159bd74b0f933725f2ac11bb4da95f2
      https://github.com/ckan/ckan/commit/74aabf9d7159bd74b0f933725f2ac11bb4da95f2
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M ckan/plugins/toolkit.py
    M doc/extensions/best-practices.rst
    M doc/extensions/plugins-toolkit.rst
    M doc/maintaining/background-tasks.rst

  Log Message:
  -----------
  [#2977] Add `ckan.plugins.toolkit.enqueue_job`.

  Commit: 4ef202b7423c8e64db4b0cf758fb7fc946255843
      https://github.com/ckan/ckan/commit/4ef202b7423c8e64db4b0cf758fb7fc946255843
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M doc/extensions/best-practices.rst

  Log Message:
  -----------
  [#2977] Improve wording of best practice regarding name clashes.

The documentation previously talked about sharing databases which could
be misunderstood as the possibility of sharing PostgreSQL databases
between CKAN instances (instead of sharing Redis databases, which was
the intended meaning).

  Commit: 18e140b5c07e952630747dbd1789d0b089b5e5f4
      https://github.com/ckan/ckan/commit/18e140b5c07e952630747dbd1789d0b089b5e5f4
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    A ckan/config/celery-supervisor.conf

  Log Message:
  -----------
  [#2977] Don't remove ckan/config/celery-supervisor.conf after all.

A previous commit for #2977 removed `ckan/config/celery-supervisor.conf`
as part of deprecating the old Celery background task system. However,
the old documentation told people to copy *or link* that file, so
removing it could break existing installations. Hence this commit
restores the file, it should be kept around until support for the Celery
system is removed completely.

  Commit: e855b55ad7b69a72057ea4b1c9d52cbe9a40d73c
      https://github.com/ckan/ckan/commit/e855b55ad7b69a72057ea4b1c9d52cbe9a40d73c
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-12 (Mon, 12 Sep 2016)

  Changed paths:
    M doc/extensions/best-practices.rst

  Log Message:
  -----------
  [#2977] Remove CKAN prefix from suggested configuration option name.

The docs now suggest to use `my_extension.my_setting` instead of the
previously suggested `ckan.my_extension.my_setting`.

  Commit: c3cf34e9b852f7c0198bd2ea1cacf6a5b13ebb34
      https://github.com/ckan/ckan/commit/c3cf34e9b852f7c0198bd2ea1cacf6a5b13ebb34
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-13 (Tue, 13 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/lib/redis.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M doc/maintaining/background-tasks.rst
    M doc/maintaining/configuration.rst
    M doc/maintaining/paster.rst

  Log Message:
  -----------
  [#2977] Change CKAN release from 2.6 to 2.7.

  Commit: a967a9b990954d78b156a8161219aed3f4e81631
      https://github.com/ckan/ckan/commit/a967a9b990954d78b156a8161219aed3f4e81631
  Author: Florian Brucker <florian.brucker at mb.karlsruhe.de>
  Date:   2016-09-14 (Wed, 14 Sep 2016)

  Changed paths:
    M ckan/lib/jobs.py
    M ckan/lib/redis.py
    M ckan/tests/lib/test_cli.py
    M ckan/tests/lib/test_jobs.py
    M doc/extensions/best-practices.rst

  Log Message:
  -----------
  [#2977] Import `config` from `ckan.common` instead of `pylons`.

  Commit: a483968c0da18975dd89c7af90a974f872fe5d41
      https://github.com/ckan/ckan/commit/a483968c0da18975dd89c7af90a974f872fe5d41
  Author: Ian Ward <ian at excess.org>
  Date:   2016-09-16 (Fri, 16 Sep 2016)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckan/config/environment.py
    A ckan/config/supervisor-ckan-worker.conf
    M ckan/lib/celery_app.py
    M ckan/lib/cli.py
    A ckan/lib/jobs.py
    A ckan/lib/redis.py
    M ckan/logic/action/delete.py
    M ckan/logic/action/get.py
    M ckan/logic/auth/delete.py
    M ckan/logic/auth/get.py
    M ckan/logic/schema.py
    M ckan/plugins/toolkit.py
    M ckan/tests/helpers.py
    M ckan/tests/lib/test_cli.py
    A ckan/tests/lib/test_jobs.py
    M ckan/tests/logic/action/test_delete.py
    M ckan/tests/logic/action/test_get.py
    M doc/api/index.rst
    M doc/contributing/test.rst
    M doc/extensions/best-practices.rst
    M doc/extensions/custom-config-settings.rst
    M doc/extensions/plugins-toolkit.rst
    M doc/maintaining/background-tasks.rst
    M doc/maintaining/configuration.rst
    M doc/maintaining/installing/deployment.rst
    M doc/maintaining/installing/install-from-package.rst
    M doc/maintaining/installing/install-from-source.rst
    M doc/maintaining/paster.rst
    M doc/theming/best-practices.rst
    M doc/theming/javascript.rst
    M doc/theming/templates.rst
    M requirements.in
    M requirements.txt
    M setup.py
    M test-core.ini

  Log Message:
  -----------
  Merge pull request #3165 from torfsen/2977-rq-background-tasks

[#2977] Basic implementation of background jobs via python-rq

Compare: https://github.com/ckan/ckan/compare/d1a78b982856...a483968c0da1