[ckan-changes] [ckan/ckan] 83d865: [#2856] Avoid submitting resources to the DataPush...

Wed Mar 30 15:16:48 UTC 2016

  Branch: refs/heads/master
  Home:   https://github.com/ckan/ckan
  Commit: 83d865e9e80ea79cdc4ee74f73b5c01699ac4357
      https://github.com/ckan/ckan/commit/83d865e9e80ea79cdc4ee74f73b5c01699ac4357
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-17 (Thu, 17 Mar 2016)

  Changed paths:
    M ckanext/datapusher/plugin.py

  Log Message:
  -----------
  [#2856] Avoid submitting resources to the DataPusher multiple times

The datapusher plugin uses the notify extension point to check if a
resource was created or updated. `datapusher_submit` will create a new
task with status pending when first called and `datapusher_hook` will
mark it as complete when necessary. We can use that to avoid resending
jobs unnecessarily.

  Commit: c4306a248dd064ac1a5fbbcf4d37d23a18561e0c
      https://github.com/ckan/ckan/commit/c4306a248dd064ac1a5fbbcf4d37d23a18561e0c
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-17 (Thu, 17 Mar 2016)

  Changed paths:
    M ckanext/datastore/logic/action.py

  Log Message:
  -----------
  [#2856] Only set datastore_active when necessary

Rather than updating the resource always, just do it when is actually
needed, ie:

* On datastore_create, when the resource doesn't already has a
  datastore_active=True extra

* On datastore_delete, only if the resource has datastore_active=False

  Commit: 461afadd285c9036e18572470bd19901754835b8
      https://github.com/ckan/ckan/commit/461afadd285c9036e18572470bd19901754835b8
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-17 (Thu, 17 Mar 2016)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#2856] Add some tests

  Commit: c7ae0c64cecb47467e37420e67f05c0f9a291a97
      https://github.com/ckan/ckan/commit/c7ae0c64cecb47467e37420e67f05c0f9a291a97
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-17 (Thu, 17 Mar 2016)

  Changed paths:
    M ckanext/datastore/logic/action.py

  Log Message:
  -----------
  [#2856] Better True check

  Commit: 83b963cdb16ca541601201409d37b5503dd27116
      https://github.com/ckan/ckan/commit/83b963cdb16ca541601201409d37b5503dd27116
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-23 (Wed, 23 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Resubmit to DataPusher if file is reuploaded or URL changes

Previous commits prevent new DataPusher jobs from being created if there
is an ongoing one. There are cases though where we actually want to
resubmit, basically if the data source has changed since the first
DataPusher job started. This can be because a new file was uploaded or
because the URL was changed.

* For file uploads we check the `last_modified` resource property, which
  is set after an upload against the task creation date.
* For URLs we check if the original URL is different from the current
  one.

 We store both the task creation date and the original URL in the
 metadata sent to (and returned from) the DataPusher.

  Commit: 31b607f83882f577ae1ce4d7bda22ed8c002659e
      https://github.com/ckan/ckan/commit/31b607f83882f577ae1ce4d7bda22ed8c002659e
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-23 (Wed, 23 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Move variable to correct scope

  Commit: b6c3b82646fea949dfce0647920fbd9f0c9404e2
      https://github.com/ckan/ckan/commit/b6c3b82646fea949dfce0647920fbd9f0c9404e2
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-24 (Thu, 24 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] PEP8]

  Commit: a442577e77640b5a1e4a1d75097cbdcd68084a5a
      https://github.com/ckan/ckan/commit/a442577e77640b5a1e4a1d75097cbdcd68084a5a
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-25 (Fri, 25 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Tweaks after review

  Commit: f63a9723886bd601fdf893e772c335859f25afe5
      https://github.com/ckan/ckan/commit/f63a9723886bd601fdf893e772c335859f25afe5
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-25 (Fri, 25 Mar 2016)

  Changed paths:
    M ckan/tests/helpers.py

  Log Message:
  -----------
  [#2856] Add decorator to mock a CKAN action in tests

Because of the way actions are accessed via `get_action`, it is a pain to mock
them. This decorator adds a mock object for the provided action as a parameter
to the test function. The mock is discarded at the end of the function, even
if there is an exception raised.

This mocks the action both when it's called directly via
`ckan.logic.get_action` and via `ckan.plugins.toolkit.get_action`

  Commit: 534621c5f3005bb1836b890b2bc8a38f20d61404
      https://github.com/ckan/ckan/commit/534621c5f3005bb1836b890b2bc8a38f20d61404
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-25 (Fri, 25 Mar 2016)

  Changed paths:
    A ckanext/datapusher/tests/test_action.py

  Log Message:
  -----------
  [#2856] Added tests for DataPusher submissions

Test basic behaviour and resubmissions if updates are performed during
an ongoing job.

  Commit: f69e28a62c845cecd47219b6e682a5efb51ff3ba
      https://github.com/ckan/ckan/commit/f69e28a62c845cecd47219b6e682a5efb51ff3ba
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-25 (Fri, 25 Mar 2016)

  Changed paths:
    A bin/solr_init/create_core.sh
    M circle.yml
    M ckan/config/deployment.ini_tmpl
    M ckan/config/environment.py
    M ckan/controllers/user.py
    M ckan/lib/app_globals.py
    M ckan/lib/base.py
    M ckan/logic/validators.py
    M ckan/model/domain_object.py
    M ckan/tests/controllers/test_admin.py
    M ckan/tests/controllers/test_package.py
    M ckan/tests/lib/test_base.py
    M ckan/tests/model/test_resource.py
    M ckanext/datapusher/tests/test.py
    M doc/maintaining/configuration.rst
    M test-core.ini

  Log Message:
  -----------
  Merge branch 'master' into 2856-fix-datapusher-multiple-calls

  Commit: b937f6ae542bcc64146bf99a6475001332ad2b77
      https://github.com/ckan/ckan/commit/b937f6ae542bcc64146bf99a6475001332ad2b77
  Author: Ian Ward <ian at excess.org>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckan/tests/helpers.py
    M ckanext/datapusher/logic/action.py
    M ckanext/datapusher/plugin.py
    A ckanext/datapusher/tests/test_action.py
    M ckanext/datastore/logic/action.py
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  Merge pull request #2921 from ckan/2856-fix-datapusher-multiple-calls

[#2856] Limit calls between DataPusher -> DataStore -> CKAN

Compare: https://github.com/ckan/ckan/compare/3bd3e2d5cbfc...b937f6ae542b