[ckan-changes] [ckan/ckan] 8a7b0d: [#2856] Avoid submitting resources to the DataPush...

Wed Mar 30 16:29:12 UTC 2016

  Branch: refs/heads/release-v2.5.2
  Home:   https://github.com/ckan/ckan
  Commit: 8a7b0d299a7d72d978cc300e535fe126ecd877cf
      https://github.com/ckan/ckan/commit/8a7b0d299a7d72d978cc300e535fe126ecd877cf
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datapusher/plugin.py

  Log Message:
  -----------
  [#2856] Avoid submitting resources to the DataPusher multiple times

The datapusher plugin uses the notify extension point to check if a
resource was created or updated. `datapusher_submit` will create a new
task with status pending when first called and `datapusher_hook` will
mark it as complete when necessary. We can use that to avoid resending
jobs unnecessarily.

  Commit: d4fc959b8fa38e942c8067299a4b6cf82415a731
      https://github.com/ckan/ckan/commit/d4fc959b8fa38e942c8067299a4b6cf82415a731
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datastore/logic/action.py

  Log Message:
  -----------
  [#2856] Only set datastore_active when necessary

Rather than updating the resource always, just do it when is actually
needed, ie:

* On datastore_create, when the resource doesn't already has a
  datastore_active=True extra

* On datastore_delete, only if the resource has datastore_active=False

  Commit: b295b91791a2f9a4d98c19bd20d989ea6d32e354
      https://github.com/ckan/ckan/commit/b295b91791a2f9a4d98c19bd20d989ea6d32e354
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#2856] Add some tests

  Commit: 17d1f86ed7d196ee450a07afe62c29ba56449f1b
      https://github.com/ckan/ckan/commit/17d1f86ed7d196ee450a07afe62c29ba56449f1b
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datastore/logic/action.py

  Log Message:
  -----------
  [#2856] Better True check

  Commit: 329ff2051fa8f1c2949b67202734f5d42b4c9560
      https://github.com/ckan/ckan/commit/329ff2051fa8f1c2949b67202734f5d42b4c9560
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Resubmit to DataPusher if file is reuploaded or URL changes

Previous commits prevent new DataPusher jobs from being created if there
is an ongoing one. There are cases though where we actually want to
resubmit, basically if the data source has changed since the first
DataPusher job started. This can be because a new file was uploaded or
because the URL was changed.

* For file uploads we check the `last_modified` resource property, which
  is set after an upload against the task creation date.
* For URLs we check if the original URL is different from the current
  one.

 We store both the task creation date and the original URL in the
 metadata sent to (and returned from) the DataPusher.

  Commit: 42f96c29b67ccf2c28e0f3e2db30a60074554066
      https://github.com/ckan/ckan/commit/42f96c29b67ccf2c28e0f3e2db30a60074554066
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Move variable to correct scope

  Commit: e67192536f1f962be131c1495ecf80f778815635
      https://github.com/ckan/ckan/commit/e67192536f1f962be131c1495ecf80f778815635
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] PEP8]

  Commit: b5a0aca9b294215aacbcdbe13a8d8813e9b2a70b
      https://github.com/ckan/ckan/commit/b5a0aca9b294215aacbcdbe13a8d8813e9b2a70b
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckanext/datapusher/logic/action.py

  Log Message:
  -----------
  [#2856] Tweaks after review

  Commit: ee3fb807b68e5368878adfea0b740486c55e4c34
      https://github.com/ckan/ckan/commit/ee3fb807b68e5368878adfea0b740486c55e4c34
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    M ckan/tests/helpers.py

  Log Message:
  -----------
  [#2856] Add decorator to mock a CKAN action in tests

Because of the way actions are accessed via `get_action`, it is a pain to mock
them. This decorator adds a mock object for the provided action as a parameter
to the test function. The mock is discarded at the end of the function, even
if there is an exception raised.

This mocks the action both when it's called directly via
`ckan.logic.get_action` and via `ckan.plugins.toolkit.get_action`

  Commit: ffc46cf48f467a32e72d03a94ab0ed41cba986e3
      https://github.com/ckan/ckan/commit/ffc46cf48f467a32e72d03a94ab0ed41cba986e3
  Author: amercader <amercadero at gmail.com>
  Date:   2016-03-30 (Wed, 30 Mar 2016)

  Changed paths:
    A ckanext/datapusher/tests/test_action.py

  Log Message:
  -----------
  [#2856] Added tests for DataPusher submissions

Test basic behaviour and resubmissions if updates are performed during
an ongoing job.

Compare: https://github.com/ckan/ckan/compare/39a890119e2d...ffc46cf48f46