[ckan-changes] [ckan/ckan] 83d865: [#2856] Avoid submitting resources to the DataPush...
GitHub
noreply at github.com
Wed Mar 30 15:16:48 UTC 2016
Branch: refs/heads/master
Home: https://github.com/ckan/ckan
Commit: 83d865e9e80ea79cdc4ee74f73b5c01699ac4357
https://github.com/ckan/ckan/commit/83d865e9e80ea79cdc4ee74f73b5c01699ac4357
Author: amercader <amercadero at gmail.com>
Date: 2016-03-17 (Thu, 17 Mar 2016)
Changed paths:
M ckanext/datapusher/plugin.py
Log Message:
-----------
[#2856] Avoid submitting resources to the DataPusher multiple times
The datapusher plugin uses the notify extension point to check if a
resource was created or updated. `datapusher_submit` will create a new
task with status pending when first called and `datapusher_hook` will
mark it as complete when necessary. We can use that to avoid resending
jobs unnecessarily.
Commit: c4306a248dd064ac1a5fbbcf4d37d23a18561e0c
https://github.com/ckan/ckan/commit/c4306a248dd064ac1a5fbbcf4d37d23a18561e0c
Author: amercader <amercadero at gmail.com>
Date: 2016-03-17 (Thu, 17 Mar 2016)
Changed paths:
M ckanext/datastore/logic/action.py
Log Message:
-----------
[#2856] Only set datastore_active when necessary
Rather than updating the resource always, just do it when is actually
needed, ie:
* On datastore_create, when the resource doesn't already has a
datastore_active=True extra
* On datastore_delete, only if the resource has datastore_active=False
Commit: 461afadd285c9036e18572470bd19901754835b8
https://github.com/ckan/ckan/commit/461afadd285c9036e18572470bd19901754835b8
Author: amercader <amercadero at gmail.com>
Date: 2016-03-17 (Thu, 17 Mar 2016)
Changed paths:
M ckanext/datastore/tests/test_create.py
Log Message:
-----------
[#2856] Add some tests
Commit: c7ae0c64cecb47467e37420e67f05c0f9a291a97
https://github.com/ckan/ckan/commit/c7ae0c64cecb47467e37420e67f05c0f9a291a97
Author: amercader <amercadero at gmail.com>
Date: 2016-03-17 (Thu, 17 Mar 2016)
Changed paths:
M ckanext/datastore/logic/action.py
Log Message:
-----------
[#2856] Better True check
Commit: 83b963cdb16ca541601201409d37b5503dd27116
https://github.com/ckan/ckan/commit/83b963cdb16ca541601201409d37b5503dd27116
Author: amercader <amercadero at gmail.com>
Date: 2016-03-23 (Wed, 23 Mar 2016)
Changed paths:
M ckanext/datapusher/logic/action.py
Log Message:
-----------
[#2856] Resubmit to DataPusher if file is reuploaded or URL changes
Previous commits prevent new DataPusher jobs from being created if there
is an ongoing one. There are cases though where we actually want to
resubmit, basically if the data source has changed since the first
DataPusher job started. This can be because a new file was uploaded or
because the URL was changed.
* For file uploads we check the `last_modified` resource property, which
is set after an upload against the task creation date.
* For URLs we check if the original URL is different from the current
one.
We store both the task creation date and the original URL in the
metadata sent to (and returned from) the DataPusher.
Commit: 31b607f83882f577ae1ce4d7bda22ed8c002659e
https://github.com/ckan/ckan/commit/31b607f83882f577ae1ce4d7bda22ed8c002659e
Author: amercader <amercadero at gmail.com>
Date: 2016-03-23 (Wed, 23 Mar 2016)
Changed paths:
M ckanext/datapusher/logic/action.py
Log Message:
-----------
[#2856] Move variable to correct scope
Commit: b6c3b82646fea949dfce0647920fbd9f0c9404e2
https://github.com/ckan/ckan/commit/b6c3b82646fea949dfce0647920fbd9f0c9404e2
Author: amercader <amercadero at gmail.com>
Date: 2016-03-24 (Thu, 24 Mar 2016)
Changed paths:
M ckanext/datapusher/logic/action.py
Log Message:
-----------
[#2856] PEP8]
Commit: a442577e77640b5a1e4a1d75097cbdcd68084a5a
https://github.com/ckan/ckan/commit/a442577e77640b5a1e4a1d75097cbdcd68084a5a
Author: amercader <amercadero at gmail.com>
Date: 2016-03-25 (Fri, 25 Mar 2016)
Changed paths:
M ckanext/datapusher/logic/action.py
Log Message:
-----------
[#2856] Tweaks after review
Commit: f63a9723886bd601fdf893e772c335859f25afe5
https://github.com/ckan/ckan/commit/f63a9723886bd601fdf893e772c335859f25afe5
Author: amercader <amercadero at gmail.com>
Date: 2016-03-25 (Fri, 25 Mar 2016)
Changed paths:
M ckan/tests/helpers.py
Log Message:
-----------
[#2856] Add decorator to mock a CKAN action in tests
Because of the way actions are accessed via `get_action`, it is a pain to mock
them. This decorator adds a mock object for the provided action as a parameter
to the test function. The mock is discarded at the end of the function, even
if there is an exception raised.
This mocks the action both when it's called directly via
`ckan.logic.get_action` and via `ckan.plugins.toolkit.get_action`
Commit: 534621c5f3005bb1836b890b2bc8a38f20d61404
https://github.com/ckan/ckan/commit/534621c5f3005bb1836b890b2bc8a38f20d61404
Author: amercader <amercadero at gmail.com>
Date: 2016-03-25 (Fri, 25 Mar 2016)
Changed paths:
A ckanext/datapusher/tests/test_action.py
Log Message:
-----------
[#2856] Added tests for DataPusher submissions
Test basic behaviour and resubmissions if updates are performed during
an ongoing job.
Commit: f69e28a62c845cecd47219b6e682a5efb51ff3ba
https://github.com/ckan/ckan/commit/f69e28a62c845cecd47219b6e682a5efb51ff3ba
Author: amercader <amercadero at gmail.com>
Date: 2016-03-25 (Fri, 25 Mar 2016)
Changed paths:
A bin/solr_init/create_core.sh
M circle.yml
M ckan/config/deployment.ini_tmpl
M ckan/config/environment.py
M ckan/controllers/user.py
M ckan/lib/app_globals.py
M ckan/lib/base.py
M ckan/logic/validators.py
M ckan/model/domain_object.py
M ckan/tests/controllers/test_admin.py
M ckan/tests/controllers/test_package.py
M ckan/tests/lib/test_base.py
M ckan/tests/model/test_resource.py
M ckanext/datapusher/tests/test.py
M doc/maintaining/configuration.rst
M test-core.ini
Log Message:
-----------
Merge branch 'master' into 2856-fix-datapusher-multiple-calls
Commit: b937f6ae542bcc64146bf99a6475001332ad2b77
https://github.com/ckan/ckan/commit/b937f6ae542bcc64146bf99a6475001332ad2b77
Author: Ian Ward <ian at excess.org>
Date: 2016-03-30 (Wed, 30 Mar 2016)
Changed paths:
M ckan/tests/helpers.py
M ckanext/datapusher/logic/action.py
M ckanext/datapusher/plugin.py
A ckanext/datapusher/tests/test_action.py
M ckanext/datastore/logic/action.py
M ckanext/datastore/tests/test_create.py
Log Message:
-----------
Merge pull request #2921 from ckan/2856-fix-datapusher-multiple-calls
[#2856] Limit calls between DataPusher -> DataStore -> CKAN
Compare: https://github.com/ckan/ckan/compare/3bd3e2d5cbfc...b937f6ae542b
More information about the ckan-changes
mailing list