[ckan-discuss] Circular updates problem

Thu Aug 12 16:48:14 BST 2010

On 12 August 2010 13:12, Friedrich Lindenberg <friedrich at pudo.org> wrote:
> Hi again,
>
> I have the following problem: Queue workers like the archiver will typically listen on the CKAN message queue for any updates, then execute a task and finally they will want to feed some return data (e.g. a resource hash) back into CKAN. The problem with this is pretty obvious: when CKAN is updated, it will notify any clients of the change and thus trigger the queue listener again.
>
> This is further complicated by the fact that we might want some of the queue listeners to execute (e.g. index update) while the event source should probably not be notified again.
>
> I currently see two possible solutions to this:
>
> 1. Demand that any queue listener should stabilize over time (i.e. it will first check whether the hash already saved with the resource is changed, and only update if there is an actual change).

Prefer this option. However it is hard to determine what part of a
resource has changed (eg suppose all that has changed is the hash
field then we don't want to retrigger the resource-caching worker).

To assist with identifying changes propose that we extend the queue
notification payload to include a *diff* in addition to the existing
json representation of the object. We are already generating these
diffs for the rest revision api and the package history page so it
would not be hard to add this.

Rufus

[...]