[ckan-discuss] Circular updates problem

Sat Aug 14 08:59:54 BST 2010

On 10-08-12 13:12, Friedrich Lindenberg wrote:
> I have the following problem: Queue workers like the archiver will typically listen on the CKAN message queue for any updates, then execute a task and finally they will want to feed some return data (e.g. a resource hash) back into CKAN. The problem with this is pretty obvious: when CKAN is updated, it will notify any clients of the change and thus trigger the queue listener again. 
>
> This is further complicated by the fact that we might want some of the queue listeners to execute (e.g. index update) while the event source should probably not be notified again.
>
> I currently see two possible solutions to this:

You could have the notion of a message and a reply. When an update
happens it sends a message with a message ID to the queue. Workers work
and then send a reply or further update that contains a the original
message in a "in-reply-to" attribute back to the source. If the source
sees a message with "in-reply-to" to something they have already sent,
they process it, but do not generate a new message for the queue.

Alternatively you can have a TTL mechanism where the original message
gets assigned a time to live, probably a pretty low number like 2 or 4.
When a message is received, the TTL is decremented if the TTL is 0 it is
dropped. When a new update is made based on this information it gets
sent with the decremented TTL.

Cheers,
-w

-- 
William Waites           <william.waites at okfn.org>
Mob: +44 789 798 9965    Open Knowledge Foundation
Fax: +44 131 464 4948                Edinburgh, UK

RDF Indexing, Clustering and Inferencing in Python
		http://ordf.org/