[ckan-discuss] Circular updates problem

Thu Aug 12 13:12:49 BST 2010

Hi again, 

I have the following problem: Queue workers like the archiver will typically listen on the CKAN message queue for any updates, then execute a task and finally they will want to feed some return data (e.g. a resource hash) back into CKAN. The problem with this is pretty obvious: when CKAN is updated, it will notify any clients of the change and thus trigger the queue listener again. 

This is further complicated by the fact that we might want some of the queue listeners to execute (e.g. index update) while the event source should probably not be notified again.

I currently see two possible solutions to this: 

1. Demand that any queue listener should stabilize over time (i.e. it will first check whether the hash already saved with the resource is changed, and only update if there is an actual change). 

2. Introduce a semi-hidden admin-only flag that can be interpreteted by the queue consumer to determine whether it should run at all. This is particularly ugly since it will have to be carried through the entire app (REST API, ORM and Notifications module). 

Since both solutions are pretty ugly, I'm wondering: is there anything better that you can imagine? 

Thanks, 

 Friedrich 