[ckan-discuss] Circular updates problem

Thu Aug 12 14:17:34 BST 2010

Pudo,

Great to see all the good work with plugins etc.!

On the queue workers problem, solution 1 doesn't seem ugly to me. If
some parts of the package are 'auto-generated' by a worker then these
changes should be notified to clients in the same way as normal
changes. You only get a problem when you get an infinite loop - maybe
we should test for this being caused by badly written workers?

For example, your archiver will check for any new resources, and if
so, add a resource pointing to the cached copy. The next time it is
run it needs to be intelligent enough to see that there is the
original resource and the cache of it, so no new changes are required.
You'd surely have to put this basic intelligence in, whatever the
method of avoiding a loop?

David

On 12 August 2010 13:12, Friedrich Lindenberg <friedrich at pudo.org> wrote:
> Hi again,
>
> I have the following problem: Queue workers like the archiver will typically listen on the CKAN message queue for any updates, then execute a task and finally they will want to feed some return data (e.g. a resource hash) back into CKAN. The problem with this is pretty obvious: when CKAN is updated, it will notify any clients of the change and thus trigger the queue listener again.
>
> This is further complicated by the fact that we might want some of the queue listeners to execute (e.g. index update) while the event source should probably not be notified again.
>
> I currently see two possible solutions to this:
>
> 1. Demand that any queue listener should stabilize over time (i.e. it will first check whether the hash already saved with the resource is changed, and only update if there is an actual change).
>
> 2. Introduce a semi-hidden admin-only flag that can be interpreteted by the queue consumer to determine whether it should run at all. This is particularly ugly since it will have to be carried through the entire app (REST API, ORM and Notifications module).
>
> Since both solutions are pretty ugly, I'm wondering: is there anything better that you can imagine?
>
> Thanks,
>
>  Friedrich
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>