[ckan-dev] Running multiple harvesters

Fri Nov 14 15:14:17 UTC 2014

Stefan,

There's only one gather queue and one fetch queue, however many sorts
of harvesters you have, so there is no need to run more than one of
each of the consumers.

David

On 14 November 2014 14:32, Stefan Oderbolz <stefan.oderbolz at liip.ch> wrote:
> Hi there,
>
> just a quick question: When I run multiple harvesters (all based on
> ckanext-harvest), it doesn't make sense to run a "gather_consumer" and
> "fetch_consumer" for each of them if they all share the same queue?
>
> I'm asking because up until now we always did that, and now ran into
> problems with too many commits to Solr at the same time. I suspect that the
> multiple processes all do the same and just conflict each other.
>
> Or is it best practice to use separate queues for each harvester? If so, how
> do set that up?
>
> - Stefan
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>