[ckan-dev] Running multiple harvesters

[Stefan Oderbolz]

Hi there,

just a quick question: When I run multiple harvesters (all based on
ckanext-harvest), it doesn't make sense to run a "gather_consumer" and
"fetch_consumer" for each of them if they all share the same queue?

I'm asking because up until now we always did that, and now ran into
problems with too many commits to Solr at the same time. I suspect that the
multiple processes all do the same and just conflict each other.

Or is it best practice to use separate queues for each harvester? If so,
how do set that up?

- Stefan

-- 
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20141114/9e0a4461/attachment-0002.html>