[ckan-dev] AMQP topic vs direct exchange

[James Gardner]

Hi Friedrich,

> This is true. I've had some problems in the past where a lot of items
> accumulated on the queue so I chose the topic queue partly for that
> reason.

Nils is setting up some pretty serious monitoring so I'd expect his
setup to be able to catch these types of problems directly rather than
resorting to a setup where messages are lost instead?

> The other reason is that it allows us to play with instance
> identifiers, i.e. someone could listen on #.ckan.net to get updates
> from all country instances (so you would only have to run the archiver
> once instead of per-instance). Since we don't do that at the moment,
> this could be removed, though. Either way, it should be a config
> setting.

Maybe we need two types of ckanext-queue then? I think message
persistence is more important than the ability to having just one worker
instance, certainly for the things I'd use it for eg harvesting, link
checker etc. You can still do routing with the direct exchange, just
needs considering upfront more.

Interested to hear other people's views too?

Cheers,

James