[ckan-dev] Harvest cron removal

Tue Jul 30 12:37:27 UTC 2013

We (Canada/data.gc.ca) are not using the harvester at the moment, so this
does not have an impact on us.

Thanks.

On 30 July 2013 04:56, David Read <david.read at hackneyworkshop.com> wrote:

> I'm keen to make the CKAN harvester start immediately when you hit the
> 'refresh source' button. Currently it has to wait for the 'harvester
> run' cron which is usually configured to occur only every 15 minutes.
>
> I get plenty of support calls from organisations who are setting up a
> harvester, are iterating through problems. They are further frustrated
> by this curious wait each time. Since the wait time is somewhat
> 'unknown' it also breeds distrust. And we all hate being made to
> context-switch away while debugging.
>
> From conversations with James, the 15 minute cron was originally
> designed to ensure that a harvest source couldn't be harvested more
> than once simultaneously and get confused. Since the harvest job
> becomes multiple gathers and further split into fetches, James was
> keen to make sure a source could only have one job for each run of the
> cron. BUT I'm sure we could get the backend (Redis if not RabbitMQ) to
> tell us the status of the job (and its related gather/fetches) and
> stop further ones being created until it is done.
>
> Would the various CKAN sites using the harvester (OKF, Frauenhofer,
> Canada, Washington etc.) be happy about this change? It would be on
> the CKAN 2.x code.
>
> Dave
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130730/4f246467/attachment-0001.html>