[ckan-dev] (no subject) harvesting

Tue Oct 31 04:36:14 UTC 2017

If the job is done, the paster run command will mark it as "Finished",
otherwise you see a message of an ongoing job.

There are many possibilites why this might fail, here is usually what I do
to debug the situation:

1. Check the logs (apache error log and/or CKAN logs), you can either spot
activity (i.e. Ongoing harvest job or maybe an error message)

2. Check if gather_consumer and fetch_consumer are still running (using ps
ax | grep paster). If they are not, restart them. Note: this might happen
if an exception was raised during the harvesting

3. Maybe some data in the queues is wrong, causes esceptions etc. so at
this point I purge the queues using the paster purge_queues command
followed by a paster run

4. As a kind of last resort, if nothing else helps, I'll run paster
job_abort, which will set the status to "Finished"

On Oct 30, 2017 21:33, "Karen Turner" <karen at catalyst.net.nz> wrote:

> Hi
>
> I have also noticed that jobs do not always end when reharvesting.
> Sometimes I can get them to end by doing  a harvest run on the job using
> paster.
> The jobs do appear to have finished - all the data sets are harvested by
> some last step is not being done.
> At this point I do an abort as a work around. I haven't reported the
> problem as I am not on the latest harvest or the latest ckan 2.6. I am
> in the process of upgrading.
>
> On 30/10/17 17:42, 김다솔 wrote:
>
> > Also, If I run reharvest on the web, the job will not end. This is also
> > done by running the harvest run command in shell.
> >
> > I do not know how to solve these problems.
>
>
> --
> Karen Turner
>
> Catalyst IT Limited
> Level 6,150-154 Willis Street, PO Box 11-053
> Wellington New Zealand
>
> w: http://catalyst.net.nz
> p: +64 4 803 2375
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171031/899537a2/attachment-0003.html>