[ckan-dev] (no subject) harvesting

[Emanuele Tajariol]

Hi all,

my usual setup is:

- running the gather_consumer and the fetch_consumer using supervisord; in 
this way, should one of them terminate for an exception, it will be 
automatically restarted;

- having a cron job that every 15 minutes performs an "harvester run", which 
  will 
  - put in the finished state all the harvesters that have completed their 
    work, 
  - will launch harvesters set to run periodically.

It's a quite stable setup, and I have seldom, if ever, needed to cleanup the 
queues.


   Cheers,
   Emanuele


Alle 05:36:14 di Tuesday 31 October 2017, Stefan Oderbolz ha scritto:
> If the job is done, the paster run command will mark it as "Finished",
> otherwise you see a message of an ongoing job.
> 
> There are many possibilites why this might fail, here is usually what I do
> to debug the situation:
> 
> 1. Check the logs (apache error log and/or CKAN logs), you can either spot
> activity (i.e. Ongoing harvest job or maybe an error message)
> 
> 2. Check if gather_consumer and fetch_consumer are still running (using ps
> ax | grep paster). If they are not, restart them. Note: this might happen
> if an exception was raised during the harvesting
> 
> 3. Maybe some data in the queues is wrong, causes esceptions etc. so at
> this point I purge the queues using the paster purge_queues command
> followed by a paster run
> 
> 4. As a kind of last resort, if nothing else helps, I'll run paster
> job_abort, which will set the status to "Finished"
> 
> On Oct 30, 2017 21:33, "Karen Turner" <karen at catalyst.net.nz> wrote:
> > Hi
> > 
> > I have also noticed that jobs do not always end when reharvesting.
> > Sometimes I can get them to end by doing  a harvest run on the job using
> > paster.
> > The jobs do appear to have finished - all the data sets are harvested by
> > some last step is not being done.
> > At this point I do an abort as a work around. I haven't reported the
> > problem as I am not on the latest harvest or the latest ckan 2.6. I am
> > in the process of upgrading.
> > 
> > On 30/10/17 17:42, 김다솔 wrote:
> > > Also, If I run reharvest on the web, the job will not end. This is also
> > > done by running the harvest run command in shell.
> > > 
> > > I do not know how to solve these problems.
> > 
> > --
> > Karen Turner
> > 
> > Catalyst IT Limited
> > Level 6,150-154 Willis Street, PO Box 11-053
> > Wellington New Zealand
> > 
> > w: http://catalyst.net.nz
> > p: +64 4 803 2375
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev


-- 

Regards,
Emanuele Tajariol
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V 
for more information.
==

Ing. Emanuele Tajariol
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
 
The information in this message and/or attachments, is intended solely for the 
attention and use of the named addressee(s) and may be confidential or 
proprietary in nature or covered by the provisions of privacy act (Legislative 
Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not 
in accord with its purpose, any disclosure, reproduction, copying, 
distribution, or either dissemination, either whole or partial, is strictly 
forbidden except previous formal approval of the named addressee(s). If you 
are not the intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message that has 
been received in error. The sender does not give any warranty or accept 
liability as the content, accuracy or completeness of sent messages and 
accepts no responsibility  for changes made after they were sent or for other 
risks which arise as a result of e-mail transmission, viruses, etc.