[ckan-dev] ckanext-harvest: "There are no new harvesting jobs"

Thu Jul 23 10:16:43 UTC 2015

Thanks all for the explanations and the fix!

Cheers,
Knud

> On 21 Jul 2015, at 12:20, Adrià Mercader <adria.mercader at okfn.org> wrote:
> 
> Hi,
> 
> I've merged Stefan's patch to avoid the exception.
> 
> As a general rule, if there is something really ugly happening in CKAN
> it's probably because it's really old (this code was written on April
> 2011) and we somehow have got used to its ugliness. Any PRs like this
> to make it prettier are very welcome!
> 
> Cheers,
> 
> Adrià
> 
> On 20 July 2015 at 17:25, Stefan Oderbolz <stefan.oderbolz at liip.ch> wrote:
>> I created a ticket on the ckanext-harvest repo
>> (https://github.com/ckan/ckanext-harvest/issues/145) and will quickly add a
>> PR to fix this ugly situation.
>> 
>> On Mon, Jul 20, 2015 at 6:03 PM, Denis Zgonjanin <deniszgonjanin at gmail.com>
>> wrote:
>>> 
>>> Yes, I'm not sure whether there is a reason that needs to be there.
>>> Perhaps someone more familiar with harvester could weigh in.
>>> 
>>> But it would be better if it was just a message instead of a stack trace.
>>> For new users especially, where seeing an error will usually give you the
>>> impression that you didn't set something up properly.
>>> 
>>> On Mon, Jul 20, 2015 at 11:49 AM, Stefan Oderbolz
>>> <stefan.oderbolz at liip.ch> wrote:
>>>> 
>>>> This is a good point, I'm already used to see this exception, it no
>>>> longer bothers me. But it might be worth catching this exception and simply
>>>> return a nice message indicating what's going on.
>>>> Or is there a good reason for this behavior?
>>>> 
>>>> - Stefan
>>>> 
>>>> On Mon, Jul 20, 2015 at 5:28 PM, Denis Zgonjanin
>>>> <deniszgonjanin at gmail.com> wrote:
>>>>> 
>>>>> Hi Knud,
>>>>> 
>>>>> This is normal behavior (although one could make an argument that in the
>>>>> case of normal behavior software shouldn't throw an error).
>>>>> 
>>>>> Jobs will run on a harvesting schedule, which you define when you create
>>>>> the harvester in the web UI. For example, you create a daily harvester and
>>>>> do 'harvester run'. The job runs, and harvested will compute when the next
>>>>> job is due to be ran (24 hours from now). If you do `harvester run` before
>>>>> then, you will get the expected error above.
>>>>> 
>>>>> - Denis
>>>>> 
>>>>> On Mon, Jul 20, 2015 at 11:23 AM, Knud Hinnerk Möller
>>>>> <knud at datalysator.com> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I have a problem setting up ckanext-harvest for production. I’m
>>>>>> following the steps explained in
>>>>>> https://github.com/ckan/ckanext-harvest/blob/master/README.rst#setting-up-the-harvesters-on-a-production-server,
>>>>>> which works great at first. However, after the first time the harvester is
>>>>>> run, I get an error message saying "Exception: There are no new harvesting
>>>>>> jobs” each time the harvester is run again via cron. It seems I have to
>>>>>> first create a new harvesting job via command line or Web UI. Is this
>>>>>> intended behaviour? Or am I missing something? Could someone point me in the
>>>>>> right direction?
>>>>>> 
>>>>>> ---------
>>>>>> 2015-07-20 17:00:03,530 DEBUG [ckan.config.environment] Loading the
>>>>>> synchronous search plugin
>>>>>> 2015-07-20 17:00:06,606 DEBUG [ckanext.harvest.model] Harvest tables
>>>>>> defined in memory
>>>>>> 2015-07-20 17:00:06,620 DEBUG [ckanext.harvest.model] Harvest tables
>>>>>> already exist
>>>>>> 2015-07-20 17:00:06,636 DEBUG [ckanext.berlin.plugin] get_actions
>>>>>> 2015-07-20 17:00:06,667 DEBUG [ckanext.harvest.plugin] Using auth
>>>>>> profile at ckanext.harvest.logic.auth
>>>>>> 2015-07-20 17:00:06,699 INFO  [ckanext.harvest.logic.action.update]
>>>>>> Harvest job run: {}
>>>>>> 2015-07-20 17:00:06,734 INFO  [ckanext.harvest.logic.action.update] No
>>>>>> new harvest jobs.
>>>>>> 
>>>>>> Traceback (most recent call last):
>>>>>>  File ".../pyenv/bin/paster", line 9, in <module>
>>>>>>    load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>>>> 'paster')()
>>>>>>  File
>>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>>> 104, in run
>>>>>>    invoke(command, command_name, options, args[1:])
>>>>>>  File
>>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>>> 143, in invoke
>>>>>>    exit_code = runner.run(args)
>>>>>>  File
>>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>>> 238, in run
>>>>>>    result = self.command()
>>>>>>  File
>>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line
>>>>>> 108, in command
>>>>>>    self.run_harvester()
>>>>>>  File
>>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line
>>>>>> 243, in run_harvester
>>>>>>    jobs = get_action('harvest_jobs_run')(context,{})
>>>>>>  File
>>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line
>>>>>> 145, in harvest_jobs_run
>>>>>>    raise Exception('There are no new harvesting jobs')
>>>>>> Exception: There are no new harvesting jobs
>>>>>> ------------
>>>>>> 
>>>>>> I should add that I’m running an old version of the ckanext-harvest
>>>>>> (@0609262731) and an old version of CKAN (1.8.2). I don’t think that’s the
>>>>>> reason, though.
>>>>>> 
>>>>>> Cheers,
>>>>>> Knud
>>>>>> 
>>>>>> --
>>>>>> Dr. Knud Hinnerk Möller
>>>>>> Datalysator - Web Data Solutions and Training
>>>>>> knud at datalysator.com
>>>>>> +49 151 46 44 83 25
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> ckan-dev mailing list
>>>>>> ckan-dev at lists.okfn.org
>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>> 
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> 
>> 
>> 
>> 
>> --
>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 
Dr. Knud Hinnerk Möller
Datalysator - Web Data Solutions and Training
knud at datalysator.com
+49 151 46 44 83 25

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150723/27a8ded8/attachment-0003.html>