[ckan-dev] ckanext-harvest: "There are no new harvesting jobs"

Tue Jul 21 10:20:45 UTC 2015

Hi,

I've merged Stefan's patch to avoid the exception.

As a general rule, if there is something really ugly happening in CKAN
it's probably because it's really old (this code was written on April
2011) and we somehow have got used to its ugliness. Any PRs like this
to make it prettier are very welcome!

Cheers,

Adrià

On 20 July 2015 at 17:25, Stefan Oderbolz <stefan.oderbolz at liip.ch> wrote:
> I created a ticket on the ckanext-harvest repo
> (https://github.com/ckan/ckanext-harvest/issues/145) and will quickly add a
> PR to fix this ugly situation.
>
> On Mon, Jul 20, 2015 at 6:03 PM, Denis Zgonjanin <deniszgonjanin at gmail.com>
> wrote:
>>
>> Yes, I'm not sure whether there is a reason that needs to be there.
>> Perhaps someone more familiar with harvester could weigh in.
>>
>> But it would be better if it was just a message instead of a stack trace.
>> For new users especially, where seeing an error will usually give you the
>> impression that you didn't set something up properly.
>>
>> On Mon, Jul 20, 2015 at 11:49 AM, Stefan Oderbolz
>> <stefan.oderbolz at liip.ch> wrote:
>>>
>>> This is a good point, I'm already used to see this exception, it no
>>> longer bothers me. But it might be worth catching this exception and simply
>>> return a nice message indicating what's going on.
>>> Or is there a good reason for this behavior?
>>>
>>> - Stefan
>>>
>>> On Mon, Jul 20, 2015 at 5:28 PM, Denis Zgonjanin
>>> <deniszgonjanin at gmail.com> wrote:
>>>>
>>>> Hi Knud,
>>>>
>>>> This is normal behavior (although one could make an argument that in the
>>>> case of normal behavior software shouldn't throw an error).
>>>>
>>>> Jobs will run on a harvesting schedule, which you define when you create
>>>> the harvester in the web UI. For example, you create a daily harvester and
>>>> do 'harvester run'. The job runs, and harvested will compute when the next
>>>> job is due to be ran (24 hours from now). If you do `harvester run` before
>>>> then, you will get the expected error above.
>>>>
>>>> - Denis
>>>>
>>>> On Mon, Jul 20, 2015 at 11:23 AM, Knud Hinnerk Möller
>>>> <knud at datalysator.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have a problem setting up ckanext-harvest for production. I’m
>>>>> following the steps explained in
>>>>> https://github.com/ckan/ckanext-harvest/blob/master/README.rst#setting-up-the-harvesters-on-a-production-server,
>>>>> which works great at first. However, after the first time the harvester is
>>>>> run, I get an error message saying "Exception: There are no new harvesting
>>>>> jobs” each time the harvester is run again via cron. It seems I have to
>>>>> first create a new harvesting job via command line or Web UI. Is this
>>>>> intended behaviour? Or am I missing something? Could someone point me in the
>>>>> right direction?
>>>>>
>>>>> ---------
>>>>> 2015-07-20 17:00:03,530 DEBUG [ckan.config.environment] Loading the
>>>>> synchronous search plugin
>>>>> 2015-07-20 17:00:06,606 DEBUG [ckanext.harvest.model] Harvest tables
>>>>> defined in memory
>>>>> 2015-07-20 17:00:06,620 DEBUG [ckanext.harvest.model] Harvest tables
>>>>> already exist
>>>>> 2015-07-20 17:00:06,636 DEBUG [ckanext.berlin.plugin] get_actions
>>>>> 2015-07-20 17:00:06,667 DEBUG [ckanext.harvest.plugin] Using auth
>>>>> profile at ckanext.harvest.logic.auth
>>>>> 2015-07-20 17:00:06,699 INFO  [ckanext.harvest.logic.action.update]
>>>>> Harvest job run: {}
>>>>> 2015-07-20 17:00:06,734 INFO  [ckanext.harvest.logic.action.update] No
>>>>> new harvest jobs.
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File ".../pyenv/bin/paster", line 9, in <module>
>>>>>     load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>>> 'paster')()
>>>>>   File
>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>> 104, in run
>>>>>     invoke(command, command_name, options, args[1:])
>>>>>   File
>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>> 143, in invoke
>>>>>     exit_code = runner.run(args)
>>>>>   File
>>>>> ".../pyenv/local/lib/python2.7/site-packages/paste/script/command.py", line
>>>>> 238, in run
>>>>>     result = self.command()
>>>>>   File
>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line
>>>>> 108, in command
>>>>>     self.run_harvester()
>>>>>   File
>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line
>>>>> 243, in run_harvester
>>>>>     jobs = get_action('harvest_jobs_run')(context,{})
>>>>>   File
>>>>> ".../pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line
>>>>> 145, in harvest_jobs_run
>>>>>     raise Exception('There are no new harvesting jobs')
>>>>> Exception: There are no new harvesting jobs
>>>>> ------------
>>>>>
>>>>> I should add that I’m running an old version of the ckanext-harvest
>>>>> (@0609262731) and an old version of CKAN (1.8.2). I don’t think that’s the
>>>>> reason, though.
>>>>>
>>>>> Cheers,
>>>>> Knud
>>>>>
>>>>> --
>>>>> Dr. Knud Hinnerk Möller
>>>>> Datalysator - Web Data Solutions and Training
>>>>> knud at datalysator.com
>>>>> +49 151 46 44 83 25
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>