[ckan-dev] ckan-harvest NoneType problem

Thu Mar 15 12:43:47 UTC 2012

Hi again,

We have confirmed that this is a bug caused by recent changes on ckan
core, and we'll try to push a fix as soon as we discuss the best way
to deal with it.

For reference, here is the corresponding ticket:

http://trac.ckan.org/ticket/2231

Cheers,

Adrià

On 15 March 2012 09:28, Adrià Mercader <amercadero at gmail.com> wrote:
> Hi Michael,
>
> It looks like a bug on ckanext-harvest. I'll have a look this morning
> and come back to you as soon as I get more details.
>
> Sorry about that,
>
> Adrià
>
> On 15 March 2012 08:45, Michael Reichart <michael.reichart at gmail.com> wrote:
>> Hi Adrià,
>>
>> I'm using ckan 1.6.1a with the latest ckanext-harvest from github.
>> I found that the error is indicating, that ckan does not expect to be given
>> an id when creating a new package. When I remove the id like this in
>> ckanext-harvest base.py:
>>
>> del package_dict['id']
>>
>> importing works fine - but creates new packages every time the harvester is
>> run even if nothing changed - because obviously the id's from remote and
>> local don't match.
>>
>> Do I have to change the ckan package - creating - process to give a remote
>> id? Or would you say it is better to change the lookup if the package
>> already exists?
>>
>> Can I get the expected behaviour with other ckan / harvester versions?
>> Unfortunately it looks like we would have to use ckan 1.6.1 due to other
>> features we need.
>>
>> Thanks!
>> michael
>>
>>
>> 2012/3/14 Adrià Mercader <amercadero at gmail.com>
>>>
>>> Hi Michael,
>>>
>>> Sorry to hear you are having trouble with the harvesting extension.
>>> Can you specify which versions of CKAN and ckanext-harvest are you
>>> using?
>>>
>>> Thanks,
>>>
>>> Adrià
>>>
>>> On 14 March 2012 13:21, Michael Reichart <michael.reichart at gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > I managed to get rid of the NoneType-Error which seemed to result from
>>> > some
>>> > harvester-packages that where from previous calls.
>>> >
>>> > But now I have the problem, that the fetching of every single dataset
>>> > results in an error:
>>> >
>>> > 2012-03-14 14:08:46,730 ERROR [ckanext.harvest.harvesters.base] Invalid
>>> > package with GUID zuzuege: {'id': [u'Das Eingabefeld id wurde nicht
>>> > erwartet.']}
>>> >
>>> > which basically means "didn't expect to get the input field 'id'"
>>> >
>>> > Any ideas why this could happen?? As I said, i try to harvest datasets
>>> > from
>>> > ckan.data.linz.gv.at to a relatively clean new ckan-installation ...
>>> >
>>> > thanks
>>> > michael
>>> >
>>> >
>>> >
>>> > 2012/3/14 Michael Reichart <michael.reichart at gmail.com>
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm trying to use the ckan-harvest extension to harvest from a ckan
>>> >> (tried
>>> >> different ones - at the moment: ckan.data.linz.gv.at) to import in my
>>> >> local
>>> >> dev ckan.
>>> >>
>>> >> I wasn't able to import a single package since all failed with
>>> >> different
>>> >> errors
>>> >> (mainly: NotFound('group not found') or ValidationError (because
>>> >> of {'id':
>>> >> [u'Das Eingabefeld id wurde nicht erwartet.']})
>>> >>
>>> >> Because of this I updated to the last version from github. Now I'm
>>> >> permanently getting the error:
>>> >>
>>> >> AttributeError: 'NoneType' object has no attribute 'type'
>>> >>
>>> >> I mean, ok this is clearly a python problem - but am I the only one
>>> >> experiencing this? Am I missing some configuration? Maybe another
>>> >> python
>>> >> version? I'm using Python 2.6
>>> >> I also tried some of the harvesters vom ckanext-pdeu - but they are
>>> >> failing with similar errors - the 'NoneType' appears on different
>>> >> places in
>>> >> code - even if I fix some, it reappears elsewhere.
>>> >>
>>> >> Is there any sort of deeper documentation to ckan-harvest?
>>> >>
>>> >> Thanks in advance!
>>> >>
>>> >> Michael
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>