[ckan-dev] ckan-harvest NoneType problem

Thu Mar 15 15:15:46 UTC 2012

Hi,

thanks for the info.

I've found a possible workaround for me.
Looking into the code from ckan release branch 1.6.1 I saw that
ckan/logic/action/get.py was changed on line 373 to

name_or_id = data_dict.get("id") or data_dict['name_or_id']

So I changed ckanext/harvest/harvesters/base.py on line 144 to:

data_dict['name_or_id'] = package_dict['name']

So although  the ID's from remote and local are different it returns the
right dataset as long as the name is not changed, which is enough for me at
the moment.

But I hope for a cleaner solution from you guys!

Thanks
michael

2012/3/15 Adrià Mercader <amercadero at gmail.com>

> Hi again,
>
> We have confirmed that this is a bug caused by recent changes on ckan
> core, and we'll try to push a fix as soon as we discuss the best way
> to deal with it.
>
> For reference, here is the corresponding ticket:
>
> http://trac.ckan.org/ticket/2231
>
>
> Cheers,
>
> Adrià
>
> On 15 March 2012 09:28, Adrià Mercader <amercadero at gmail.com> wrote:
> > Hi Michael,
> >
> > It looks like a bug on ckanext-harvest. I'll have a look this morning
> > and come back to you as soon as I get more details.
> >
> > Sorry about that,
> >
> > Adrià
> >
> > On 15 March 2012 08:45, Michael Reichart <michael.reichart at gmail.com>
> wrote:
> >> Hi Adrià,
> >>
> >> I'm using ckan 1.6.1a with the latest ckanext-harvest from github.
> >> I found that the error is indicating, that ckan does not expect to be
> given
> >> an id when creating a new package. When I remove the id like this in
> >> ckanext-harvest base.py:
> >>
> >> del package_dict['id']
> >>
> >> importing works fine - but creates new packages every time the
> harvester is
> >> run even if nothing changed - because obviously the id's from remote and
> >> local don't match.
> >>
> >> Do I have to change the ckan package - creating - process to give a
> remote
> >> id? Or would you say it is better to change the lookup if the package
> >> already exists?
> >>
> >> Can I get the expected behaviour with other ckan / harvester versions?
> >> Unfortunately it looks like we would have to use ckan 1.6.1 due to other
> >> features we need.
> >>
> >> Thanks!
> >> michael
> >>
> >>
> >> 2012/3/14 Adrià Mercader <amercadero at gmail.com>
> >>>
> >>> Hi Michael,
> >>>
> >>> Sorry to hear you are having trouble with the harvesting extension.
> >>> Can you specify which versions of CKAN and ckanext-harvest are you
> >>> using?
> >>>
> >>> Thanks,
> >>>
> >>> Adrià
> >>>
> >>> On 14 March 2012 13:21, Michael Reichart <michael.reichart at gmail.com>
> >>> wrote:
> >>> > Hi,
> >>> >
> >>> > I managed to get rid of the NoneType-Error which seemed to result
> from
> >>> > some
> >>> > harvester-packages that where from previous calls.
> >>> >
> >>> > But now I have the problem, that the fetching of every single dataset
> >>> > results in an error:
> >>> >
> >>> > 2012-03-14 14:08:46,730 ERROR [ckanext.harvest.harvesters.base]
> Invalid
> >>> > package with GUID zuzuege: {'id': [u'Das Eingabefeld id wurde nicht
> >>> > erwartet.']}
> >>> >
> >>> > which basically means "didn't expect to get the input field 'id'"
> >>> >
> >>> > Any ideas why this could happen?? As I said, i try to harvest
> datasets
> >>> > from
> >>> > ckan.data.linz.gv.at to a relatively clean new ckan-installation ...
> >>> >
> >>> > thanks
> >>> > michael
> >>> >
> >>> >
> >>> >
> >>> > 2012/3/14 Michael Reichart <michael.reichart at gmail.com>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I'm trying to use the ckan-harvest extension to harvest from a ckan
> >>> >> (tried
> >>> >> different ones - at the moment: ckan.data.linz.gv.at) to import in
> my
> >>> >> local
> >>> >> dev ckan.
> >>> >>
> >>> >> I wasn't able to import a single package since all failed with
> >>> >> different
> >>> >> errors
> >>> >> (mainly: NotFound('group not found') or ValidationError (because
> >>> >> of {'id':
> >>> >> [u'Das Eingabefeld id wurde nicht erwartet.']})
> >>> >>
> >>> >> Because of this I updated to the last version from github. Now I'm
> >>> >> permanently getting the error:
> >>> >>
> >>> >> AttributeError: 'NoneType' object has no attribute 'type'
> >>> >>
> >>> >> I mean, ok this is clearly a python problem - but am I the only one
> >>> >> experiencing this? Am I missing some configuration? Maybe another
> >>> >> python
> >>> >> version? I'm using Python 2.6
> >>> >> I also tried some of the harvesters vom ckanext-pdeu - but they are
> >>> >> failing with similar errors - the 'NoneType' appears on different
> >>> >> places in
> >>> >> code - even if I fix some, it reappears elsewhere.
> >>> >>
> >>> >> Is there any sort of deeper documentation to ckan-harvest?
> >>> >>
> >>> >> Thanks in advance!
> >>> >>
> >>> >> Michael
> >>> >>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > ckan-dev mailing list
> >>> > ckan-dev at lists.okfn.org
> >>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20120315/94ad832b/attachment-0001.html>