[ckan-dev] Problem with ckanext-harvester and remote_orgs:create

Thu Jan 22 21:19:37 UTC 2015

Can you pass me your Harvest configuration and your authoritation settings.

Mine are

{"read_only": true, "default_tags": ["Gobierno"], "remote_groups":
"only_local", "remote_orgs": "create", "user": "rodrigogrl", "api_key":
"<myapikey>", "force_all": false}

and

## Authorization Settings

ckan.auth.anon_create_dataset = False
ckan.auth.create_unowned_dataset = True
ckan.auth.create_dataset_if_not_in_organization = False
ckan.auth.user_create_groups = False
ckan.auth.user_create_organizations = True
ckan.auth.user_delete_groups = False
ckan.auth.user_delete_organizations = False
ckan.auth.create_user_via_api = False
ckan.auth.create_user_via_web = True
ckan.auth.roles_that_cascade_to_sub_groups = admin

I have now the following error

2015-01-22 22:12:05,100 INFO  [ckanext.harvest.harvesters.ckanharvester]
Organization af31bd09-9949-4793-9056-769ac8daf3d7 is not available
2015-01-22 22:12:05,965 DEBUG [ckanext.harvest.harvesters.base]
NotAuthorized()

Sorry for the inconvenience

Rodrigo Valdez
0981-530572

2015-01-22 14:12 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:

> Nevermind, I just learned, that it should work with either slug-name
> or the GUID. So false alarm from my side, my previous (already merged)
> fix is sufficient to solve your problem :)
>
> I can harvest http://datos.gov.py/ with previous fix as well, so it
> should work for you as well.
> Please check that you killed the gather_consumer and fetch_consumer
> after the update and restarted them again.
>
> - Stefan
>
> On Thu, Jan 22, 2015 at 5:46 PM, Stefan Oderbolz
> <stefan.oderbolz at liip.ch> wrote:
> > Thanks for the URLs, I could replicate the error. The CKAN instance I
> > used to test with, didn't have the organizations GUID in the
> > "owner_org" field, but rather it's slug name.
> > Therefore I didn't catch that this could be the case on other instances.
> >
> > I just created another PR to fix this issue:
> > https://github.com/ckan/ckanext-harvest/pull/123
> >
> > With this fix I could successfully harvest http://datos.gov.py/
> > including all its organizations and groups.
> >
> > - Stefan
> >
> >
> > On Thu, Jan 22, 2015 at 4:42 PM, Rodrigo Valdez <rodri.valdez at gmail.com>
> wrote:
> >> Sorry, I think you are asking for the URLs
> >>
> >> The local instance is http://datos.org.py/datos/  and the remote is
> >> http://datos.gov.py
> >>
> >> Thanks for the help
> >>
> >>
> >> Rodrigo Valdez
> >> 0981-530572
> >>
> >>
> >> 2015-01-22 12:21 GMT-03:00 Rodrigo Valdez <rodri.valdez at gmail.com>:
> >>
> >>> Hi Stefan,
> >>>
> >>> The local and the remote are 2.2 CKAN instances.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Rodrigo Valdez
> >>> 0981-530572
> >>>
> >>>
> >>> 2015-01-22 12:07 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
> >>>
> >>>> Hi Rodrigo,
> >>>>
> >>>> can you tell me which instance you're trying to harvest?
> >>>>
> >>>> - Stefan
> >>>>
> >>>> On Wed, Jan 21, 2015 at 8:31 PM, Rodrigo Valdez <
> rodri.valdez at gmail.com>
> >>>> wrote:
> >>>> > Hi Stefan,
> >>>> >
> >>>> > Thank you for helping us, I was trying again to create the orgs and
> I
> >>>> > found
> >>>> > a new problem.
> >>>> >
> >>>> > INFO  [ckanext.harvest.harvesters.ckanharvester] Organization
> >>>> > a4016a94-ea46-4671-907e-b56b51d86efa is not available
> >>>> > ERROR [ckanext.harvest.harvesters.ckanharvester] Could not get
> remote
> >>>> > org
> >>>> > a4016a94-ea46-4671-907e-b56b51d86efa
> >>>> >
> >>>> > Its seem to be an error here
> >>>> >
> >>>> > Regards
> >>>> > Rodrigo Valdez
> >>>> >
> >>>> >
> >>>> >
> >>>> > Rodrigo Valdez
> >>>> > 0981-530572
> >>>> >
> >>>> >
> >>>> > 2015-01-15 7:56 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch
> >:
> >>>> >
> >>>> >> Hi again,
> >>>> >>
> >>>> >> my PR just got merged, so if you update ckanext-harvest it should
> work
> >>>> >> :)
> >>>> >>
> >>>> >> - Stefan
> >>>> >>
> >>>> >> On Tue, Jan 13, 2015 at 2:53 PM, Stefan Oderbolz
> >>>> >> <stefan.oderbolz at liip.ch> wrote:
> >>>> >> > Hi Rodrigo(s),
> >>>> >> >
> >>>> >> > sorry for the confusion, didn't pay close enough attention to
> spot
> >>>> >> > your last names :)
> >>>> >> > In the meantime I had a look at the issue and created a pull
> request
> >>>> >> > with a possible solution:
> >>>> >> > https://github.com/ckan/ckanext-harvest/pull/121
> >>>> >> >
> >>>> >> > - Stefan
> >>>> >> >
> >>>> >> > On Mon, Jan 5, 2015 at 2:23 PM, Rodrigo Parra <
> rodpar07 at gmail.com>
> >>>> >> > wrote:
> >>>> >> >> Thanks Stefan,
> >>>> >> >>
> >>>> >> >> Actually, Rodrigo Valdez is a different developer, we are both
> >>>> >> >> paraguayans
> >>>> >> >> though :)
> >>>> >> >> He is working with the "client" CKAN instance, while I am
> >>>> >> >> maintaining
> >>>> >> >> the
> >>>> >> >> "source" CKAN instance.
> >>>> >> >> I didn't mean to spam the list, I just did not see his previous
> >>>> >> >> mail.
> >>>> >> >>
> >>>> >> >> I would really appreciate your help, if you give me a hint on
> where
> >>>> >> >> to
> >>>> >> >> look
> >>>> >> >> at, I might be able to lend a hand.
> >>>> >> >> Otherwise, we'll probably wait for your fix and cherry-pick it.
> >>>> >> >>
> >>>> >> >> Cheers,
> >>>> >> >> Rodrigo (Parra)
> >>>> >> >>
> >>>> >> >> 2015-01-05 8:46 GMT-03:00 Stefan Oderbolz
> >>>> >> >> <stefan.oderbolz at liip.ch>:
> >>>> >> >>
> >>>> >> >>> Ah, just found your "newer" email.
> >>>> >> >>>
> >>>> >> >>> It seems that this route used to return organizations as well,
> but
> >>>> >> >>> doesn't
> >>>> >> >>> return them anymore now. This seems to be a regression. If I
> find
> >>>> >> >>> some
> >>>> >> >>> time
> >>>> >> >>> this week, I'll look into that.
> >>>> >> >>>
> >>>> >> >>> Regards Stefan
> >>>> >> >>>
> >>>> >> >>> On Mon, Dec 22, 2014 at 7:45 PM, Rodrigo Parra
> >>>> >> >>> <rodpar07 at gmail.com>
> >>>> >> >>> wrote:
> >>>> >> >>>>
> >>>> >> >>>> Hello all,
> >>>> >> >>>>
> >>>> >> >>>> I'm trying to import datasets from a remote CKAN 2.2 instance,
> >>>> >> >>>> and I
> >>>> >> >>>> would like to import owner organizations as well. According to
> >>>> >> >>>> the
> >>>> >> >>>> harvester
> >>>> >> >>>> docs, it should be possible (and fairly easy) to do so using
> the
> >>>> >> >>>> remote_orgs:"create" configuration. However, after doing some
> >>>> >> >>>> tests,
> >>>> >> >>>> no
> >>>> >> >>>> organization is being imported.
> >>>> >> >>>>
> >>>> >> >>>> Looking at the code, the remote organizations are being
> fetched
> >>>> >> >>>> using
> >>>> >> >>>> the
> >>>> >> >>>> /group route of the v2 of the API. But testing the
> >>>> >> >>>> api/2/rest/group/
> >>>> >> >>>> with
> >>>> >> >>>> Postman, I can see it only returns a list of ids from groups,
> not
> >>>> >> >>>> organizations.
> >>>> >> >>>>
> >>>> >> >>>> I believe that's the reason why the harvester can't find the
> >>>> >> >>>> owner_org.
> >>>> >> >>>> Is this a known bug? Should api/2/rest/group/ return
> organization
> >>>> >> >>>> ids
> >>>> >> >>>> too
> >>>> >> >>>> or is what I'm getting expected behavior?
> >>>> >> >>>>
> >>>> >> >>>> In case the latter is true, should we change CKANHarvester to
> use
> >>>> >> >>>> v3
> >>>> >> >>>> of
> >>>> >> >>>> the api and /organization_show ? Or is there a better
> workaround?
> >>>> >> >>>>
> >>>> >> >>>> Thanks for your help in advance.
> >>>> >> >>>> Cheers,
> >>>> >> >>>>
> >>>> >> >>>> Rodrigo Parra
> >>>> >> >>>>
> >>>> >> >>>> _______________________________________________
> >>>> >> >>>> ckan-dev mailing list
> >>>> >> >>>> ckan-dev at lists.okfn.org
> >>>> >> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>> >> >>>>
> >>>> >> >>>
> >>>> >> >>>
> >>>> >> >>>
> >>>> >> >>> --
> >>>> >> >>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>>> >> >>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>>> >> >>>
> >>>> >> >>> _______________________________________________
> >>>> >> >>> ckan-dev mailing list
> >>>> >> >>> ckan-dev at lists.okfn.org
> >>>> >> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>> >> >>>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> _______________________________________________
> >>>> >> >> ckan-dev mailing list
> >>>> >> >> ckan-dev at lists.okfn.org
> >>>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>> >> >>
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> > --
> >>>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>>> >> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>>> >> _______________________________________________
> >>>> >> ckan-dev mailing list
> >>>> >> ckan-dev at lists.okfn.org
> >>>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>> >
> >>>> >
> >>>> >
> >>>> > _______________________________________________
> >>>> > ckan-dev mailing list
> >>>> > ckan-dev at lists.okfn.org
> >>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>>> _______________________________________________
> >>>> ckan-dev mailing list
> >>>> ckan-dev at lists.okfn.org
> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> >
> >
> > --
> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>
>
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150122/cf442fa7/attachment-0003.html>