[ckan-dev] Problem with ckanext-harvester and remote_orgs:create

Fri Jan 23 08:42:15 UTC 2015

My harvester config:

{"force_all": true, "remote_groups": "create", "remote_orgs": "create"}

Here are my auth settings:

ckan.auth.anon_create_dataset = false
ckan.auth.create_unowned_dataset = true
ckan.auth.create_dataset_if_not_in_organization = true
ckan.auth.user_create_groups = true
ckan.auth.user_create_organizations = true
ckan.auth.user_delete_groups = true
ckan.auth.user_delete_organizations = true
ckan.auth.create_user_via_api = false

I'm not sure this helps.

On Thu, Jan 22, 2015 at 10:19 PM, Rodrigo Valdez <rodri.valdez at gmail.com> wrote:
> Can you pass me your Harvest configuration and your authoritation settings.
>
> Mine are
>
> {"read_only": true, "default_tags": ["Gobierno"], "remote_groups":
> "only_local", "remote_orgs": "create", "user": "rodrigogrl", "api_key":
> "<myapikey>", "force_all": false}
>
> and
>
> ## Authorization Settings
>
> ckan.auth.anon_create_dataset = False
> ckan.auth.create_unowned_dataset = True
> ckan.auth.create_dataset_if_not_in_organization = False
> ckan.auth.user_create_groups = False
> ckan.auth.user_create_organizations = True
> ckan.auth.user_delete_groups = False
> ckan.auth.user_delete_organizations = False
> ckan.auth.create_user_via_api = False
> ckan.auth.create_user_via_web = True
> ckan.auth.roles_that_cascade_to_sub_groups = admin
>
>
> I have now the following error
>
>
> 2015-01-22 22:12:05,100 INFO  [ckanext.harvest.harvesters.ckanharvester]
> Organization af31bd09-9949-4793-9056-769ac8daf3d7 is not available
> 2015-01-22 22:12:05,965 DEBUG [ckanext.harvest.harvesters.base]
> NotAuthorized()
>
> Sorry for the inconvenience
>
>
>
>
>
>
> Rodrigo Valdez
> 0981-530572
>
>
> 2015-01-22 14:12 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>
>> Nevermind, I just learned, that it should work with either slug-name
>> or the GUID. So false alarm from my side, my previous (already merged)
>> fix is sufficient to solve your problem :)
>>
>> I can harvest http://datos.gov.py/ with previous fix as well, so it
>> should work for you as well.
>> Please check that you killed the gather_consumer and fetch_consumer
>> after the update and restarted them again.
>>
>> - Stefan
>>
>> On Thu, Jan 22, 2015 at 5:46 PM, Stefan Oderbolz
>> <stefan.oderbolz at liip.ch> wrote:
>> > Thanks for the URLs, I could replicate the error. The CKAN instance I
>> > used to test with, didn't have the organizations GUID in the
>> > "owner_org" field, but rather it's slug name.
>> > Therefore I didn't catch that this could be the case on other instances.
>> >
>> > I just created another PR to fix this issue:
>> > https://github.com/ckan/ckanext-harvest/pull/123
>> >
>> > With this fix I could successfully harvest http://datos.gov.py/
>> > including all its organizations and groups.
>> >
>> > - Stefan
>> >
>> >
>> > On Thu, Jan 22, 2015 at 4:42 PM, Rodrigo Valdez <rodri.valdez at gmail.com>
>> > wrote:
>> >> Sorry, I think you are asking for the URLs
>> >>
>> >> The local instance is http://datos.org.py/datos/  and the remote is
>> >> http://datos.gov.py
>> >>
>> >> Thanks for the help
>> >>
>> >>
>> >> Rodrigo Valdez
>> >> 0981-530572
>> >>
>> >>
>> >> 2015-01-22 12:21 GMT-03:00 Rodrigo Valdez <rodri.valdez at gmail.com>:
>> >>
>> >>> Hi Stefan,
>> >>>
>> >>> The local and the remote are 2.2 CKAN instances.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Rodrigo Valdez
>> >>> 0981-530572
>> >>>
>> >>>
>> >>> 2015-01-22 12:07 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>> >>>
>> >>>> Hi Rodrigo,
>> >>>>
>> >>>> can you tell me which instance you're trying to harvest?
>> >>>>
>> >>>> - Stefan
>> >>>>
>> >>>> On Wed, Jan 21, 2015 at 8:31 PM, Rodrigo Valdez
>> >>>> <rodri.valdez at gmail.com>
>> >>>> wrote:
>> >>>> > Hi Stefan,
>> >>>> >
>> >>>> > Thank you for helping us, I was trying again to create the orgs and
>> >>>> > I
>> >>>> > found
>> >>>> > a new problem.
>> >>>> >
>> >>>> > INFO  [ckanext.harvest.harvesters.ckanharvester] Organization
>> >>>> > a4016a94-ea46-4671-907e-b56b51d86efa is not available
>> >>>> > ERROR [ckanext.harvest.harvesters.ckanharvester] Could not get
>> >>>> > remote
>> >>>> > org
>> >>>> > a4016a94-ea46-4671-907e-b56b51d86efa
>> >>>> >
>> >>>> > Its seem to be an error here
>> >>>> >
>> >>>> > Regards
>> >>>> > Rodrigo Valdez
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Rodrigo Valdez
>> >>>> > 0981-530572
>> >>>> >
>> >>>> >
>> >>>> > 2015-01-15 7:56 GMT-03:00 Stefan Oderbolz
>> >>>> > <stefan.oderbolz at liip.ch>:
>> >>>> >
>> >>>> >> Hi again,
>> >>>> >>
>> >>>> >> my PR just got merged, so if you update ckanext-harvest it should
>> >>>> >> work
>> >>>> >> :)
>> >>>> >>
>> >>>> >> - Stefan
>> >>>> >>
>> >>>> >> On Tue, Jan 13, 2015 at 2:53 PM, Stefan Oderbolz
>> >>>> >> <stefan.oderbolz at liip.ch> wrote:
>> >>>> >> > Hi Rodrigo(s),
>> >>>> >> >
>> >>>> >> > sorry for the confusion, didn't pay close enough attention to
>> >>>> >> > spot
>> >>>> >> > your last names :)
>> >>>> >> > In the meantime I had a look at the issue and created a pull
>> >>>> >> > request
>> >>>> >> > with a possible solution:
>> >>>> >> > https://github.com/ckan/ckanext-harvest/pull/121
>> >>>> >> >
>> >>>> >> > - Stefan
>> >>>> >> >
>> >>>> >> > On Mon, Jan 5, 2015 at 2:23 PM, Rodrigo Parra
>> >>>> >> > <rodpar07 at gmail.com>
>> >>>> >> > wrote:
>> >>>> >> >> Thanks Stefan,
>> >>>> >> >>
>> >>>> >> >> Actually, Rodrigo Valdez is a different developer, we are both
>> >>>> >> >> paraguayans
>> >>>> >> >> though :)
>> >>>> >> >> He is working with the "client" CKAN instance, while I am
>> >>>> >> >> maintaining
>> >>>> >> >> the
>> >>>> >> >> "source" CKAN instance.
>> >>>> >> >> I didn't mean to spam the list, I just did not see his previous
>> >>>> >> >> mail.
>> >>>> >> >>
>> >>>> >> >> I would really appreciate your help, if you give me a hint on
>> >>>> >> >> where
>> >>>> >> >> to
>> >>>> >> >> look
>> >>>> >> >> at, I might be able to lend a hand.
>> >>>> >> >> Otherwise, we'll probably wait for your fix and cherry-pick it.
>> >>>> >> >>
>> >>>> >> >> Cheers,
>> >>>> >> >> Rodrigo (Parra)
>> >>>> >> >>
>> >>>> >> >> 2015-01-05 8:46 GMT-03:00 Stefan Oderbolz
>> >>>> >> >> <stefan.oderbolz at liip.ch>:
>> >>>> >> >>
>> >>>> >> >>> Ah, just found your "newer" email.
>> >>>> >> >>>
>> >>>> >> >>> It seems that this route used to return organizations as well,
>> >>>> >> >>> but
>> >>>> >> >>> doesn't
>> >>>> >> >>> return them anymore now. This seems to be a regression. If I
>> >>>> >> >>> find
>> >>>> >> >>> some
>> >>>> >> >>> time
>> >>>> >> >>> this week, I'll look into that.
>> >>>> >> >>>
>> >>>> >> >>> Regards Stefan
>> >>>> >> >>>
>> >>>> >> >>> On Mon, Dec 22, 2014 at 7:45 PM, Rodrigo Parra
>> >>>> >> >>> <rodpar07 at gmail.com>
>> >>>> >> >>> wrote:
>> >>>> >> >>>>
>> >>>> >> >>>> Hello all,
>> >>>> >> >>>>
>> >>>> >> >>>> I'm trying to import datasets from a remote CKAN 2.2
>> >>>> >> >>>> instance,
>> >>>> >> >>>> and I
>> >>>> >> >>>> would like to import owner organizations as well. According
>> >>>> >> >>>> to
>> >>>> >> >>>> the
>> >>>> >> >>>> harvester
>> >>>> >> >>>> docs, it should be possible (and fairly easy) to do so using
>> >>>> >> >>>> the
>> >>>> >> >>>> remote_orgs:"create" configuration. However, after doing some
>> >>>> >> >>>> tests,
>> >>>> >> >>>> no
>> >>>> >> >>>> organization is being imported.
>> >>>> >> >>>>
>> >>>> >> >>>> Looking at the code, the remote organizations are being
>> >>>> >> >>>> fetched
>> >>>> >> >>>> using
>> >>>> >> >>>> the
>> >>>> >> >>>> /group route of the v2 of the API. But testing the
>> >>>> >> >>>> api/2/rest/group/
>> >>>> >> >>>> with
>> >>>> >> >>>> Postman, I can see it only returns a list of ids from groups,
>> >>>> >> >>>> not
>> >>>> >> >>>> organizations.
>> >>>> >> >>>>
>> >>>> >> >>>> I believe that's the reason why the harvester can't find the
>> >>>> >> >>>> owner_org.
>> >>>> >> >>>> Is this a known bug? Should api/2/rest/group/ return
>> >>>> >> >>>> organization
>> >>>> >> >>>> ids
>> >>>> >> >>>> too
>> >>>> >> >>>> or is what I'm getting expected behavior?
>> >>>> >> >>>>
>> >>>> >> >>>> In case the latter is true, should we change CKANHarvester to
>> >>>> >> >>>> use
>> >>>> >> >>>> v3
>> >>>> >> >>>> of
>> >>>> >> >>>> the api and /organization_show ? Or is there a better
>> >>>> >> >>>> workaround?
>> >>>> >> >>>>
>> >>>> >> >>>> Thanks for your help in advance.
>> >>>> >> >>>> Cheers,
>> >>>> >> >>>>
>> >>>> >> >>>> Rodrigo Parra
>> >>>> >> >>>>
>> >>>> >> >>>> _______________________________________________
>> >>>> >> >>>> ckan-dev mailing list
>> >>>> >> >>>> ckan-dev at lists.okfn.org
>> >>>> >> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >> >>>>
>> >>>> >> >>>
>> >>>> >> >>>
>> >>>> >> >>>
>> >>>> >> >>> --
>> >>>> >> >>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>>> >> >>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>>> >> >>>
>> >>>> >> >>> _______________________________________________
>> >>>> >> >>> ckan-dev mailing list
>> >>>> >> >>> ckan-dev at lists.okfn.org
>> >>>> >> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >> >>>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> _______________________________________________
>> >>>> >> >> ckan-dev mailing list
>> >>>> >> >> ckan-dev at lists.okfn.org
>> >>>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> --
>> >>>> >> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>>> >> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>>> >> _______________________________________________
>> >>>> >> ckan-dev mailing list
>> >>>> >> ckan-dev at lists.okfn.org
>> >>>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > _______________________________________________
>> >>>> > ckan-dev mailing list
>> >>>> > ckan-dev at lists.okfn.org
>> >>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>>> _______________________________________________
>> >>>> ckan-dev mailing list
>> >>>> ckan-dev at lists.okfn.org
>> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> ckan-dev mailing list
>> >> ckan-dev at lists.okfn.org
>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>
>> >
>> >
>> >
>> > --
>> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>
>>
>>
>> --
>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>

-- 
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch