[ckan-dev] Problem with ckanext-harvester and remote_orgs:create

Thu Jan 22 16:46:34 UTC 2015

Thanks for the URLs, I could replicate the error. The CKAN instance I
used to test with, didn't have the organizations GUID in the
"owner_org" field, but rather it's slug name.
Therefore I didn't catch that this could be the case on other instances.

I just created another PR to fix this issue:
https://github.com/ckan/ckanext-harvest/pull/123

With this fix I could successfully harvest http://datos.gov.py/
including all its organizations and groups.

- Stefan

On Thu, Jan 22, 2015 at 4:42 PM, Rodrigo Valdez <rodri.valdez at gmail.com> wrote:
> Sorry, I think you are asking for the URLs
>
> The local instance is http://datos.org.py/datos/  and the remote is
> http://datos.gov.py
>
> Thanks for the help
>
>
> Rodrigo Valdez
> 0981-530572
>
>
> 2015-01-22 12:21 GMT-03:00 Rodrigo Valdez <rodri.valdez at gmail.com>:
>
>> Hi Stefan,
>>
>> The local and the remote are 2.2 CKAN instances.
>>
>>
>>
>>
>>
>>
>> Rodrigo Valdez
>> 0981-530572
>>
>>
>> 2015-01-22 12:07 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>>
>>> Hi Rodrigo,
>>>
>>> can you tell me which instance you're trying to harvest?
>>>
>>> - Stefan
>>>
>>> On Wed, Jan 21, 2015 at 8:31 PM, Rodrigo Valdez <rodri.valdez at gmail.com>
>>> wrote:
>>> > Hi Stefan,
>>> >
>>> > Thank you for helping us, I was trying again to create the orgs and I
>>> > found
>>> > a new problem.
>>> >
>>> > INFO  [ckanext.harvest.harvesters.ckanharvester] Organization
>>> > a4016a94-ea46-4671-907e-b56b51d86efa is not available
>>> > ERROR [ckanext.harvest.harvesters.ckanharvester] Could not get remote
>>> > org
>>> > a4016a94-ea46-4671-907e-b56b51d86efa
>>> >
>>> > Its seem to be an error here
>>> >
>>> > Regards
>>> > Rodrigo Valdez
>>> >
>>> >
>>> >
>>> > Rodrigo Valdez
>>> > 0981-530572
>>> >
>>> >
>>> > 2015-01-15 7:56 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>>> >
>>> >> Hi again,
>>> >>
>>> >> my PR just got merged, so if you update ckanext-harvest it should work
>>> >> :)
>>> >>
>>> >> - Stefan
>>> >>
>>> >> On Tue, Jan 13, 2015 at 2:53 PM, Stefan Oderbolz
>>> >> <stefan.oderbolz at liip.ch> wrote:
>>> >> > Hi Rodrigo(s),
>>> >> >
>>> >> > sorry for the confusion, didn't pay close enough attention to spot
>>> >> > your last names :)
>>> >> > In the meantime I had a look at the issue and created a pull request
>>> >> > with a possible solution:
>>> >> > https://github.com/ckan/ckanext-harvest/pull/121
>>> >> >
>>> >> > - Stefan
>>> >> >
>>> >> > On Mon, Jan 5, 2015 at 2:23 PM, Rodrigo Parra <rodpar07 at gmail.com>
>>> >> > wrote:
>>> >> >> Thanks Stefan,
>>> >> >>
>>> >> >> Actually, Rodrigo Valdez is a different developer, we are both
>>> >> >> paraguayans
>>> >> >> though :)
>>> >> >> He is working with the "client" CKAN instance, while I am
>>> >> >> maintaining
>>> >> >> the
>>> >> >> "source" CKAN instance.
>>> >> >> I didn't mean to spam the list, I just did not see his previous
>>> >> >> mail.
>>> >> >>
>>> >> >> I would really appreciate your help, if you give me a hint on where
>>> >> >> to
>>> >> >> look
>>> >> >> at, I might be able to lend a hand.
>>> >> >> Otherwise, we'll probably wait for your fix and cherry-pick it.
>>> >> >>
>>> >> >> Cheers,
>>> >> >> Rodrigo (Parra)
>>> >> >>
>>> >> >> 2015-01-05 8:46 GMT-03:00 Stefan Oderbolz
>>> >> >> <stefan.oderbolz at liip.ch>:
>>> >> >>
>>> >> >>> Ah, just found your "newer" email.
>>> >> >>>
>>> >> >>> It seems that this route used to return organizations as well, but
>>> >> >>> doesn't
>>> >> >>> return them anymore now. This seems to be a regression. If I find
>>> >> >>> some
>>> >> >>> time
>>> >> >>> this week, I'll look into that.
>>> >> >>>
>>> >> >>> Regards Stefan
>>> >> >>>
>>> >> >>> On Mon, Dec 22, 2014 at 7:45 PM, Rodrigo Parra
>>> >> >>> <rodpar07 at gmail.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> Hello all,
>>> >> >>>>
>>> >> >>>> I'm trying to import datasets from a remote CKAN 2.2 instance,
>>> >> >>>> and I
>>> >> >>>> would like to import owner organizations as well. According to
>>> >> >>>> the
>>> >> >>>> harvester
>>> >> >>>> docs, it should be possible (and fairly easy) to do so using the
>>> >> >>>> remote_orgs:"create" configuration. However, after doing some
>>> >> >>>> tests,
>>> >> >>>> no
>>> >> >>>> organization is being imported.
>>> >> >>>>
>>> >> >>>> Looking at the code, the remote organizations are being fetched
>>> >> >>>> using
>>> >> >>>> the
>>> >> >>>> /group route of the v2 of the API. But testing the
>>> >> >>>> api/2/rest/group/
>>> >> >>>> with
>>> >> >>>> Postman, I can see it only returns a list of ids from groups, not
>>> >> >>>> organizations.
>>> >> >>>>
>>> >> >>>> I believe that's the reason why the harvester can't find the
>>> >> >>>> owner_org.
>>> >> >>>> Is this a known bug? Should api/2/rest/group/ return organization
>>> >> >>>> ids
>>> >> >>>> too
>>> >> >>>> or is what I'm getting expected behavior?
>>> >> >>>>
>>> >> >>>> In case the latter is true, should we change CKANHarvester to use
>>> >> >>>> v3
>>> >> >>>> of
>>> >> >>>> the api and /organization_show ? Or is there a better workaround?
>>> >> >>>>
>>> >> >>>> Thanks for your help in advance.
>>> >> >>>> Cheers,
>>> >> >>>>
>>> >> >>>> Rodrigo Parra
>>> >> >>>>
>>> >> >>>> _______________________________________________
>>> >> >>>> ckan-dev mailing list
>>> >> >>>> ckan-dev at lists.okfn.org
>>> >> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >>>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> >> >>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> >> >>>
>>> >> >>> _______________________________________________
>>> >> >>> ckan-dev mailing list
>>> >> >>> ckan-dev at lists.okfn.org
>>> >> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> ckan-dev mailing list
>>> >> >> ckan-dev at lists.okfn.org
>>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> >> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> >> _______________________________________________
>>> >> ckan-dev mailing list
>>> >> ckan-dev at lists.okfn.org
>>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>>
>>>
>>>
>>> --
>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>

-- 
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch