[ckan-dev] Problem with ckanext-harvester and remote_orgs:create

Stefan Oderbolz stefan.oderbolz at liip.ch
Thu Jan 22 17:12:50 UTC 2015 

Nevermind, I just learned, that it should work with either slug-name
or the GUID. So false alarm from my side, my previous (already merged)
fix is sufficient to solve your problem :)

I can harvest http://datos.gov.py/ with previous fix as well, so it
should work for you as well.
Please check that you killed the gather_consumer and fetch_consumer
after the update and restarted them again.

- Stefan

On Thu, Jan 22, 2015 at 5:46 PM, Stefan Oderbolz
<stefan.oderbolz at liip.ch> wrote:
> Thanks for the URLs, I could replicate the error. The CKAN instance I
> used to test with, didn't have the organizations GUID in the
> "owner_org" field, but rather it's slug name.
> Therefore I didn't catch that this could be the case on other instances.
>
> I just created another PR to fix this issue:
> https://github.com/ckan/ckanext-harvest/pull/123
>
> With this fix I could successfully harvest http://datos.gov.py/
> including all its organizations and groups.
>
> - Stefan
>
>
> On Thu, Jan 22, 2015 at 4:42 PM, Rodrigo Valdez <rodri.valdez at gmail.com> wrote:
>> Sorry, I think you are asking for the URLs
>>
>> The local instance is http://datos.org.py/datos/  and the remote is
>> http://datos.gov.py
>>
>> Thanks for the help
>>
>>
>> Rodrigo Valdez
>> 0981-530572
>>
>>
>> 2015-01-22 12:21 GMT-03:00 Rodrigo Valdez <rodri.valdez at gmail.com>:
>>
>>> Hi Stefan,
>>>
>>> The local and the remote are 2.2 CKAN instances.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Rodrigo Valdez
>>> 0981-530572
>>>
>>>
>>> 2015-01-22 12:07 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>>>
>>>> Hi Rodrigo,
>>>>
>>>> can you tell me which instance you're trying to harvest?
>>>>
>>>> - Stefan
>>>>
>>>> On Wed, Jan 21, 2015 at 8:31 PM, Rodrigo Valdez <rodri.valdez at gmail.com>
>>>> wrote:
>>>> > Hi Stefan,
>>>> >
>>>> > Thank you for helping us, I was trying again to create the orgs and I
>>>> > found
>>>> > a new problem.
>>>> >
>>>> > INFO  [ckanext.harvest.harvesters.ckanharvester] Organization
>>>> > a4016a94-ea46-4671-907e-b56b51d86efa is not available
>>>> > ERROR [ckanext.harvest.harvesters.ckanharvester] Could not get remote
>>>> > org
>>>> > a4016a94-ea46-4671-907e-b56b51d86efa
>>>> >
>>>> > Its seem to be an error here
>>>> >
>>>> > Regards
>>>> > Rodrigo Valdez
>>>> >
>>>> >
>>>> >
>>>> > Rodrigo Valdez
>>>> > 0981-530572
>>>> >
>>>> >
>>>> > 2015-01-15 7:56 GMT-03:00 Stefan Oderbolz <stefan.oderbolz at liip.ch>:
>>>> >
>>>> >> Hi again,
>>>> >>
>>>> >> my PR just got merged, so if you update ckanext-harvest it should work
>>>> >> :)
>>>> >>
>>>> >> - Stefan
>>>> >>
>>>> >> On Tue, Jan 13, 2015 at 2:53 PM, Stefan Oderbolz
>>>> >> <stefan.oderbolz at liip.ch> wrote:
>>>> >> > Hi Rodrigo(s),
>>>> >> >
>>>> >> > sorry for the confusion, didn't pay close enough attention to spot
>>>> >> > your last names :)
>>>> >> > In the meantime I had a look at the issue and created a pull request
>>>> >> > with a possible solution:
>>>> >> > https://github.com/ckan/ckanext-harvest/pull/121
>>>> >> >
>>>> >> > - Stefan
>>>> >> >
>>>> >> > On Mon, Jan 5, 2015 at 2:23 PM, Rodrigo Parra <rodpar07 at gmail.com>
>>>> >> > wrote:
>>>> >> >> Thanks Stefan,
>>>> >> >>
>>>> >> >> Actually, Rodrigo Valdez is a different developer, we are both
>>>> >> >> paraguayans
>>>> >> >> though :)
>>>> >> >> He is working with the "client" CKAN instance, while I am
>>>> >> >> maintaining
>>>> >> >> the
>>>> >> >> "source" CKAN instance.
>>>> >> >> I didn't mean to spam the list, I just did not see his previous
>>>> >> >> mail.
>>>> >> >>
>>>> >> >> I would really appreciate your help, if you give me a hint on where
>>>> >> >> to
>>>> >> >> look
>>>> >> >> at, I might be able to lend a hand.
>>>> >> >> Otherwise, we'll probably wait for your fix and cherry-pick it.
>>>> >> >>
>>>> >> >> Cheers,
>>>> >> >> Rodrigo (Parra)
>>>> >> >>
>>>> >> >> 2015-01-05 8:46 GMT-03:00 Stefan Oderbolz
>>>> >> >> <stefan.oderbolz at liip.ch>:
>>>> >> >>
>>>> >> >>> Ah, just found your "newer" email.
>>>> >> >>>
>>>> >> >>> It seems that this route used to return organizations as well, but
>>>> >> >>> doesn't
>>>> >> >>> return them anymore now. This seems to be a regression. If I find
>>>> >> >>> some
>>>> >> >>> time
>>>> >> >>> this week, I'll look into that.
>>>> >> >>>
>>>> >> >>> Regards Stefan
>>>> >> >>>
>>>> >> >>> On Mon, Dec 22, 2014 at 7:45 PM, Rodrigo Parra
>>>> >> >>> <rodpar07 at gmail.com>
>>>> >> >>> wrote:
>>>> >> >>>>
>>>> >> >>>> Hello all,
>>>> >> >>>>
>>>> >> >>>> I'm trying to import datasets from a remote CKAN 2.2 instance,
>>>> >> >>>> and I
>>>> >> >>>> would like to import owner organizations as well. According to
>>>> >> >>>> the
>>>> >> >>>> harvester
>>>> >> >>>> docs, it should be possible (and fairly easy) to do so using the
>>>> >> >>>> remote_orgs:"create" configuration. However, after doing some
>>>> >> >>>> tests,
>>>> >> >>>> no
>>>> >> >>>> organization is being imported.
>>>> >> >>>>
>>>> >> >>>> Looking at the code, the remote organizations are being fetched
>>>> >> >>>> using
>>>> >> >>>> the
>>>> >> >>>> /group route of the v2 of the API. But testing the
>>>> >> >>>> api/2/rest/group/
>>>> >> >>>> with
>>>> >> >>>> Postman, I can see it only returns a list of ids from groups, not
>>>> >> >>>> organizations.
>>>> >> >>>>
>>>> >> >>>> I believe that's the reason why the harvester can't find the
>>>> >> >>>> owner_org.
>>>> >> >>>> Is this a known bug? Should api/2/rest/group/ return organization
>>>> >> >>>> ids
>>>> >> >>>> too
>>>> >> >>>> or is what I'm getting expected behavior?
>>>> >> >>>>
>>>> >> >>>> In case the latter is true, should we change CKANHarvester to use
>>>> >> >>>> v3
>>>> >> >>>> of
>>>> >> >>>> the api and /organization_show ? Or is there a better workaround?
>>>> >> >>>>
>>>> >> >>>> Thanks for your help in advance.
>>>> >> >>>> Cheers,
>>>> >> >>>>
>>>> >> >>>> Rodrigo Parra
>>>> >> >>>>
>>>> >> >>>> _______________________________________________
>>>> >> >>>> ckan-dev mailing list
>>>> >> >>>> ckan-dev at lists.okfn.org
>>>> >> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> >> >>>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> --
>>>> >> >>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>> >> >>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>> >> >>>
>>>> >> >>> _______________________________________________
>>>> >> >>> ckan-dev mailing list
>>>> >> >>> ckan-dev at lists.okfn.org
>>>> >> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> >> >>>
>>>> >> >>
>>>> >> >>
>>>> >> >> _______________________________________________
>>>> >> >> ckan-dev mailing list
>>>> >> >> ckan-dev at lists.okfn.org
>>>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>> >> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>> >> _______________________________________________
>>>> >> ckan-dev mailing list
>>>> >> ckan-dev at lists.okfn.org
>>>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > ckan-dev mailing list
>>>> > ckan-dev at lists.okfn.org
>>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch



-- 
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch

More information about the ckan-dev mailing list