[ckan-dev] pycsw

Hildegard Gerlach hildegard.gerlach at jrc.ec.europa.eu
Tue Oct 15 12:32:54 UTC 2013 

That is what I did, but I really get a Job Error.
Error gathering the identifiers from the CSW server [Please use the 
updated 'getrecords2' method instead of 'getrecords'. The 'getrecords' 
method will be upgraded to use the 'getrecords2' parameters in a future 
version of OWSLib.]
Nothing is updated or deleted or added.

Here they seem to have commented out the lines causing the error.
https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide#Harvest_remote_metadata_into_CKAN_39s_catalogue

I am also wondering why I don't get anything in my window where  I run 
the command
paster --plugin=ckanext-harvest harvester gather_consumer 
--config=development.ini

I think in the past I got information what was going on (like on the 
Webpage just mentioned before), now nothing.
Maybe something is corrupt in my installation ?

Hilde


On 10/15/2013 1:41 PM, Tom Kralidis wrote:
>
> FYI the getrecords2 is a warning and should not affect functionality 
> whatsoever. Try a harvesting job that worked before?
>
> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>
>> Date: Tue, 15 Oct 2013 11:51:54 +0200
>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Subject: Re: [ckan-dev] pycsw
>>
>> Hi again,
>>
>> I am not sure how to proceed now. I will put one problem at a time, 
>> otherwise it gets too confusing.
>>
>> Harvesting from a csw server had worked before, but in the meantime I 
>> have upgraded CKAN (2.1 now and the extensions to stable) and 
>> installed pycsw, so I don't know where from I get the problem 
>> (getrecords2).
>>
>> When I look at the pip-requirements file in ckanext-spatial, there is
>> GeoAlchemy>=0.6
>> Shapely>=1.2.13
>> # Temporal requirement until there is an OWSLib release
>> # that requires python-dateutil<2.0
>> -e 
>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
>> lxml>=2.3
>> argparse
>> pyparsing==1.5.6
>>
>> But my version of owslib is different
>> pip freeze | grep -i owslib
>> -e 
>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev
>>
>> which has been probably installed with pycsw  following
>> http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw 
>>
>>
>> # From now on the virtualenv should be activated
>>
>> git clone https://github.com/geopython/pycsw.git
>> cd pycsw
>> # Remember to use at least pycsw 1.6.1
>> git checkout 1.6.1
>> pip install -e .
>> python setup.py build
>> python setup.py install
>>
>>
>> So, in order to be able to harvest from csw servers, should I 
>> manually susbtitute the file
>>
>> ckanext/spatial/lib/csw_client.py
>> with this one ?
>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>
>>
>> Thanks
>>
>> Hilde
>>
>> On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>>>
>>>
>>> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>>>
>>>> Hi Tom,
>>>>
>>>> Many thanks for this. I don't think there are automated tests for this
>>>> part of the code, just send the PR and I'll test it to make sure it
>>>> works.
>>>>
>>>
>>> Done: https://github.com/okfn/ckanext-spatial/pull/42
>>>
>>>> BTW, I'm I right in thinking that this change is present in owslib 
>>>> 0.8.2?
>>>>
>>>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664 
>>>>
>>>> Just so we can link it from our requirements file instead of the
>>>> particular commit:
>>>>
>>>
>>> Correct, I've updated in my PR.
>>>
>>>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5 
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Adrià
>>>>
>>>>
>>>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>>>>> Hi: I will try to setup a CKAN development environment, but in the
>>>>> meantime, comments interleaved:
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader 
>>>>> <adria.mercader at okfn.org> wrote:
>>>>>> Hi Hilde,
>>>>>>
>>>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>>>> relatively recent addition and may still have some rough edges.
>>>>>>
>>>>>> Let's take it one step at a time and see if we can make progress, 
>>>>>> see
>>>>>> below for comments.
>>>>>>
>>>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>> Dear Angelos and Adrià,
>>>>>>> I am using Redhat 6, ckan version 2.2a
>>>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>>>
>>>>>> I strongly recommend always using stable versions unless you are
>>>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>>>> branches of ckanext-spatial and ckanext-harvest
>>>>>> The rest of the versions seem fine
>>>>>>
>>>>>>> I am using a Postgis database for CKAN and have done harvesting 
>>>>>>> from 2
>>>>>>> remote sources.
>>>>>> Great.
>>>>>>
>>>>>>> However I don't manage to load these sources into the newly
>>>>>>> created database for pycsw.
>>>>>>> In the pycsw.cfg file I have only changed the home and database 
>>>>>>> settings.
>>>>>>> The database table has been created successfully with this command
>>>>>>>
>>>>>
>>>>> As reference, can we see/you post what a few of the metadata records
>>>>> look like from these sources?
>>>>>
>>>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>>>> That sounds fine
>>>>>>
>>>>>>
>>>>>>> In the meantime I went back to version 2.1, but I am not sure if 
>>>>>>> this made
>>>>>>> any change.
>>>>>>> I did
>>>>>>>
>>>>>>> git checkout release-v2.1
>>>>>>> git pull
>>>>>>>
>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>
>>>>>>> for CKAN, the harvester and spatial extension. Is this the 
>>>>>>> correct way ?
>>>>>> I wouldn't use the --upgrade flag unless you find some version 
>>>>>> number
>>>>>> conflicts. This will upgrade all dependencies which might have
>>>>>> unexpected results
>>>>>>
>>>>>>
>>>>>>> When harvesting before I had not specified anything in the 
>>>>>>> development.ini
>>>>>>> file for
>>>>>>> Now I have set it to
>>>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>>>> That is good, this is probably the best profile for generic iso
>>>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>>>
>>>>>>> and tried to reharvest, but then I get another error:
>>>>>>> Error gathering the identifiers from the CSW server [Please use 
>>>>>>> the updated
>>>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' 
>>>>>>> method will
>>>>>>> be upgraded to use the 'getrecords2' parameters in a future 
>>>>>>> version of
>>>>>>> OWSLib
>>>>>> That might be an issue with an incompatible version of OWSLib 
>>>>>> that got
>>>>>> installed when you upgraded the requirements, we've had some issues
>>>>>> with incompatibilities with owslib.
>>>>>>
>>>>>> What version of OWSLib do you have installed? Run the following with
>>>>>> your virtualenv activated and post the result:
>>>>>>
>>>>>
>>>>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>>>>> have been some huge improvements to CSW in OWSLib (in
>>>>> owslib.csw.getrecords2, basically improved filter support), which 
>>>>> will
>>>>> replace/become owslib.csw.getrecords).  Having said this, this is 
>>>>> just
>>>>> a warning for downstream developers to update their use of OWSLib.
>>>>>
>>>>>
>>>>>> pip freeze | grep -i owslib
>>>>>>
>>>>>>> A workaround, but not solution, is here:
>>>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>>>> The workaround (not a fix) for the above error is to comment out 
>>>>>>> the lines
>>>>>>> (186-188) that raised the DeprecationWarning exception in the 
>>>>>>> following
>>>>>>> Python source file 
>>>>>>> (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>>>> #raise DeprecationWarning("""Please use the updated 
>>>>>>> 'getrecords2' method
>>>>>>> instead of 'getrecords'. #The 'getrecords' method will be 
>>>>>>> upgraded to use
>>>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>>>> have never found this, so there definitely seems to be a conflict 
>>>>>> with
>>>>>> a newer version of OWSlib.
>>>>>
>>>>> ckanext-spatial devs:
>>>>>
>>>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>>>> doing `grep -R getrecords *`, the only thing you have to change is
>>>>> your handling of typenames.
>>>>>
>>>>> In owslib.csw.getrecords, qtype is a helper to define what kind of 
>>>>> CSW
>>>>> metadata to query for (default is 'dataset', others are 'service',
>>>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>>>
>>>>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>>>>> you pass this as an OWSLib constraint object (see
>>>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) in 
>>>>>
>>>>> a list.
>>>>>
>>>>> I've updated my fork of ckanext-spatial per above:
>>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>>> (this is UNTESTED).  Should I submit this as a PR?  Do 
>>>>> ckanext-spatial
>>>>> PRs run tests? Then I can submit PR.
>>>>>
>>>>>
>>>>>> We'll investigate, but we'll need to OWSLib version you are using 
>>>>>> that
>>>>>> you can find with the command I mentioned before.
>>>>>>
>>>>>>
>>>>>>> Anyway, if  I look at the harvest object like you described I 
>>>>>>> don't know
>>>>>>> what is wrong with it. It is too big to post on the list.
>>>>>> Can you post it online somewhere and link to it? I'm curious to see
>>>>>> what might be causing the ckan-pycsw command to fail.
>>>>>>
>>>>>>
>>>>>>> I have also another problem, I cannot see the "Add harvest 
>>>>>>> source" button on
>>>>>>> the
>>>>>>>
>>>>>>> http://{my-ckan-instance}/harvest
>>>>>>>
>>>>>>> Website. It has disapperared.
>>>>>> Make sure that you are using the latest "stable" branch on
>>>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>>>
>>>>>> git checkout stable
>>>>>> git pull
>>>>>>
>>>>>> (and of course that you are logged in)
>>>>>>
>>>>>>
>>>>>>> Sorry, but I am getting so many issues at the same time.
>>>>>> Sorry about the troubles, hopefully we'll get there.
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hilde
>>>>>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

More information about the ckan-dev mailing list