[ckan-dev] pycsw

Tom Kralidis tomkralidis at gmail.com
Tue Oct 15 11:41:53 UTC 2013


FYI the getrecords2 is a warning and should not affect functionality whatsoever. Try a harvesting job that worked before?

On Tue, 15 Oct 2013, Hildegard Gerlach wrote:

> Date: Tue, 15 Oct 2013 11:51:54 +0200
> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] pycsw
> 
> Hi again,
>
> I am not sure how to proceed now. I will put one problem at a time, otherwise 
> it gets too confusing.
>
> Harvesting from a csw server had worked before, but in the meantime I have 
> upgraded CKAN (2.1 now and the extensions to stable) and installed pycsw, so 
> I don't know where from I get the problem (getrecords2).
>
> When I look at the pip-requirements file in ckanext-spatial, there is
> GeoAlchemy>=0.6
> Shapely>=1.2.13
> # Temporal requirement until there is an OWSLib release
> # that requires python-dateutil<2.0
> -e 
> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
> lxml>=2.3
> argparse
> pyparsing==1.5.6
>
> But my version of owslib is different
> pip freeze | grep -i owslib
> -e 
> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev
>
> which has been probably installed with pycsw  following
> http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw
>
> # From now on the virtualenv should be activated
>
> git clone https://github.com/geopython/pycsw.git
> cd pycsw
> # Remember to use at least pycsw 1.6.1
> git checkout 1.6.1
> pip install -e .
> python setup.py build
> python setup.py install
>
>
> So, in order to be able to harvest from csw servers, should I manually 
> susbtitute the file
>
> ckanext/spatial/lib/csw_client.py
> with this one ?
> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e
>
> Thanks
>
> Hilde
>
> On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>> 
>> 
>> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>> 
>>> Hi Tom,
>>> 
>>> Many thanks for this. I don't think there are automated tests for this
>>> part of the code, just send the PR and I'll test it to make sure it
>>> works.
>>> 
>> 
>> Done: https://github.com/okfn/ckanext-spatial/pull/42
>> 
>>> BTW, I'm I right in thinking that this change is present in owslib 0.8.2?
>>> 
>>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664 
>>> 
>>> Just so we can link it from our requirements file instead of the
>>> particular commit:
>>> 
>> 
>> Correct, I've updated in my PR.
>> 
>>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5 
>>> 
>>> 
>>> Cheers,
>>> 
>>> Adrià
>>> 
>>> 
>>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>>>> Hi: I will try to setup a CKAN development environment, but in the
>>>> meantime, comments interleaved:
>>>> 
>>>> 
>>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader <adria.mercader at okfn.org> 
>>>> wrote:
>>>>> Hi Hilde,
>>>>> 
>>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>>> relatively recent addition and may still have some rough edges.
>>>>> 
>>>>> Let's take it one step at a time and see if we can make progress, see
>>>>> below for comments.
>>>>> 
>>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>> Dear Angelos and Adrià,
>>>>>> I am using Redhat 6, ckan version 2.2a
>>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>> 
>>>>> I strongly recommend always using stable versions unless you are
>>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>>> branches of ckanext-spatial and ckanext-harvest
>>>>> The rest of the versions seem fine
>>>>> 
>>>>>> I am using a Postgis database for CKAN and have done harvesting from 2
>>>>>> remote sources.
>>>>> Great.
>>>>> 
>>>>>> However I don't manage to load these sources into the newly
>>>>>> created database for pycsw.
>>>>>> In the pycsw.cfg file I have only changed the home and database 
>>>>>> settings.
>>>>>> The database table has been created successfully with this command
>>>>>> 
>>>> 
>>>> As reference, can we see/you post what a few of the metadata records
>>>> look like from these sources?
>>>> 
>>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>>> That sounds fine
>>>>> 
>>>>> 
>>>>>> In the meantime I went back to version 2.1, but I am not sure if this 
>>>>>> made
>>>>>> any change.
>>>>>> I did
>>>>>> 
>>>>>> git checkout release-v2.1
>>>>>> git pull
>>>>>> 
>>>>>> pip install --upgrade -r requirements.txt
>>>>>> 
>>>>>> for CKAN, the harvester and spatial extension. Is this the correct way 
>>>>>> ?
>>>>> I wouldn't use the --upgrade flag unless you find some version number
>>>>> conflicts. This will upgrade all dependencies which might have
>>>>> unexpected results
>>>>> 
>>>>> 
>>>>>> When harvesting before I had not specified anything in the 
>>>>>> development.ini
>>>>>> file for
>>>>>> Now I have set it to
>>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>>> That is good, this is probably the best profile for generic iso
>>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>> 
>>>>>> and tried to reharvest, but then I get another error:
>>>>>> Error gathering the identifiers from the CSW server [Please use the 
>>>>>> updated
>>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' method 
>>>>>> will
>>>>>> be upgraded to use the 'getrecords2' parameters in a future version of
>>>>>> OWSLib
>>>>> That might be an issue with an incompatible version of OWSLib that got
>>>>> installed when you upgraded the requirements, we've had some issues
>>>>> with incompatibilities with owslib.
>>>>> 
>>>>> What version of OWSLib do you have installed? Run the following with
>>>>> your virtualenv activated and post the result:
>>>>> 
>>>> 
>>>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>>>> have been some huge improvements to CSW in OWSLib (in
>>>> owslib.csw.getrecords2, basically improved filter support), which will
>>>> replace/become owslib.csw.getrecords).  Having said this, this is just
>>>> a warning for downstream developers to update their use of OWSLib.
>>>> 
>>>> 
>>>>> pip freeze | grep -i owslib
>>>>> 
>>>>>> A workaround, but not solution, is here:
>>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>>> The workaround (not a fix) for the above error is to comment out the 
>>>>>> lines
>>>>>> (186-188) that raised the DeprecationWarning exception in the following
>>>>>> Python source file (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>>> #raise DeprecationWarning("""Please use the updated 'getrecords2' 
>>>>>> method
>>>>>> instead of 'getrecords'. #The 'getrecords' method will be upgraded to 
>>>>>> use
>>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>>> have never found this, so there definitely seems to be a conflict with
>>>>> a newer version of OWSlib.
>>>> 
>>>> ckanext-spatial devs:
>>>> 
>>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>>> doing `grep -R getrecords *`, the only thing you have to change is
>>>> your handling of typenames.
>>>> 
>>>> In owslib.csw.getrecords, qtype is a helper to define what kind of CSW
>>>> metadata to query for (default is 'dataset', others are 'service',
>>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>> 
>>>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>>>> you pass this as an OWSLib constraint object (see
>>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) in
>>>> a list.
>>>> 
>>>> I've updated my fork of ckanext-spatial per above:
>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>> (this is UNTESTED).  Should I submit this as a PR?  Do ckanext-spatial
>>>> PRs run tests? Then I can submit PR.
>>>> 
>>>> 
>>>>> We'll investigate, but we'll need to OWSLib version you are using that
>>>>> you can find with the command I mentioned before.
>>>>> 
>>>>> 
>>>>>> Anyway, if  I look at the harvest object like you described I don't 
>>>>>> know
>>>>>> what is wrong with it. It is too big to post on the list.
>>>>> Can you post it online somewhere and link to it? I'm curious to see
>>>>> what might be causing the ckan-pycsw command to fail.
>>>>> 
>>>>> 
>>>>>> I have also another problem, I cannot see the "Add harvest source" 
>>>>>> button on
>>>>>> the
>>>>>> 
>>>>>> http://{my-ckan-instance}/harvest
>>>>>> 
>>>>>> Website. It has disapperared.
>>>>> Make sure that you are using the latest "stable" branch on
>>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>> 
>>>>> git checkout stable
>>>>> git pull
>>>>> 
>>>>> (and of course that you are logged in)
>>>>> 
>>>>> 
>>>>>> Sorry, but I am getting so many issues at the same time.
>>>>> Sorry about the troubles, hopefully we'll get there.
>>>>> 
>>>>> Adrià
>>>>> 
>>>>> 
>>>>> 
>>>>>> Hilde
>>>>>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>




More information about the ckan-dev mailing list