[ckan-dev] pycsw

Tom Kralidis tomkralidis at gmail.com
Tue Oct 15 15:01:00 UTC 2013


Thanks. Ah.

It looks like the version of OWSLib that ckanext-spatial pegs has that warning implemented as a raise.  This is fixed by doing a warnings.warn instead in 0.8.2 [1].

Once ckanext-spatial accpets my PR [2] (which includes using getrecords2 support and pegging to OWSLib 0.8.2), this should be fixed.  At which point one can update ckanext-spatial and we can continue testing.

[1] https://github.com/geopython/OWSLib/issues/74
[2] https://github.com/okfn/ckanext-spatial/pull/42


On Tue, 15 Oct 2013, Hildegard Gerlach wrote:

> Date: Tue, 15 Oct 2013 14:32:54 +0200
> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] pycsw
> 
> That is what I did, but I really get a Job Error.
> Error gathering the identifiers from the CSW server [Please use the updated 
> 'getrecords2' method instead of 'getrecords'. The 'getrecords' method will be 
> upgraded to use the 'getrecords2' parameters in a future version of OWSLib.]
> Nothing is updated or deleted or added.
>
> Here they seem to have commented out the lines causing the error.
> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide#Harvest_remote_metadata_into_CKAN_39s_catalogue
>
> I am also wondering why I don't get anything in my window where  I run the 
> command
> paster --plugin=ckanext-harvest harvester gather_consumer 
> --config=development.ini
>
> I think in the past I got information what was going on (like on the Webpage 
> just mentioned before), now nothing.
> Maybe something is corrupt in my installation ?
>
> Hilde
>
>
> On 10/15/2013 1:41 PM, Tom Kralidis wrote:
>> 
>> FYI the getrecords2 is a warning and should not affect functionality 
>> whatsoever. Try a harvesting job that worked before?
>> 
>> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>> 
>>> Date: Tue, 15 Oct 2013 11:51:54 +0200
>>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>> Subject: Re: [ckan-dev] pycsw
>>> 
>>> Hi again,
>>> 
>>> I am not sure how to proceed now. I will put one problem at a time, 
>>> otherwise it gets too confusing.
>>> 
>>> Harvesting from a csw server had worked before, but in the meantime I have 
>>> upgraded CKAN (2.1 now and the extensions to stable) and installed pycsw, 
>>> so I don't know where from I get the problem (getrecords2).
>>> 
>>> When I look at the pip-requirements file in ckanext-spatial, there is
>>> GeoAlchemy>=0.6
>>> Shapely>=1.2.13
>>> # Temporal requirement until there is an OWSLib release
>>> # that requires python-dateutil<2.0
>>> -e 
>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
>>> lxml>=2.3
>>> argparse
>>> pyparsing==1.5.6
>>> 
>>> But my version of owslib is different
>>> pip freeze | grep -i owslib
>>> -e 
>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev
>>> 
>>> which has been probably installed with pycsw  following
>>> http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw 
>>> 
>>> # From now on the virtualenv should be activated
>>> 
>>> git clone https://github.com/geopython/pycsw.git
>>> cd pycsw
>>> # Remember to use at least pycsw 1.6.1
>>> git checkout 1.6.1
>>> pip install -e .
>>> python setup.py build
>>> python setup.py install
>>> 
>>> 
>>> So, in order to be able to harvest from csw servers, should I manually 
>>> susbtitute the file
>>> 
>>> ckanext/spatial/lib/csw_client.py
>>> with this one ?
>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>> 
>>> Thanks
>>> 
>>> Hilde
>>> 
>>> On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>>>> 
>>>> 
>>>> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>>>> 
>>>>> Hi Tom,
>>>>> 
>>>>> Many thanks for this. I don't think there are automated tests for this
>>>>> part of the code, just send the PR and I'll test it to make sure it
>>>>> works.
>>>>> 
>>>> 
>>>> Done: https://github.com/okfn/ckanext-spatial/pull/42
>>>> 
>>>>> BTW, I'm I right in thinking that this change is present in owslib 
>>>>> 0.8.2?
>>>>> 
>>>>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664 
>>>>> Just so we can link it from our requirements file instead of the
>>>>> particular commit:
>>>>> 
>>>> 
>>>> Correct, I've updated in my PR.
>>>> 
>>>>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5 
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Adrià
>>>>> 
>>>>> 
>>>>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>>>>>> Hi: I will try to setup a CKAN development environment, but in the
>>>>>> meantime, comments interleaved:
>>>>>> 
>>>>>> 
>>>>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader 
>>>>>> <adria.mercader at okfn.org> wrote:
>>>>>>> Hi Hilde,
>>>>>>> 
>>>>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>>>>> relatively recent addition and may still have some rough edges.
>>>>>>> 
>>>>>>> Let's take it one step at a time and see if we can make progress, see
>>>>>>> below for comments.
>>>>>>> 
>>>>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>> Dear Angelos and Adrià,
>>>>>>>> I am using Redhat 6, ckan version 2.2a
>>>>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>>>> 
>>>>>>> I strongly recommend always using stable versions unless you are
>>>>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>>>>> branches of ckanext-spatial and ckanext-harvest
>>>>>>> The rest of the versions seem fine
>>>>>>> 
>>>>>>>> I am using a Postgis database for CKAN and have done harvesting from 
>>>>>>>> 2
>>>>>>>> remote sources.
>>>>>>> Great.
>>>>>>> 
>>>>>>>> However I don't manage to load these sources into the newly
>>>>>>>> created database for pycsw.
>>>>>>>> In the pycsw.cfg file I have only changed the home and database 
>>>>>>>> settings.
>>>>>>>> The database table has been created successfully with this command
>>>>>>>> 
>>>>>> 
>>>>>> As reference, can we see/you post what a few of the metadata records
>>>>>> look like from these sources?
>>>>>> 
>>>>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>>>>> That sounds fine
>>>>>>> 
>>>>>>> 
>>>>>>>> In the meantime I went back to version 2.1, but I am not sure if this 
>>>>>>>> made
>>>>>>>> any change.
>>>>>>>> I did
>>>>>>>> 
>>>>>>>> git checkout release-v2.1
>>>>>>>> git pull
>>>>>>>> 
>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>> 
>>>>>>>> for CKAN, the harvester and spatial extension. Is this the correct 
>>>>>>>> way ?
>>>>>>> I wouldn't use the --upgrade flag unless you find some version number
>>>>>>> conflicts. This will upgrade all dependencies which might have
>>>>>>> unexpected results
>>>>>>> 
>>>>>>> 
>>>>>>>> When harvesting before I had not specified anything in the 
>>>>>>>> development.ini
>>>>>>>> file for
>>>>>>>> Now I have set it to
>>>>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>>>>> That is good, this is probably the best profile for generic iso
>>>>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>>>> 
>>>>>>>> and tried to reharvest, but then I get another error:
>>>>>>>> Error gathering the identifiers from the CSW server [Please use the 
>>>>>>>> updated
>>>>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' method 
>>>>>>>> will
>>>>>>>> be upgraded to use the 'getrecords2' parameters in a future version 
>>>>>>>> of
>>>>>>>> OWSLib
>>>>>>> That might be an issue with an incompatible version of OWSLib that got
>>>>>>> installed when you upgraded the requirements, we've had some issues
>>>>>>> with incompatibilities with owslib.
>>>>>>> 
>>>>>>> What version of OWSLib do you have installed? Run the following with
>>>>>>> your virtualenv activated and post the result:
>>>>>>> 
>>>>>> 
>>>>>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>>>>>> have been some huge improvements to CSW in OWSLib (in
>>>>>> owslib.csw.getrecords2, basically improved filter support), which will
>>>>>> replace/become owslib.csw.getrecords).  Having said this, this is just
>>>>>> a warning for downstream developers to update their use of OWSLib.
>>>>>> 
>>>>>> 
>>>>>>> pip freeze | grep -i owslib
>>>>>>> 
>>>>>>>> A workaround, but not solution, is here:
>>>>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>>>>> The workaround (not a fix) for the above error is to comment out the 
>>>>>>>> lines
>>>>>>>> (186-188) that raised the DeprecationWarning exception in the 
>>>>>>>> following
>>>>>>>> Python source file (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>>>>> #raise DeprecationWarning("""Please use the updated 'getrecords2' 
>>>>>>>> method
>>>>>>>> instead of 'getrecords'. #The 'getrecords' method will be upgraded to 
>>>>>>>> use
>>>>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>>>>> have never found this, so there definitely seems to be a conflict with
>>>>>>> a newer version of OWSlib.
>>>>>> 
>>>>>> ckanext-spatial devs:
>>>>>> 
>>>>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>>>>> doing `grep -R getrecords *`, the only thing you have to change is
>>>>>> your handling of typenames.
>>>>>> 
>>>>>> In owslib.csw.getrecords, qtype is a helper to define what kind of CSW
>>>>>> metadata to query for (default is 'dataset', others are 'service',
>>>>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>>>> 
>>>>>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>>>>>> you pass this as an OWSLib constraint object (see
>>>>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) in 
>>>>>> a list.
>>>>>> 
>>>>>> I've updated my fork of ckanext-spatial per above:
>>>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>>>> (this is UNTESTED).  Should I submit this as a PR?  Do ckanext-spatial
>>>>>> PRs run tests? Then I can submit PR.
>>>>>> 
>>>>>> 
>>>>>>> We'll investigate, but we'll need to OWSLib version you are using that
>>>>>>> you can find with the command I mentioned before.
>>>>>>> 
>>>>>>> 
>>>>>>>> Anyway, if  I look at the harvest object like you described I don't 
>>>>>>>> know
>>>>>>>> what is wrong with it. It is too big to post on the list.
>>>>>>> Can you post it online somewhere and link to it? I'm curious to see
>>>>>>> what might be causing the ckan-pycsw command to fail.
>>>>>>> 
>>>>>>> 
>>>>>>>> I have also another problem, I cannot see the "Add harvest source" 
>>>>>>>> button on
>>>>>>>> the
>>>>>>>> 
>>>>>>>> http://{my-ckan-instance}/harvest
>>>>>>>> 
>>>>>>>> Website. It has disapperared.
>>>>>>> Make sure that you are using the latest "stable" branch on
>>>>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>>>> 
>>>>>>> git checkout stable
>>>>>>> git pull
>>>>>>> 
>>>>>>> (and of course that you are logged in)
>>>>>>> 
>>>>>>> 
>>>>>>>> Sorry, but I am getting so many issues at the same time.
>>>>>>> Sorry about the troubles, hopefully we'll get there.
>>>>>>> 
>>>>>>> Adrià
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> Hilde
>>>>>>>> 
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> 
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>




More information about the ckan-dev mailing list