[ckan-dev] pycsw

Wed Oct 16 13:03:43 UTC 2013

On 15 October 2013 16:01, Tom Kralidis <tomkralidis at gmail.com> wrote:
> [2] https://github.com/okfn/ckanext-spatial/pull/42
This has now been merged.

Hopefully now that we are using owslib 0.8.2 we won't have the
problems coming from pointing to a random commit.

Adrià

>
>
> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>
>> Date: Tue, 15 Oct 2013 14:32:54 +0200
>>
>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Subject: Re: [ckan-dev] pycsw
>>
>> That is what I did, but I really get a Job Error.
>> Error gathering the identifiers from the CSW server [Please use the
>> updated 'getrecords2' method instead of 'getrecords'. The 'getrecords'
>> method will be upgraded to use the 'getrecords2' parameters in a future
>> version of OWSLib.]
>> Nothing is updated or deleted or added.
>>
>> Here they seem to have commented out the lines causing the error.
>>
>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide#Harvest_remote_metadata_into_CKAN_39s_catalogue
>>
>> I am also wondering why I don't get anything in my window where  I run the
>> command
>> paster --plugin=ckanext-harvest harvester gather_consumer
>> --config=development.ini
>>
>> I think in the past I got information what was going on (like on the
>> Webpage just mentioned before), now nothing.
>> Maybe something is corrupt in my installation ?
>>
>> Hilde
>>
>>
>> On 10/15/2013 1:41 PM, Tom Kralidis wrote:
>>>
>>>
>>> FYI the getrecords2 is a warning and should not affect functionality
>>> whatsoever. Try a harvesting job that worked before?
>>>
>>> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>>>
>>>> Date: Tue, 15 Oct 2013 11:51:54 +0200
>>>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>>>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>> Subject: Re: [ckan-dev] pycsw
>>>>
>>>> Hi again,
>>>>
>>>> I am not sure how to proceed now. I will put one problem at a time,
>>>> otherwise it gets too confusing.
>>>>
>>>> Harvesting from a csw server had worked before, but in the meantime I
>>>> have upgraded CKAN (2.1 now and the extensions to stable) and installed
>>>> pycsw, so I don't know where from I get the problem (getrecords2).
>>>>
>>>> When I look at the pip-requirements file in ckanext-spatial, there is
>>>> GeoAlchemy>=0.6
>>>> Shapely>=1.2.13
>>>> # Temporal requirement until there is an OWSLib release
>>>> # that requires python-dateutil<2.0
>>>> -e
>>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
>>>> lxml>=2.3
>>>> argparse
>>>> pyparsing==1.5.6
>>>>
>>>> But my version of owslib is different
>>>> pip freeze | grep -i owslib
>>>> -e
>>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev
>>>>
>>>> which has been probably installed with pycsw  following
>>>>
>>>> http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw
>>>> # From now on the virtualenv should be activated
>>>>
>>>> git clone https://github.com/geopython/pycsw.git
>>>> cd pycsw
>>>> # Remember to use at least pycsw 1.6.1
>>>> git checkout 1.6.1
>>>> pip install -e .
>>>> python setup.py build
>>>> python setup.py install
>>>>
>>>>
>>>> So, in order to be able to harvest from csw servers, should I manually
>>>> susbtitute the file
>>>>
>>>> ckanext/spatial/lib/csw_client.py
>>>> with this one ?
>>>>
>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e
>>>> Thanks
>>>>
>>>> Hilde
>>>>
>>>> On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>>>>>
>>>>>> Hi Tom,
>>>>>>
>>>>>> Many thanks for this. I don't think there are automated tests for this
>>>>>> part of the code, just send the PR and I'll test it to make sure it
>>>>>> works.
>>>>>>
>>>>>
>>>>> Done: https://github.com/okfn/ckanext-spatial/pull/42
>>>>>
>>>>>> BTW, I'm I right in thinking that this change is present in owslib
>>>>>> 0.8.2?
>>>>>>
>>>>>>
>>>>>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664
>>>>>> Just so we can link it from our requirements file instead of the
>>>>>> particular commit:
>>>>>>
>>>>>
>>>>> Correct, I've updated in my PR.
>>>>>
>>>>>>
>>>>>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5
>>>>>> Cheers,
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi: I will try to setup a CKAN development environment, but in the
>>>>>>> meantime, comments interleaved:
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader
>>>>>>> <adria.mercader at okfn.org> wrote:
>>>>>>>>
>>>>>>>> Hi Hilde,
>>>>>>>>
>>>>>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>>>>>> relatively recent addition and may still have some rough edges.
>>>>>>>>
>>>>>>>> Let's take it one step at a time and see if we can make progress,
>>>>>>>> see
>>>>>>>> below for comments.
>>>>>>>>
>>>>>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>>
>>>>>>>>> Dear Angelos and Adrià,
>>>>>>>>> I am using Redhat 6, ckan version 2.2a
>>>>>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>>>>>
>>>>>>>>
>>>>>>>> I strongly recommend always using stable versions unless you are
>>>>>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>>>>>> branches of ckanext-spatial and ckanext-harvest
>>>>>>>> The rest of the versions seem fine
>>>>>>>>
>>>>>>>>> I am using a Postgis database for CKAN and have done harvesting
>>>>>>>>> from 2
>>>>>>>>> remote sources.
>>>>>>>>
>>>>>>>> Great.
>>>>>>>>
>>>>>>>>> However I don't manage to load these sources into the newly
>>>>>>>>> created database for pycsw.
>>>>>>>>> In the pycsw.cfg file I have only changed the home and database
>>>>>>>>> settings.
>>>>>>>>> The database table has been created successfully with this command
>>>>>>>>>
>>>>>>>
>>>>>>> As reference, can we see/you post what a few of the metadata records
>>>>>>> look like from these sources?
>>>>>>>
>>>>>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>>>>>>
>>>>>>>> That sounds fine
>>>>>>>>
>>>>>>>>
>>>>>>>>> In the meantime I went back to version 2.1, but I am not sure if
>>>>>>>>> this made
>>>>>>>>> any change.
>>>>>>>>> I did
>>>>>>>>>
>>>>>>>>> git checkout release-v2.1
>>>>>>>>> git pull
>>>>>>>>>
>>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>>>
>>>>>>>>> for CKAN, the harvester and spatial extension. Is this the correct
>>>>>>>>> way ?
>>>>>>>>
>>>>>>>> I wouldn't use the --upgrade flag unless you find some version
>>>>>>>> number
>>>>>>>> conflicts. This will upgrade all dependencies which might have
>>>>>>>> unexpected results
>>>>>>>>
>>>>>>>>
>>>>>>>>> When harvesting before I had not specified anything in the
>>>>>>>>> development.ini
>>>>>>>>> file for
>>>>>>>>> Now I have set it to
>>>>>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>>>>>>
>>>>>>>> That is good, this is probably the best profile for generic iso
>>>>>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>>>>>
>>>>>>>>> and tried to reharvest, but then I get another error:
>>>>>>>>> Error gathering the identifiers from the CSW server [Please use the
>>>>>>>>> updated
>>>>>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords'
>>>>>>>>> method will
>>>>>>>>> be upgraded to use the 'getrecords2' parameters in a future version
>>>>>>>>> of
>>>>>>>>> OWSLib
>>>>>>>>
>>>>>>>> That might be an issue with an incompatible version of OWSLib that
>>>>>>>> got
>>>>>>>> installed when you upgraded the requirements, we've had some issues
>>>>>>>> with incompatibilities with owslib.
>>>>>>>>
>>>>>>>> What version of OWSLib do you have installed? Run the following with
>>>>>>>> your virtualenv activated and post the result:
>>>>>>>>
>>>>>>>
>>>>>>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>>>>>>> have been some huge improvements to CSW in OWSLib (in
>>>>>>> owslib.csw.getrecords2, basically improved filter support), which
>>>>>>> will
>>>>>>> replace/become owslib.csw.getrecords).  Having said this, this is
>>>>>>> just
>>>>>>> a warning for downstream developers to update their use of OWSLib.
>>>>>>>
>>>>>>>
>>>>>>>> pip freeze | grep -i owslib
>>>>>>>>
>>>>>>>>> A workaround, but not solution, is here:
>>>>>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>>>>>> The workaround (not a fix) for the above error is to comment out
>>>>>>>>> the lines
>>>>>>>>> (186-188) that raised the DeprecationWarning exception in the
>>>>>>>>> following
>>>>>>>>> Python source file
>>>>>>>>> (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>>>>>> #raise DeprecationWarning("""Please use the updated 'getrecords2'
>>>>>>>>> method
>>>>>>>>> instead of 'getrecords'. #The 'getrecords' method will be upgraded
>>>>>>>>> to use
>>>>>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>>>>>>
>>>>>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>>>>>> have never found this, so there definitely seems to be a conflict
>>>>>>>> with
>>>>>>>> a newer version of OWSlib.
>>>>>>>
>>>>>>>
>>>>>>> ckanext-spatial devs:
>>>>>>>
>>>>>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>>>>>> doing `grep -R getrecords *`, the only thing you have to change is
>>>>>>> your handling of typenames.
>>>>>>>
>>>>>>> In owslib.csw.getrecords, qtype is a helper to define what kind of
>>>>>>> CSW
>>>>>>> metadata to query for (default is 'dataset', others are 'service',
>>>>>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>>>>>
>>>>>>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>>>>>>> you pass this as an OWSLib constraint object (see
>>>>>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284)
>>>>>>> in a list.
>>>>>>>
>>>>>>> I've updated my fork of ckanext-spatial per above:
>>>>>>>
>>>>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e
>>>>>>> (this is UNTESTED).  Should I submit this as a PR?  Do ckanext-spatial
>>>>>>> PRs run tests? Then I can submit PR.
>>>>>>>
>>>>>>>
>>>>>>>> We'll investigate, but we'll need to OWSLib version you are using
>>>>>>>> that
>>>>>>>> you can find with the command I mentioned before.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Anyway, if  I look at the harvest object like you described I don't
>>>>>>>>> know
>>>>>>>>> what is wrong with it. It is too big to post on the list.
>>>>>>>>
>>>>>>>> Can you post it online somewhere and link to it? I'm curious to see
>>>>>>>> what might be causing the ckan-pycsw command to fail.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I have also another problem, I cannot see the "Add harvest source"
>>>>>>>>> button on
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> http://{my-ckan-instance}/harvest
>>>>>>>>>
>>>>>>>>> Website. It has disapperared.
>>>>>>>>
>>>>>>>> Make sure that you are using the latest "stable" branch on
>>>>>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>>>>>
>>>>>>>> git checkout stable
>>>>>>>> git pull
>>>>>>>>
>>>>>>>> (and of course that you are logged in)
>>>>>>>>
>>>>>>>>
>>>>>>>>> Sorry, but I am getting so many issues at the same time.
>>>>>>>>
>>>>>>>> Sorry about the troubles, hopefully we'll get there.
>>>>>>>>
>>>>>>>> Adrià
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hilde
>>>>>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev