[ckan-dev] pycsw

Tue Oct 15 09:51:54 UTC 2013

Hi again,

I am not sure how to proceed now. I will put one problem at a time, 
otherwise it gets too confusing.

Harvesting from a csw server had worked before, but in the meantime I 
have upgraded CKAN (2.1 now and the extensions to stable) and installed 
pycsw, so I don't know where from I get the problem (getrecords2).

When I look at the pip-requirements file in ckanext-spatial, there is
GeoAlchemy>=0.6
Shapely>=1.2.13
# Temporal requirement until there is an OWSLib release
# that requires python-dateutil<2.0
-e 
git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
lxml>=2.3
argparse
pyparsing==1.5.6

But my version of owslib is different
pip freeze | grep -i owslib
-e 
git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev

which has been probably installed with pycsw  following
http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw

# From now on the virtualenv should be activated

git clone https://github.com/geopython/pycsw.git
cd pycsw
# Remember to use at least pycsw 1.6.1
git checkout 1.6.1
pip install -e .
python setup.py build
python setup.py install

So, in order to be able to harvest from csw servers, should I manually 
susbtitute the file

ckanext/spatial/lib/csw_client.py
with this one ?
https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e

Thanks

Hilde

On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>
>
> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>
>> Hi Tom,
>>
>> Many thanks for this. I don't think there are automated tests for this
>> part of the code, just send the PR and I'll test it to make sure it
>> works.
>>
>
> Done: https://github.com/okfn/ckanext-spatial/pull/42
>
>> BTW, I'm I right in thinking that this change is present in owslib 
>> 0.8.2?
>>
>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664 
>>
>>
>> Just so we can link it from our requirements file instead of the
>> particular commit:
>>
>
> Correct, I've updated in my PR.
>
>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5 
>>
>>
>>
>> Cheers,
>>
>> Adrià
>>
>>
>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>>> Hi: I will try to setup a CKAN development environment, but in the
>>> meantime, comments interleaved:
>>>
>>>
>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader 
>>> <adria.mercader at okfn.org> wrote:
>>>> Hi Hilde,
>>>>
>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>> relatively recent addition and may still have some rough edges.
>>>>
>>>> Let's take it one step at a time and see if we can make progress, see
>>>> below for comments.
>>>>
>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>> Dear Angelos and Adrià,
>>>>> I am using Redhat 6, ckan version 2.2a
>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>
>>>> I strongly recommend always using stable versions unless you are
>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>> branches of ckanext-spatial and ckanext-harvest
>>>> The rest of the versions seem fine
>>>>
>>>>> I am using a Postgis database for CKAN and have done harvesting 
>>>>> from 2
>>>>> remote sources.
>>>> Great.
>>>>
>>>>> However I don't manage to load these sources into the newly
>>>>> created database for pycsw.
>>>>> In the pycsw.cfg file I have only changed the home and database 
>>>>> settings.
>>>>> The database table has been created successfully with this command
>>>>>
>>>
>>> As reference, can we see/you post what a few of the metadata records
>>> look like from these sources?
>>>
>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>> That sounds fine
>>>>
>>>>
>>>>> In the meantime I went back to version 2.1, but I am not sure if 
>>>>> this made
>>>>> any change.
>>>>> I did
>>>>>
>>>>> git checkout release-v2.1
>>>>> git pull
>>>>>
>>>>> pip install --upgrade -r requirements.txt
>>>>>
>>>>> for CKAN, the harvester and spatial extension. Is this the correct 
>>>>> way ?
>>>> I wouldn't use the --upgrade flag unless you find some version number
>>>> conflicts. This will upgrade all dependencies which might have
>>>> unexpected results
>>>>
>>>>
>>>>> When harvesting before I had not specified anything in the 
>>>>> development.ini
>>>>> file for
>>>>> Now I have set it to
>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>> That is good, this is probably the best profile for generic iso
>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>
>>>>> and tried to reharvest, but then I get another error:
>>>>> Error gathering the identifiers from the CSW server [Please use 
>>>>> the updated
>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' 
>>>>> method will
>>>>> be upgraded to use the 'getrecords2' parameters in a future 
>>>>> version of
>>>>> OWSLib
>>>> That might be an issue with an incompatible version of OWSLib that got
>>>> installed when you upgraded the requirements, we've had some issues
>>>> with incompatibilities with owslib.
>>>>
>>>> What version of OWSLib do you have installed? Run the following with
>>>> your virtualenv activated and post the result:
>>>>
>>>
>>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>>> have been some huge improvements to CSW in OWSLib (in
>>> owslib.csw.getrecords2, basically improved filter support), which will
>>> replace/become owslib.csw.getrecords).  Having said this, this is just
>>> a warning for downstream developers to update their use of OWSLib.
>>>
>>>
>>>> pip freeze | grep -i owslib
>>>>
>>>>> A workaround, but not solution, is here:
>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>> The workaround (not a fix) for the above error is to comment out 
>>>>> the lines
>>>>> (186-188) that raised the DeprecationWarning exception in the 
>>>>> following
>>>>> Python source file (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>> #raise DeprecationWarning("""Please use the updated 'getrecords2' 
>>>>> method
>>>>> instead of 'getrecords'. #The 'getrecords' method will be upgraded 
>>>>> to use
>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>> have never found this, so there definitely seems to be a conflict with
>>>> a newer version of OWSlib.
>>>
>>> ckanext-spatial devs:
>>>
>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>> doing `grep -R getrecords *`, the only thing you have to change is
>>> your handling of typenames.
>>>
>>> In owslib.csw.getrecords, qtype is a helper to define what kind of CSW
>>> metadata to query for (default is 'dataset', others are 'service',
>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>
>>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>>> you pass this as an OWSLib constraint object (see
>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) in
>>> a list.
>>>
>>> I've updated my fork of ckanext-spatial per above:
>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>
>>> (this is UNTESTED).  Should I submit this as a PR?  Do ckanext-spatial
>>> PRs run tests? Then I can submit PR.
>>>
>>>
>>>> We'll investigate, but we'll need to OWSLib version you are using that
>>>> you can find with the command I mentioned before.
>>>>
>>>>
>>>>> Anyway, if  I look at the harvest object like you described I 
>>>>> don't know
>>>>> what is wrong with it. It is too big to post on the list.
>>>> Can you post it online somewhere and link to it? I'm curious to see
>>>> what might be causing the ckan-pycsw command to fail.
>>>>
>>>>
>>>>> I have also another problem, I cannot see the "Add harvest source" 
>>>>> button on
>>>>> the
>>>>>
>>>>> http://{my-ckan-instance}/harvest
>>>>>
>>>>> Website. It has disapperared.
>>>> Make sure that you are using the latest "stable" branch on
>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>
>>>> git checkout stable
>>>> git pull
>>>>
>>>> (and of course that you are logged in)
>>>>
>>>>
>>>>> Sorry, but I am getting so many issues at the same time.
>>>> Sorry about the troubles, hopefully we'll get there.
>>>>
>>>> Adrià
>>>>
>>>>
>>>>
>>>>> Hilde
>>>>>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev