[ckan-dev] pycsw

Tue Oct 15 15:30:56 UTC 2013

Ok, thanks. What I did now is uninstall OWSLib and install version 
0.8.2. The harvesting jobs are running now.

I also found the console output which I have sent to a log file instead 
(filelog handler).

Hilde

On 10/15/2013 5:01 PM, Tom Kralidis wrote:
> Thanks. Ah.
>
> It looks like the version of OWSLib that ckanext-spatial pegs has that 
> warning implemented as a raise.  This is fixed by doing a 
> warnings.warn instead in 0.8.2 [1].
>
> Once ckanext-spatial accpets my PR [2] (which includes using 
> getrecords2 support and pegging to OWSLib 0.8.2), this should be 
> fixed.  At which point one can update ckanext-spatial and we can 
> continue testing.
>
> [1] https://github.com/geopython/OWSLib/issues/74
> [2] https://github.com/okfn/ckanext-spatial/pull/42
>
>
> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>
>> Date: Tue, 15 Oct 2013 14:32:54 +0200
>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Subject: Re: [ckan-dev] pycsw
>>
>> That is what I did, but I really get a Job Error.
>> Error gathering the identifiers from the CSW server [Please use the 
>> updated 'getrecords2' method instead of 'getrecords'. The 
>> 'getrecords' method will be upgraded to use the 'getrecords2' 
>> parameters in a future version of OWSLib.]
>> Nothing is updated or deleted or added.
>>
>> Here they seem to have commented out the lines causing the error.
>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide#Harvest_remote_metadata_into_CKAN_39s_catalogue 
>>
>>
>> I am also wondering why I don't get anything in my window where I run 
>> the command
>> paster --plugin=ckanext-harvest harvester gather_consumer 
>> --config=development.ini
>>
>> I think in the past I got information what was going on (like on the 
>> Webpage just mentioned before), now nothing.
>> Maybe something is corrupt in my installation ?
>>
>> Hilde
>>
>>
>> On 10/15/2013 1:41 PM, Tom Kralidis wrote:
>>>
>>> FYI the getrecords2 is a warning and should not affect functionality 
>>> whatsoever. Try a harvesting job that worked before?
>>>
>>> On Tue, 15 Oct 2013, Hildegard Gerlach wrote:
>>>
>>>> Date: Tue, 15 Oct 2013 11:51:54 +0200
>>>> From: Hildegard Gerlach <hildegard.gerlach at jrc.ec.europa.eu>
>>>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>> Subject: Re: [ckan-dev] pycsw
>>>>
>>>> Hi again,
>>>>
>>>> I am not sure how to proceed now. I will put one problem at a time, 
>>>> otherwise it gets too confusing.
>>>>
>>>> Harvesting from a csw server had worked before, but in the meantime 
>>>> I have upgraded CKAN (2.1 now and the extensions to stable) and 
>>>> installed pycsw, so I don't know where from I get the problem 
>>>> (getrecords2).
>>>>
>>>> When I look at the pip-requirements file in ckanext-spatial, there is
>>>> GeoAlchemy>=0.6
>>>> Shapely>=1.2.13
>>>> # Temporal requirement until there is an OWSLib release
>>>> # that requires python-dateutil<2.0
>>>> -e 
>>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=owslib
>>>> lxml>=2.3
>>>> argparse
>>>> pyparsing==1.5.6
>>>>
>>>> But my version of owslib is different
>>>> pip freeze | grep -i owslib
>>>> -e 
>>>> git+https://github.com/geopython/OWSLib.git@4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664#egg=OWSLib-dev
>>>>
>>>> which has been probably installed with pycsw  following
>>>> http://docs.ckan.org/projects/ckanext-spatial/en/latest/csw.html#ckan-pycsw 
>>>>
>>>> # From now on the virtualenv should be activated
>>>>
>>>> git clone https://github.com/geopython/pycsw.git
>>>> cd pycsw
>>>> # Remember to use at least pycsw 1.6.1
>>>> git checkout 1.6.1
>>>> pip install -e .
>>>> python setup.py build
>>>> python setup.py install
>>>>
>>>>
>>>> So, in order to be able to harvest from csw servers, should I 
>>>> manually susbtitute the file
>>>>
>>>> ckanext/spatial/lib/csw_client.py
>>>> with this one ?
>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>>
>>>> Thanks
>>>>
>>>> Hilde
>>>>
>>>> On 10/14/2013 9:31 PM, Tom Kralidis wrote:
>>>>>
>>>>>
>>>>> On Mon, 14 Oct 2013, Adrià Mercader wrote:
>>>>>
>>>>>> Hi Tom,
>>>>>>
>>>>>> Many thanks for this. I don't think there are automated tests for 
>>>>>> this
>>>>>> part of the code, just send the PR and I'll test it to make sure it
>>>>>> works.
>>>>>>
>>>>>
>>>>> Done: https://github.com/okfn/ckanext-spatial/pull/42
>>>>>
>>>>>> BTW, I'm I right in thinking that this change is present in 
>>>>>> owslib 0.8.2?
>>>>>>
>>>>>> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664 
>>>>>> Just so we can link it from our requirements file instead of the
>>>>>> particular commit:
>>>>>>
>>>>>
>>>>> Correct, I've updated in my PR.
>>>>>
>>>>>> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5 
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> 
>>>>>> wrote:
>>>>>>> Hi: I will try to setup a CKAN development environment, but in the
>>>>>>> meantime, comments interleaved:
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader 
>>>>>>> <adria.mercader at okfn.org> wrote:
>>>>>>>> Hi Hilde,
>>>>>>>>
>>>>>>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>>>>>>> relatively recent addition and may still have some rough edges.
>>>>>>>>
>>>>>>>> Let's take it one step at a time and see if we can make 
>>>>>>>> progress, see
>>>>>>>> below for comments.
>>>>>>>>
>>>>>>>> On 10 October 2013 15:19, Hildegard Gerlach
>>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>> Dear Angelos and Adrià,
>>>>>>>>> I am using Redhat 6, ckan version 2.2a
>>>>>>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>>>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>>>>>>
>>>>>>>> I strongly recommend always using stable versions unless you are
>>>>>>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>>>>>>> branches of ckanext-spatial and ckanext-harvest
>>>>>>>> The rest of the versions seem fine
>>>>>>>>
>>>>>>>>> I am using a Postgis database for CKAN and have done 
>>>>>>>>> harvesting from 2
>>>>>>>>> remote sources.
>>>>>>>> Great.
>>>>>>>>
>>>>>>>>> However I don't manage to load these sources into the newly
>>>>>>>>> created database for pycsw.
>>>>>>>>> In the pycsw.cfg file I have only changed the home and 
>>>>>>>>> database settings.
>>>>>>>>> The database table has been created successfully with this 
>>>>>>>>> command
>>>>>>>>>
>>>>>>>
>>>>>>> As reference, can we see/you post what a few of the metadata 
>>>>>>> records
>>>>>>> look like from these sources?
>>>>>>>
>>>>>>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>>>>>>> That sounds fine
>>>>>>>>
>>>>>>>>
>>>>>>>>> In the meantime I went back to version 2.1, but I am not sure 
>>>>>>>>> if this made
>>>>>>>>> any change.
>>>>>>>>> I did
>>>>>>>>>
>>>>>>>>> git checkout release-v2.1
>>>>>>>>> git pull
>>>>>>>>>
>>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>>>
>>>>>>>>> for CKAN, the harvester and spatial extension. Is this the 
>>>>>>>>> correct way ?
>>>>>>>> I wouldn't use the --upgrade flag unless you find some version 
>>>>>>>> number
>>>>>>>> conflicts. This will upgrade all dependencies which might have
>>>>>>>> unexpected results
>>>>>>>>
>>>>>>>>
>>>>>>>>> When harvesting before I had not specified anything in the 
>>>>>>>>> development.ini
>>>>>>>>> file for
>>>>>>>>> Now I have set it to
>>>>>>>>> ckan.spatial.validator.profiles = iso19193eden
>>>>>>>> That is good, this is probably the best profile for generic iso
>>>>>>>> documents (iso19193ngdc is even a bit more permissive)
>>>>>>>>
>>>>>>>>> and tried to reharvest, but then I get another error:
>>>>>>>>> Error gathering the identifiers from the CSW server [Please 
>>>>>>>>> use the updated
>>>>>>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' 
>>>>>>>>> method will
>>>>>>>>> be upgraded to use the 'getrecords2' parameters in a future 
>>>>>>>>> version of
>>>>>>>>> OWSLib
>>>>>>>> That might be an issue with an incompatible version of OWSLib 
>>>>>>>> that got
>>>>>>>> installed when you upgraded the requirements, we've had some 
>>>>>>>> issues
>>>>>>>> with incompatibilities with owslib.
>>>>>>>>
>>>>>>>> What version of OWSLib do you have installed? Run the following 
>>>>>>>> with
>>>>>>>> your virtualenv activated and post the result:
>>>>>>>>
>>>>>>>
>>>>>>> For the record, this is a Python DeprecationWarning in OWSLib.  
>>>>>>> There
>>>>>>> have been some huge improvements to CSW in OWSLib (in
>>>>>>> owslib.csw.getrecords2, basically improved filter support), 
>>>>>>> which will
>>>>>>> replace/become owslib.csw.getrecords).  Having said this, this 
>>>>>>> is just
>>>>>>> a warning for downstream developers to update their use of OWSLib.
>>>>>>>
>>>>>>>
>>>>>>>> pip freeze | grep -i owslib
>>>>>>>>
>>>>>>>>> A workaround, but not solution, is here:
>>>>>>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>>>>>>> The workaround (not a fix) for the above error is to comment 
>>>>>>>>> out the lines
>>>>>>>>> (186-188) that raised the DeprecationWarning exception in the 
>>>>>>>>> following
>>>>>>>>> Python source file 
>>>>>>>>> (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>>>>>>> #raise DeprecationWarning("""Please use the updated 
>>>>>>>>> 'getrecords2' method
>>>>>>>>> instead of 'getrecords'. #The 'getrecords' method will be 
>>>>>>>>> upgraded to use
>>>>>>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>>>>>>> Sounds like we should update our harvesters to use getrecords2. We
>>>>>>>> have never found this, so there definitely seems to be a 
>>>>>>>> conflict with
>>>>>>>> a newer version of OWSlib.
>>>>>>>
>>>>>>> ckanext-spatial devs:
>>>>>>>
>>>>>>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>>>>>>> doing `grep -R getrecords *`, the only thing you have to change is
>>>>>>> your handling of typenames.
>>>>>>>
>>>>>>> In owslib.csw.getrecords, qtype is a helper to define what kind 
>>>>>>> of CSW
>>>>>>> metadata to query for (default is 'dataset', others are 'service',
>>>>>>> etc.  This is loosely tied to dc:type or apiso:Type.
>>>>>>>
>>>>>>> In owslib.csw.getrecords2 (which will become 
>>>>>>> owslib.csw.getrecords),
>>>>>>> you pass this as an OWSLib constraint object (see
>>>>>>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) 
>>>>>>> in a list.
>>>>>>>
>>>>>>> I've updated my fork of ckanext-spatial per above:
>>>>>>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e 
>>>>>>> (this is UNTESTED).  Should I submit this as a PR?  Do 
>>>>>>> ckanext-spatial
>>>>>>> PRs run tests? Then I can submit PR.
>>>>>>>
>>>>>>>
>>>>>>>> We'll investigate, but we'll need to OWSLib version you are 
>>>>>>>> using that
>>>>>>>> you can find with the command I mentioned before.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Anyway, if  I look at the harvest object like you described I 
>>>>>>>>> don't know
>>>>>>>>> what is wrong with it. It is too big to post on the list.
>>>>>>>> Can you post it online somewhere and link to it? I'm curious to 
>>>>>>>> see
>>>>>>>> what might be causing the ckan-pycsw command to fail.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I have also another problem, I cannot see the "Add harvest 
>>>>>>>>> source" button on
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> http://{my-ckan-instance}/harvest
>>>>>>>>>
>>>>>>>>> Website. It has disapperared.
>>>>>>>> Make sure that you are using the latest "stable" branch on
>>>>>>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>>>>>>
>>>>>>>> git checkout stable
>>>>>>>> git pull
>>>>>>>>
>>>>>>>> (and of course that you are logged in)
>>>>>>>>
>>>>>>>>
>>>>>>>>> Sorry, but I am getting so many issues at the same time.
>>>>>>>> Sorry about the troubles, hopefully we'll get there.
>>>>>>>>
>>>>>>>> Adrià
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hilde
>>>>>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev