[ckan-dev] pycsw

Mon Oct 14 19:31:22 UTC 2013

On Mon, 14 Oct 2013, Adrià Mercader wrote:

> Hi Tom,
>
> Many thanks for this. I don't think there are automated tests for this
> part of the code, just send the PR and I'll test it to make sure it
> works.
>

Done: https://github.com/okfn/ckanext-spatial/pull/42

> BTW, I'm I right in thinking that this change is present in owslib 0.8.2?
>
> https://github.com/geopython/OWSLib/commit/4b0a62cd37a5a03aafc5c0cd9606e0658d5ac664
>
> Just so we can link it from our requirements file instead of the
> particular commit:
>

Correct, I've updated in my PR.

> https://github.com/okfn/ckanext-spatial/blob/master/pip-requirements.txt#L5
>
>
> Cheers,
>
> Adrià
>
>
> On 14 October 2013 14:18, Tom Kralidis <tomkralidis at gmail.com> wrote:
>> Hi: I will try to setup a CKAN development environment, but in the
>> meantime, comments interleaved:
>>
>>
>> On Fri, Oct 11, 2013 at 7:05 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:
>>> Hi Hilde,
>>>
>>> I'm sorry to hear that you are finding problems. The ckan-pycsw is
>>> relatively recent addition and may still have some rough edges.
>>>
>>> Let's take it one step at a time and see if we can make progress, see
>>> below for comments.
>>>
>>> On 10 October 2013 15:19, Hildegard Gerlach
>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>> Dear Angelos and Adrià,
>>>> I am using Redhat 6, ckan version 2.2a
>>>> PostgreSQL 8.4.13, Postgis 1.5.3
>>>>  SQLAlchemy-0.7.8   (installed by CKAN requirements.txt)
>>>
>>> I strongly recommend always using stable versions unless you are
>>> developing CKAN or the extensions. Use ckan 2.1, and the "stable"
>>> branches of ckanext-spatial and ckanext-harvest
>>> The rest of the versions seem fine
>>>
>>>> I am using a Postgis database for CKAN and have done harvesting from 2
>>>> remote sources.
>>> Great.
>>>
>>>> However I don't manage to load these sources into the newly
>>>> created database for pycsw.
>>>> In the pycsw.cfg file I have only changed the home and database settings.
>>>> The database table has been created successfully with this command
>>>>
>>
>> As reference, can we see/you post what a few of the metadata records
>> look like from these sources?
>>
>>>> paster ckan-pycsw setup -p /etc/ckan/default/pycsw.cfg
>>> That sounds fine
>>>
>>>
>>>> In the meantime I went back to version 2.1, but I am not sure if this made
>>>> any change.
>>>> I did
>>>>
>>>> git checkout release-v2.1
>>>> git pull
>>>>
>>>> pip install --upgrade -r requirements.txt
>>>>
>>>> for CKAN, the harvester and spatial extension. Is this the correct way ?
>>> I wouldn't use the --upgrade flag unless you find some version number
>>> conflicts. This will upgrade all dependencies which might have
>>> unexpected results
>>>
>>>
>>>> When harvesting before I had not specified anything in the development.ini
>>>> file for
>>>> Now I have set it to
>>>> ckan.spatial.validator.profiles = iso19193eden
>>> That is good, this is probably the best profile for generic iso
>>> documents (iso19193ngdc is even a bit more permissive)
>>>
>>>> and tried to reharvest, but then I get another error:
>>>> Error gathering the identifiers from the CSW server [Please use the updated
>>>> 'getrecords2' method instead of 'getrecords'. The 'getrecords' method will
>>>> be upgraded to use the 'getrecords2' parameters in a future version of
>>>> OWSLib
>>> That might be an issue with an incompatible version of OWSLib that got
>>> installed when you upgraded the requirements, we've had some issues
>>> with incompatibilities with owslib.
>>>
>>> What version of OWSLib do you have installed? Run the following with
>>> your virtualenv activated and post the result:
>>>
>>
>> For the record, this is a Python DeprecationWarning in OWSLib.  There
>> have been some huge improvements to CSW in OWSLib (in
>> owslib.csw.getrecords2, basically improved filter support), which will
>> replace/become owslib.csw.getrecords).  Having said this, this is just
>> a warning for downstream developers to update their use of OWSLib.
>>
>>
>>> pip freeze | grep -i owslib
>>>
>>>> A workaround, but not solution, is here:
>>>> https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide
>>>> The workaround (not a fix) for the above error is to comment out the lines
>>>> (186-188) that raised the DeprecationWarning exception in the following
>>>> Python source file (/usr/lib/ckan/default/src/owslib/owslib/csw.py).
>>>> #raise DeprecationWarning("""Please use the updated 'getrecords2' method
>>>> instead of 'getrecords'. #The 'getrecords' method will be upgraded to use
>>>> the 'getrecords2' parameters #in a future version of OWSLib.""")
>>> Sounds like we should update our harvesters to use getrecords2. We
>>> have never found this, so there definitely seems to be a conflict with
>>> a newer version of OWSlib.
>>
>> ckanext-spatial devs:
>>
>> Agreed. From looking at https://github.com/okfn/ckanext-spatial and
>> doing `grep -R getrecords *`, the only thing you have to change is
>> your handling of typenames.
>>
>> In owslib.csw.getrecords, qtype is a helper to define what kind of CSW
>> metadata to query for (default is 'dataset', others are 'service',
>> etc.  This is loosely tied to dc:type or apiso:Type.
>>
>> In owslib.csw.getrecords2 (which will become owslib.csw.getrecords),
>> you pass this as an OWSLib constraint object (see
>> https://github.com/geopython/OWSLib/blob/master/owslib/csw.py#L284) in
>> a list.
>>
>> I've updated my fork of ckanext-spatial per above:
>> https://github.com/tomkralidis/ckanext-spatial/commit/1fdf0cde4c6feed2b3f0675ff88145dd4b626e6e
>> (this is UNTESTED).  Should I submit this as a PR?  Do ckanext-spatial
>> PRs run tests? Then I can submit PR.
>>
>>
>>> We'll investigate, but we'll need to OWSLib version you are using that
>>> you can find with the command I mentioned before.
>>>
>>>
>>>> Anyway, if  I look at the harvest object like you described I don't know
>>>> what is wrong with it. It is too big to post on the list.
>>> Can you post it online somewhere and link to it? I'm curious to see
>>> what might be causing the ckan-pycsw command to fail.
>>>
>>>
>>>> I have also another problem, I cannot see the "Add harvest source" button on
>>>> the
>>>>
>>>> http://{my-ckan-instance}/harvest
>>>>
>>>> Website. It has disapperared.
>>> Make sure that you are using the latest "stable" branch on
>>> ckanext-harvest, we fixed some problems with this a while ago:
>>>
>>> git checkout stable
>>> git pull
>>>
>>> (and of course that you are logged in)
>>>
>>>
>>>> Sorry, but I am getting so many issues at the same time.
>>> Sorry about the troubles, hopefully we'll get there.
>>>
>>> Adrià
>>>
>>>
>>>
>>>> Hilde
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>