[ckan-dev] CKAN CSW Harvest - GeoServer Error

Tue Feb 9 08:17:07 UTC 2016

Thanks for this Tom.

I can confirm we get the same error; I am assuming this is something that needs to be taken up with the developers of the CSW plug-in for GeoServer?

In the meantime, we will apply a monkey patch to our CKAN code.

Derek 

>>> Tom Kralidis <tomkralidis at gmail.com> 02/09/16 3:40 AM >>>
FYI the CKAN CSW client functionality was updated to force sorting as part of
its interaction with a CSW [1].  This ensures concise, non-duplicated
results from
any CSW.  I'm guessing this may be an issue.

Can you try the folllowing updated workflow as part of your isolated testing? :

from owslib.csw import CatalogueServiceWeb
from owslib.fes import SortBy, SortProperty

sortby = SortBy([SortProperty('dc:identifier')])

URL = 'http://localhost:8080/geoserver/csw'
csw = CatalogueServiceWeb(URL)
csw.getrecords2(maxrecords=20, sortby=sortby)
for rec in csw.records:
    print csw.records[rec].title

If you get an exception like CKAN proper does, then it's an issue with the
CSW server's sorting support.  FWIW, I tried the above updated test against
another GeoServer-based CSW which resulted in the exact same error as below.

Otherwise, we can dig deeper.

..Tom

[1] https://github.com/ckan/ckanext-spatial/commit/eb9cd6fc1c8bac2c1dafed29a43e975a94c918b6

On Mon, Feb 8, 2016 at 5:41 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:
> This looks like a mismatch between OWSLib (the library that parses the
> rsults from the CSW server under the hood) and GeoServer.
>
> I'm CCing Tom, one of OWSLib devs who maybe can shed more light into this.
>
> Adrià
>
> On 7 February 2016 at 15:49, Derek Hohls <dhohls at csir.co.za> wrote:
>> We are trying to setup our GeoServer for harvesting by CKAN.
>>
>> The plug-in has been installed and all seems good.
>>
>> If I use Python and owslib.csw, then I am able to communicate and get
>> results e.g.
>>
>>>>> from owslib.csw import CatalogueServiceWeb
>> URL = 'http://localhost:8080/geoserver/csw'
>>>>> csw = CatalogueServiceWeb(URL)
>>>>> csw.getrecords2(maxrecords=20)
>>>>> csw.results
>> {'matches': 25, 'nextrecord': 21, 'returned': 20}
>>>>> for rec in csw.records:
>> ...     print csw.records[rec].title
>> ...
>> Tasmania water bodies
>> Tasmania cities
>> Tasmania state boundaries
>> Tasmania roads
>> USA Population
>> World rectangle
>>
>> However, trying a harvest from CKAN results in "dc:" errors:
>>
>> 2016-02-07 15:37:49,325 DEBUG [ckanext.spatial.harvesters.csw.CSW.gather]
>> Starting gathering for http://localhost:8080/geoserver/csw/
>> 2016-02-07 15:37:49,325 INFO  [ckanext.spatial.lib.csw_client] Making CSW
>> request: getrecords2 {'typenames': 'csw:Record', 'maxrecords': 10, 'sortby':
>> <owslib.fes.SortBy object at 0x7f0c541551d0>, 'outputschema':
>> 'http://www.isotc211.org/2005/gmd', 'cql': None, 'startposition': 0, 'esn':
>> 'brief', 'constraints': []}
>> 2016-02-07 15:37:50,403 ERROR [ckanext.spatial.harvesters.csw.CSW.gather]
>> Exception: Traceback (most recent call last):
>>   File
>> "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/harvesters/csw.py",
>> line 95, in gather_stage
>>     for identifier in self.csw.getidentifiers(page=10,
>> outputschema=self.output_schema(), cql=cql):
>>   File
>> "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/lib/csw_client.py",
>> line 127, in getidentifiers
>>     csw.getrecords2(**kwa)
>>   File
>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/owslib/csw.py",
>> line 341, in getrecords2
>>     self._invoke()
>>   File
>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/owslib/csw.py",
>> line 605, in _invoke
>>     raise ows.ExceptionReport(self._exml, self.owscommon.namespace)
>> ExceptionReport: 'Request failed due to: Sorting on dc:identifier/dc:value
>> is not supported.\nSorting on dc:identifier/dc:value is not supported.'
>> 2016-02-07 15:37:50,455 INFO  [ckanext.harvest.model] Error gathering the
>> identifiers from the CSW server ['Request failed due to: Sorting on
>> dc:identifier/dc:value is not supported.\nSorting on dc:identifier/dc:value
>> is not supported.']
>> 2016-02-07 15:37:50,462 ERROR [ckanext.harvest.queue] Gather stage failed
>>
>> Can anyone help with this (running GeoServer 2.8 and CKAN 2.5)
>>
>> Thanks
>> Derek
>>
>>
>> --
>> This message is subject to the CSIR's copyright terms and conditions, e-mail
>> legal notice, and implemented Open Document Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>>
>> This message has been scanned for viruses and dangerous content by
>> MailScanner,
>> and is believed to be clean.
>>
>>
>> Please consider the environment before printing this email.
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.

-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160209/c98bd7e0/attachment-0003.html>