[ckan-dev] CKAN CSW Harvest - GeoServer Error

Tue Feb 9 12:16:23 UTC 2016

Hi Derek: agreed, something to bring up with GeoServer.

Cheers

..Tom

On Tue, 9 Feb 2016, Derek Hohls wrote:

> Date: Tue, 09 Feb 2016 10:17:07 +0200
> From: Derek Hohls <dhohls at csir.co.za>
> To: tomkralidis at gmail.com, adria.mercader at okfn.org
> Cc: ckan-dev at lists.okfn.org
> Subject: Re: [ckan-dev] CKAN CSW Harvest - GeoServer Error
> 
> Thanks for this Tom.
>
> I can confirm we get the same error; I am assuming this is something that needs to be taken up with the developers of the CSW plug-in for GeoServer?
>
> In the meantime, we will apply a monkey patch to our CKAN code.
>
> Derek
>
>>>> Tom Kralidis <tomkralidis at gmail.com> 02/09/16 3:40 AM >>>
> FYI the CKAN CSW client functionality was updated to force sorting as part of
> its interaction with a CSW [1].  This ensures concise, non-duplicated
> results from
> any CSW.  I'm guessing this may be an issue.
>
> Can you try the folllowing updated workflow as part of your isolated testing? :
>
> from owslib.csw import CatalogueServiceWeb
> from owslib.fes import SortBy, SortProperty
>
> sortby = SortBy([SortProperty('dc:identifier')])
>
> URL = 'http://localhost:8080/geoserver/csw'
> csw = CatalogueServiceWeb(URL)
> csw.getrecords2(maxrecords=20, sortby=sortby)
> for rec in csw.records:
>    print csw.records[rec].title
>
> If you get an exception like CKAN proper does, then it's an issue with the
> CSW server's sorting support.  FWIW, I tried the above updated test against
> another GeoServer-based CSW which resulted in the exact same error as below.
>
> Otherwise, we can dig deeper.
>
> ..Tom
>
> [1] https://github.com/ckan/ckanext-spatial/commit/eb9cd6fc1c8bac2c1dafed29a43e975a94c918b6
>
>
> On Mon, Feb 8, 2016 at 5:41 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:
>> This looks like a mismatch between OWSLib (the library that parses the
>> rsults from the CSW server under the hood) and GeoServer.
>>
>> I'm CCing Tom, one of OWSLib devs who maybe can shed more light into this.
>>
>> Adrià
>>
>> On 7 February 2016 at 15:49, Derek Hohls <dhohls at csir.co.za> wrote:
>>> We are trying to setup our GeoServer for harvesting by CKAN.
>>>
>>> The plug-in has been installed and all seems good.
>>>
>>> If I use Python and owslib.csw, then I am able to communicate and get
>>> results e.g.
>>>
>>>>>> from owslib.csw import CatalogueServiceWeb
>>> URL = 'http://localhost:8080/geoserver/csw'
>>>>>> csw = CatalogueServiceWeb(URL)
>>>>>> csw.getrecords2(maxrecords=20)
>>>>>> csw.results
>>> {'matches': 25, 'nextrecord': 21, 'returned': 20}
>>>>>> for rec in csw.records:
>>> ...     print csw.records[rec].title
>>> ...
>>> Tasmania water bodies
>>> Tasmania cities
>>> Tasmania state boundaries
>>> Tasmania roads
>>> USA Population
>>> World rectangle
>>>
>>> However, trying a harvest from CKAN results in "dc:" errors:
>>>
>>> 2016-02-07 15:37:49,325 DEBUG [ckanext.spatial.harvesters.csw.CSW.gather]
>>> Starting gathering for http://localhost:8080/geoserver/csw/
>>> 2016-02-07 15:37:49,325 INFO  [ckanext.spatial.lib.csw_client] Making CSW
>>> request: getrecords2 {'typenames': 'csw:Record', 'maxrecords': 10, 'sortby':
>>> <owslib.fes.SortBy object at 0x7f0c541551d0>, 'outputschema':
>>> 'http://www.isotc211.org/2005/gmd', 'cql': None, 'startposition': 0, 'esn':
>>> 'brief', 'constraints': []}
>>> 2016-02-07 15:37:50,403 ERROR [ckanext.spatial.harvesters.csw.CSW.gather]
>>> Exception: Traceback (most recent call last):
>>>   File
>>> "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/harvesters/csw.py",
>>> line 95, in gather_stage
>>>     for identifier in self.csw.getidentifiers(page=10,
>>> outputschema=self.output_schema(), cql=cql):
>>>   File
>>> "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/lib/csw_client.py",
>>> line 127, in getidentifiers
>>>     csw.getrecords2(**kwa)
>>>   File
>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/owslib/csw.py",
>>> line 341, in getrecords2
>>>     self._invoke()
>>>   File
>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/owslib/csw.py",
>>> line 605, in _invoke
>>>     raise ows.ExceptionReport(self._exml, self.owscommon.namespace)
>>> ExceptionReport: 'Request failed due to: Sorting on dc:identifier/dc:value
>>> is not supported.\nSorting on dc:identifier/dc:value is not supported.'
>>> 2016-02-07 15:37:50,455 INFO  [ckanext.harvest.model] Error gathering the
>>> identifiers from the CSW server ['Request failed due to: Sorting on
>>> dc:identifier/dc:value is not supported.\nSorting on dc:identifier/dc:value
>>> is not supported.']
>>> 2016-02-07 15:37:50,462 ERROR [ckanext.harvest.queue] Gather stage failed
>>>
>>> Can anyone help with this (running GeoServer 2.8 and CKAN 2.5)
>>>
>>> Thanks
>>> Derek
>>>
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions, e-mail
>>> legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> MailScanner,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
> -- 
> This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
> The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.
>
> This message has been scanned for viruses and dangerous content by MailScanner,
> and is believed to be clean.
>
> Please consider the environment before printing this email.
>
>
>
>
> -- 
> This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
> The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.
>
> This message has been scanned for viruses and dangerous content by MailScanner,
> and is believed to be clean.
>
> Please consider the environment before printing this email.
>
>