[ckan-dev] CSW, ISO and Spatial Extension

Mon Oct 29 11:40:31 UTC 2012

Hi all,

Apologies for the late reply, I've been away for a few days, but I was
really happy to come back to see so many interest around CKAN and
CSW/ISO/harvesting!

As you probably may have noticed having a look at the sources we are
currently doing a lot of refactoring and development on this area so
any feedback is really useful. I'll try to briefly explain the current
use case and what is our roadmap on this field for the next months.

The csw/harvest/spatial/inspire extensions on their current form were
developed for an scenario where CKAN acts as a central repository for
datasets (or documents) that are harvested from remote CSW servers or
WAFs. These are imported into CKAN's own data model, exposing them on
the frontend, search, api etc. But For historical reasons, these
datasets are also exposed via a CSW interface, which supports the bare
minimum to be able to harvest the records on CKAN to a GeoNetwork
instance. As Ryan pointed out, the current implementation only exposes
via CSW the records that were harvested from other servers, not the
ones created by other means, eg frontend or API.

This is the current situation, and the following is what is likely to
happen on the following months (some things have already happened):
* Consolidate all plugins and extensions into ckanext-spatial / ckanext-harvest
* Enhance the validation: choose different ISO schemas (customizable
per source) and notify publishers of errors.
* Better and more efficient CSW and WAF harvesters
* Integration with pycsw to provide a full CSW interface

These changes, apart from a more flexible, robust and dev friendly
interface, don't add significant top level changes to the current use
case (ie harvest remote ISO records into CKAN). At a later stage
(depending on resources and availability) we would also like to
support the creation of spatial metadata directly into CKAN via the
frontend, uploading metadata files or via a form wizard with all the
relevant fields. The exact requirements are likely to change on each
project, but hopefully we can provide a nice and powerful platform for
geospatial catalogues.

We'll try to keep the list updated with any new developments, and
please keep coming with your doubts and feedback.

Cheers,

Adrià

On 29 October 2012 08:48, Florian Marienfeld
<florian.marienfeld at fokus.fraunhofer.de> wrote:
> Hi all,
>
> we are also working on the CSW harvester for the German Open Data
> Portal. We need to import geo-related ISO191xx metadata from several CSW
> servers, including geonetworks. The two key challenges are filtering the
> fraction of records that has proper resource+license and transforming
> into CKAN-package format. But instead of general CKAN metadata we want
> to procude packages that conform the the German Open Gov Data metadata
> json-schema:
> https://github.com/fraunhoferfokus/ogd-metadata
>
> We'll feed our questions, code and findings into this list and
> https://github.com/okfn/ckanext-spatial .
>
> Looking forward to collaborating on this, best regards,
>
> FLO
>
> On 25.10.12 18:50, Ryan Clark wrote:
>> Hi!
>>
>> I'm in the process of evaluating CKAN's potential for use in a
>> distributed data system for geothermal information that is being built
>> in the US. A fundamental component of our project is the concept of a
>> catalog which contains standardized metadata describing distributed data
>> resources across the web. At that level, CKAN seems like a very nice fit!
>>
>> We are committed to providing metadata according to a custom profile for
>> ISO 19139, and to implementing a CSW interface. I've started by playing
>> with the spatial, harvest, and csw extensions, which are clearly
>> undergoing significant development currently -- I see that they've all
>> been merged into one extension as of last week! What I found was that
>> while the CSW interface responds to queries, it does not expose any of
>> the resources that I had published in my development environment --
>> GetRecords and GetRecordByID never return anything.
>>
>> While I'm new to SQLalchemy, spatial.controllers.csw:GetRecords looks to
>> me like it is configured to only return records that were brought into
>> the catalog via a harvest mechanism. Is this true?
>>
>> If so, that brings up what I suspect is the primary issue I would have
>> to deal with in order to implement CKAN -- I suspect that there would
>> need to be a mapping from CKAN's "internal metadata model" (basically
>> the package table) into XML records that can be exposed via CSW. It is
>> also likely that in order to generate a complete ISO 19139 record might
>> require some additional content in the internal model.
>>
>> Does this sound like I'm on the right track? Is there any existing
>> pattern for doing this kind of translation where I might start looking
>> for guidance?
>>
>> Thanks!
>> ____________________
>>
>> Ryan Clark
>> ryan.clark at azgs.az.gov <mailto:ryan.clark at azgs.az.gov>
>> (520) 302-4871
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/florian.marienfeld%40fokus.fraunhofer.de
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/adria.mercader%40okfn.org