[ckan-dev] CSW, ISO and Spatial Extension

Wed Oct 31 12:19:36 UTC 2012

Hi,

sounds great! So we are mostly interested in this
> * Better and more efficient CSW and WAF harvesters
What branch of ckanext-spatial should we branch from and sync our fork
with? master or harvest-generic-iso?

Best regards,

FLO

On 29.10.12 12:40, Adrià Mercader wrote:
> Hi all,
> 
> Apologies for the late reply, I've been away for a few days, but I was
> really happy to come back to see so many interest around CKAN and
> CSW/ISO/harvesting!
> 
> As you probably may have noticed having a look at the sources we are
> currently doing a lot of refactoring and development on this area so
> any feedback is really useful. I'll try to briefly explain the current
> use case and what is our roadmap on this field for the next months.
> 
> The csw/harvest/spatial/inspire extensions on their current form were
> developed for an scenario where CKAN acts as a central repository for
> datasets (or documents) that are harvested from remote CSW servers or
> WAFs. These are imported into CKAN's own data model, exposing them on
> the frontend, search, api etc. But For historical reasons, these
> datasets are also exposed via a CSW interface, which supports the bare
> minimum to be able to harvest the records on CKAN to a GeoNetwork
> instance. As Ryan pointed out, the current implementation only exposes
> via CSW the records that were harvested from other servers, not the
> ones created by other means, eg frontend or API.
> 
> This is the current situation, and the following is what is likely to
> happen on the following months (some things have already happened):
> * Consolidate all plugins and extensions into ckanext-spatial / ckanext-harvest
> * Enhance the validation: choose different ISO schemas (customizable
> per source) and notify publishers of errors.
> * Better and more efficient CSW and WAF harvesters
> * Integration with pycsw to provide a full CSW interface
> 
> These changes, apart from a more flexible, robust and dev friendly
> interface, don't add significant top level changes to the current use
> case (ie harvest remote ISO records into CKAN). At a later stage
> (depending on resources and availability) we would also like to
> support the creation of spatial metadata directly into CKAN via the
> frontend, uploading metadata files or via a form wizard with all the
> relevant fields. The exact requirements are likely to change on each
> project, but hopefully we can provide a nice and powerful platform for
> geospatial catalogues.
> 
> We'll try to keep the list updated with any new developments, and
> please keep coming with your doubts and feedback.
> 
> Cheers,
> 
> Adrià
> 
> 
> 
> On 29 October 2012 08:48, Florian Marienfeld
> <florian.marienfeld at fokus.fraunhofer.de> wrote:
>> Hi all,
>>
>> we are also working on the CSW harvester for the German Open Data
>> Portal. We need to import geo-related ISO191xx metadata from several CSW
>> servers, including geonetworks. The two key challenges are filtering the
>> fraction of records that has proper resource+license and transforming
>> into CKAN-package format. But instead of general CKAN metadata we want
>> to procude packages that conform the the German Open Gov Data metadata
>> json-schema:
>> https://github.com/fraunhoferfokus/ogd-metadata
>>
>> We'll feed our questions, code and findings into this list and
>> https://github.com/okfn/ckanext-spatial .
>>
>> Looking forward to collaborating on this, best regards,
>>
>> FLO
>>
>> On 25.10.12 18:50, Ryan Clark wrote:
>>> Hi!
>>>
>>> I'm in the process of evaluating CKAN's potential for use in a
>>> distributed data system for geothermal information that is being built
>>> in the US. A fundamental component of our project is the concept of a
>>> catalog which contains standardized metadata describing distributed data
>>> resources across the web. At that level, CKAN seems like a very nice fit!
>>>
>>> We are committed to providing metadata according to a custom profile for
>>> ISO 19139, and to implementing a CSW interface. I've started by playing
>>> with the spatial, harvest, and csw extensions, which are clearly
>>> undergoing significant development currently -- I see that they've all
>>> been merged into one extension as of last week! What I found was that
>>> while the CSW interface responds to queries, it does not expose any of
>>> the resources that I had published in my development environment --
>>> GetRecords and GetRecordByID never return anything.
>>>
>>> While I'm new to SQLalchemy, spatial.controllers.csw:GetRecords looks to
>>> me like it is configured to only return records that were brought into
>>> the catalog via a harvest mechanism. Is this true?
>>>
>>> If so, that brings up what I suspect is the primary issue I would have
>>> to deal with in order to implement CKAN -- I suspect that there would
>>> need to be a mapping from CKAN's "internal metadata model" (basically
>>> the package table) into XML records that can be exposed via CSW. It is
>>> also likely that in order to generate a complete ISO 19139 record might
>>> require some additional content in the internal model.
>>>
>>> Does this sound like I'm on the right track? Is there any existing
>>> pattern for doing this kind of translation where I might start looking
>>> for guidance?
>>>
>>> Thanks!
>>> ____________________
>>>
>>> Ryan Clark
>>> ryan.clark at azgs.az.gov <mailto:ryan.clark at azgs.az.gov>
>>> (520) 302-4871
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/florian.marienfeld%40fokus.fraunhofer.de
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/adria.mercader%40okfn.org
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/florian.marienfeld%40fokus.fraunhofer.de