[ckan-dev] CSW, ISO and Spatial Extension

Thu Nov 1 10:37:02 UTC 2012

On 31 October 2012 12:19, Florian Marienfeld
<florian.marienfeld at fokus.fraunhofer.de> wrote:
> Hi,
>
> sounds great! So we are mostly interested in this
>> * Better and more efficient CSW and WAF harvesters
> What branch of ckanext-spatial should we branch from and sync our fork
> with? master or harvest-generic-iso?
The CSW and WAF harvesters will undergo significant changes, first on
a specific project extension and then we will port these changes to
ckanext-spatial.
I will suggest using the harvest-generic-iso branch of ckanext-spatial
if you want to branch it, it will probably be more useful if you want
to harvest generic iso docs.


Adrià


> Best regards,
>
> FLO
>
> On 29.10.12 12:40, Adrià Mercader wrote:
>> Hi all,
>>
>> Apologies for the late reply, I've been away for a few days, but I was
>> really happy to come back to see so many interest around CKAN and
>> CSW/ISO/harvesting!
>>
>> As you probably may have noticed having a look at the sources we are
>> currently doing a lot of refactoring and development on this area so
>> any feedback is really useful. I'll try to briefly explain the current
>> use case and what is our roadmap on this field for the next months.
>>
>> The csw/harvest/spatial/inspire extensions on their current form were
>> developed for an scenario where CKAN acts as a central repository for
>> datasets (or documents) that are harvested from remote CSW servers or
>> WAFs. These are imported into CKAN's own data model, exposing them on
>> the frontend, search, api etc. But For historical reasons, these
>> datasets are also exposed via a CSW interface, which supports the bare
>> minimum to be able to harvest the records on CKAN to a GeoNetwork
>> instance. As Ryan pointed out, the current implementation only exposes
>> via CSW the records that were harvested from other servers, not the
>> ones created by other means, eg frontend or API.
>>
>> This is the current situation, and the following is what is likely to
>> happen on the following months (some things have already happened):
>> * Consolidate all plugins and extensions into ckanext-spatial / ckanext-harvest
>> * Enhance the validation: choose different ISO schemas (customizable
>> per source) and notify publishers of errors.
>> * Better and more efficient CSW and WAF harvesters
>> * Integration with pycsw to provide a full CSW interface
>>
>> These changes, apart from a more flexible, robust and dev friendly
>> interface, don't add significant top level changes to the current use
>> case (ie harvest remote ISO records into CKAN). At a later stage
>> (depending on resources and availability) we would also like to
>> support the creation of spatial metadata directly into CKAN via the
>> frontend, uploading metadata files or via a form wizard with all the
>> relevant fields. The exact requirements are likely to change on each
>> project, but hopefully we can provide a nice and powerful platform for
>> geospatial catalogues.
>>
>> We'll try to keep the list updated with any new developments, and
>> please keep coming with your doubts and feedback.
>>
>> Cheers,
>>
>> Adrià
>>
>>
>>
>> On 29 October 2012 08:48, Florian Marienfeld
>> <florian.marienfeld at fokus.fraunhofer.de> wrote:
>>> Hi all,
>>>
>>> we are also working on the CSW harvester for the German Open Data
>>> Portal. We need to import geo-related ISO191xx metadata from several CSW
>>> servers, including geonetworks. The two key challenges are filtering the
>>> fraction of records that has proper resource+license and transforming
>>> into CKAN-package format. But instead of general CKAN metadata we want
>>> to procude packages that conform the the German Open Gov Data metadata
>>> json-schema:
>>> https://github.com/fraunhoferfokus/ogd-metadata
>>>
>>> We'll feed our questions, code and findings into this list and
>>> https://github.com/okfn/ckanext-spatial .
>>>
>>> Looking forward to collaborating on this, best regards,
>>>
>>> FLO
>>>
>>> On 25.10.12 18:50, Ryan Clark wrote:
>>>> Hi!
>>>>
>>>> I'm in the process of evaluating CKAN's potential for use in a
>>>> distributed data system for geothermal information that is being built
>>>> in the US. A fundamental component of our project is the concept of a
>>>> catalog which contains standardized metadata describing distributed data
>>>> resources across the web. At that level, CKAN seems like a very nice fit!
>>>>
>>>> We are committed to providing metadata according to a custom profile for
>>>> ISO 19139, and to implementing a CSW interface. I've started by playing
>>>> with the spatial, harvest, and csw extensions, which are clearly
>>>> undergoing significant development currently -- I see that they've all
>>>> been merged into one extension as of last week! What I found was that
>>>> while the CSW interface responds to queries, it does not expose any of
>>>> the resources that I had published in my development environment --
>>>> GetRecords and GetRecordByID never return anything.
>>>>
>>>> While I'm new to SQLalchemy, spatial.controllers.csw:GetRecords looks to
>>>> me like it is configured to only return records that were brought into
>>>> the catalog via a harvest mechanism. Is this true?
>>>>
>>>> If so, that brings up what I suspect is the primary issue I would have
>>>> to deal with in order to implement CKAN -- I suspect that there would
>>>> need to be a mapping from CKAN's "internal metadata model" (basically
>>>> the package table) into XML records that can be exposed via CSW. It is
>>>> also likely that in order to generate a complete ISO 19139 record might
>>>> require some additional content in the internal model.
>>>>
>>>> Does this sound like I'm on the right track? Is there any existing
>>>> pattern for doing this kind of translation where I might start looking
>>>> for guidance?
>>>>
>>>> Thanks!
>>>> ____________________
>>>>
>>>> Ryan Clark
>>>> ryan.clark at azgs.az.gov <mailto:ryan.clark at azgs.az.gov>
>>>> (520) 302-4871
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/florian.marienfeld%40fokus.fraunhofer.de
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/adria.mercader%40okfn.org
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/florian.marienfeld%40fokus.fraunhofer.de
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev/adria.mercader%40okfn.org