[ckan-dev] New interface for extending the Spatial Harvesters

[Adrià Mercader]

Hi all,

We know that there are quite a few of you who use the spatial
harvesters included on ckanext-spatial [1], which allow to import
spatial ISO docs to CKAN via CSW or WAFs.

One very common use case is having the need to modify the datasets
being created or updated from the ISO docs, for instance to add some
new field or modify an existing one. Up until now you had to extend to
the whole Harvester class in order to override the `get_package_dict`
method with your own.

We have just pushed a new interface makes this much easier, in the
same way that the rest of extension interface work across CKAN. You
can implement you own `get_package_dict` method on your own extension,
which will receive the default dict, all the parsed ISO values and
even the XML tree in case you want to extract something not parsed by
default:


import ckan.plugins as p
from ckanext.spatial.interfaces import ISpatialHarvester

class MyPlugin(p.SingletonPlugin):

    p.implements(ISpatialHarvester, inherit=True)

    def get_package_dict(self, context, data_dict):

        # Check the documentation below to see all that's included on data_dict

        package_dict = data_dict['package_dict']
        iso_values = data_dict['iso_values']

        package_dict['extras'].append(
            {'key': 'topic-category', 'value': iso_values.get('topic-category')}
        )

        package_dict['extras'].append(
            {'key': 'my-custom-extra', 'value': 'my-custom-value'}
        )

        return package_dict

Check the documentation for the full interface reference and more examples:

http://docs.ckan.org/projects/ckanext-spatial/en/latest/harvesters.html#customizing-the-harvesters

There is also a method available for adding your own validators for
the harvested documents cleanly from your own extension, with helper
classes for XSD or schematron based validation, as well as the ability
to write your own:

http://docs.ckan.org/projects/ckanext-spatial/en/latest/harvesters.html#writing-custom-validators

Here is complete real-life example of the interface in action, which
shows how to add new fields to the dataset dict, extract extra fields
from the ISO doc, perform actions based on the document contents, etc:

https://github.com/okfn/ckanext-tsbsatellites/blob/master/ckanext/tsbsatellites/plugin.py#L103:L231


To new ISpatialHarvester interface is available on the master and
stable branch of ckanext-spatial.

Let us know if you have any feedback or find any issues,

Adrià



[1] http://docs.ckan.org/projects/ckanext-spatial/en/latest/harvesters.html