[ckan-dev] crash while harvesting

Fri Mar 14 12:37:52 UTC 2014

FYI
This is now fixed in ckanext-spatial

https://github.com/ckan/ckanext-spatial/issues/64


Adrià

On 10 January 2014 11:37, Hildegard Gerlach
<hildegard.gerlach at jrc.ec.europa.eu> wrote:
> Dear all,
>
> I get the following error harvesting from a GeoNetwork instance using
>
> ckan.spatial.validator.profiles = iso19139eden and
> ckanext.spatial.harvest.continue_on_validation_errors = true
>
> Traceback (most recent call last):
>   File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
>     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
>   File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 104, in run
>     invoke(command, command_name, options, args[1:])
>   File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 143, in invoke
>     exit_code = runner.run(args)
>   File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 238, in run
>     result = self.command()
>   File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> line 127, in command
>     fetch_callback(consumer, method, header, body)
>   File "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/queue.py",
> line 294, in fetch_callback
>     fetch_and_import_stages(harvester, obj)
>   File "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/queue.py",
> line 311, in fetch_and_import_stages
>     success_import = harvester.import_stage(obj)
>   File
> "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/harvesters/base.py",
> line 441, in import_stage
>     is_valid, profile, errors =
> self._validate_document(harvest_object.content, harvest_object)
>   File
> "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/harvesters/base.py",
> line 726, in _validate_document
>     valid, profile, errors = validator.is_valid(xml)
>   File
> "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py",
> line 335, in is_valid
>     is_valid, error_message_list = validator.is_valid(xml)
>   File
> "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py",
> line 87, in is_valid
>     metadata_type = cls.get_record_type(xml)
>   File
> "/usr/local/ckan/pyenv/src/ckanext-spatial/ckanext/spatial/validation/validation.py",
> line 123, in get_record_type
>     return iso_parser.read_value('resource-type')[0]
> IndexError: list index out of range
>
> The fetch_consumer stage crashes and the harvesting job doesn't finish, it
> is always running (46 datasets found)
>
> If I put ckan.spatial.validator.profiles = iso19139ngdc
> the harvesting job finishes and finds 57 datasets
>
> seems to me like a bug
>
>
> Another question: Is there a way to harvest only datasets corresponding to a
> certain criteria ?
> I cannot see anything similar for the configuration field defining the
> harvester.
>
> Thanks a lot
>
> Hilde
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev