[ckan-dev] [CKAN][HARVESTER] Solr issue

Sun Oct 13 17:02:22 UTC 2013

hello didier,

does the harvesting makes problem when run under normal circumstances? - if
you run the three paster commands from shell?

there may also be problems with your proxy settings of your testsystem -
maybe you have no route to the host.

i tested the harvesting features for csw sources in spring and went to an
easier solution:

generate the json objects for ckan directly from your metadata and push
them into ckan thru its rest api - this way is smarter i think ;-) . it
will also be the better way for managing iso19139 metadata via csw - use
the transaction interfaces from the bottom to the toplevel catalogue (maybe
via trigger like functions). when the metadata changes, the records should
directly be updated in the upper level catalogue and from there propagated
to the next level. then we have always actual metadata!

here is our ckan-json port of our (geo)metadata catalogue:
http://www.geoportal.rlp.de/mapbender/php/mod_exportMapbenderLayer2CkanObjects.php

i had a small presentation about managing geo-metadata in ckan instances
this year on the fossgis conference. it is in german but maybe google
translator helps ;-) .

http://www.fossgis.de/konferenz/2013/programm/events/622.de.html
http://www.fossgis.de/w/images/e/e4/Fossgis2013_geometadaten_in_opendata_portalen_retterath.pdf

regards from germany and have a nice evening

armin

2013/10/13 Didier Richard <Didier.Richard at ign.fr>

> Hi All,
>
> I'm having a strange behavior when testing Supervisor during the
> ckanext-harvest installation (See
> https://github.com/okfn/ckanext-harvest#setting-up-the-harvesters-on-a-production-server
> ).
>
> When launching :
>
> $ sudo supervisorctl start ckan_gather_consumer
> ckan_gather_consumer: ERROR (abnormal termination)
>
> The log shows :
>
> Traceback (most recent call last):
>   File "/opt/ckan/default/bin/paster", line 9, in <module>
>     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
>   File
> "/opt/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 104, in run
>     invoke(command, command_name, options, args[1:])
>   File
> "/opt/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 143, in invoke
>     exit_code = runner.run(args)
>   File
> "/opt/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 238, in run
>     result = self.command()
>   File
> "/opt/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> line 88, in command
>     self._load_config()
>   File
> "/opt/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> line 147, in _load_config
>     super(Harvester, self)._load_config()
>   File "/opt/ckan/default/src/ckan/ckan/lib/cli.py", line 94, in
> _load_config
>     load_environment(conf.global_conf, conf.local_conf)
>   File "/opt/ckan/default/src/ckan/ckan/config/environment.py", line 232,
> in load_environment
>     p.load_all(config)
>   File "/opt/ckan/default/src/ckan/ckan/plugins/core.py", line 124, in
> load_all
>     unload_all()
>   File "/opt/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in
> unload_all
>     unload(*reversed(_PLUGINS))
>   File "/opt/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in
> unload
>     plugins_update()
>   File "/opt/ckan/default/src/ckan/ckan/plugins/core.py", line 116, in
> plugins_update
>     environment.update_config()
>   File "/opt/ckan/default/src/ckan/ckan/config/environment.py", line 270,
> in update_config
>     search.check_solr_schema_version()
>   File "/opt/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 291,
> in check_solr_schema_version
>     res = urllib2.urlopen(req)
>   File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
>     return _opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 400, in open
>     response = self._open(req, data)
>   File "/usr/lib/python2.7/urllib2.py", line 418, in _open
>     '_open', req)
>   File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
>     return self.do_open(httplib.HTTPConnection, req)
>   File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
>     raise URLError(err)
> urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
>
> The solr url is set as follows :
>
> solr_url = http://localhost:8080/solr/
>
> (solr runs under tomcat6)
>
> The CKAN admin url est working :
> http://localhost:8080/solr/admin/file/?file=schema.xml (I get the
> schema-2.0.xml)
>
> When running by hand the following command :
>
> $ /opt/ckan/default/bin/paster --plugin=ckanext-harvest harvester
> gather_consumer --config=/etc/ckan/default/production.ini
>
> everything's working without error ...
>
> Do you have any clue ?
>
> didier
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131013/e3ff2ae2/attachment-0001.html>