[ckan-dev] Fwd: Issues using Geo plugins
Tiago Ribeiro
tribeiro.tb at gmail.com
Tue Aug 13 09:32:48 UTC 2013
Thank you all for your help. I have finally the harvester job running...
but now I'm having another problem.
I'm harvesting the catalog from an INSPIRE portal, namely this one:
http://www.spatialni.gov.uk/geoportal/csw/discovery?Request=GetCapabilities&Service=CSW
I've tried to set the validator like this "ckan.spatial.validator.profiles
= inspire" in the config file but it doesn't work (KeyError: 'INSPIRE'),
and using "ckan.spatial.validator.profiles = iso19139,gemini2,constraints"
it gives validation error when reading the items...
The data.gov.uk portal have harvested from the same source so I was
wondering if there's any other profile validator that I should use?
Thanks!
Tiago
2013/8/12 David Read <david.read at hackneyworkshop.com>
> You need to create a Harvest Job for the particular Harvest Source
> that you want to harvest from. You can do this with paster or in the
> web interface.
>
> David
>
> On 12 August 2013 16:22, Tiago Ribeiro <tribeiro.tb at gmail.com> wrote:
> > Thanks Ryan.
> > I'm doing exactly what this links says:
> > https://github.com/okfn/ckanext-harvest#running-the-harvest-jobs, but
> when I
> > execute the run command on the third window it throws an exception:
> > Exception: There are no new harvesting jobs.
> >
> > In one console I have:
> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> > --plugin=ckanext-harvest harvester gather_consumer
> --config=production.ini
> > 2013-08-12 15:18:29,140 DEBUG [ckanext.harvest.model] Harvest tables
> defined
> > in memory
> > 2013-08-12 15:18:29,143 DEBUG [ckanext.harvest.model] Harvest tables
> already
> > exist
> > 2013-08-12 15:18:29,182 DEBUG [ckanext.harvest.queue] pika connection
> using
> > {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale':
> > 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> > 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> > <pika.credentials.PlainCredentials object at 0x40a6f10>,
> > 'backpressure_detection': False, 'port': 5672, 'connection_attempts': 1}
> > 2013-08-12 15:18:30,229 DEBUG [ckanext.harvest.queue] Gather queue
> consumer
> > registered
> >
> > Then in other console:
> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> > --plugin=ckanext-harvest harvester fetch_consumer --config=production.ini
> > 2013-08-12 15:19:16,846 DEBUG [ckanext.harvest.model] Harvest tables
> defined
> > in memory
> > 2013-08-12 15:19:16,850 DEBUG [ckanext.harvest.model] Harvest tables
> already
> > exist
> > 2013-08-12 15:19:16,892 DEBUG [ckanext.harvest.queue] pika connection
> using
> > {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale':
> > 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> > 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> > <pika.credentials.PlainCredentials object at 0x5e26c50>,
> > 'backpressure_detection': False, 'port': 5672, 'connection_attempts': 1}
> > 2013-08-12 15:19:17,938 DEBUG [ckanext.harvest.queue] Fetch queue
> consumer
> > registered
> >
> > Finally in other console:
> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> > --plugin=ckanext-harvest harvester run --config=production.ini 2013-08-12
> > 15:19:34,221 DEBUG [ckanext.harvest.model] Harvest tables defined in
> memory
> > 2013-08-12 15:19:34,225 DEBUG [ckanext.harvest.model] Harvest tables
> already
> > exist
> > 2013-08-12 15:19:34,264 INFO [ckanext.harvest.logic.action.update]
> Harvest
> > job run: {}
> > 2013-08-12 15:19:34,284 INFO [ckanext.harvest.logic.action.update] No
> new
> > harvest jobs.
> > Traceback (most recent call last):
> > File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
> > load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
> > File
> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> > line 104, in run
> > invoke(command, command_name, options, args[1:])
> > File
> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> > line 143, in invoke
> > exit_code = runner.run(args)
> > File
> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> > line 238, in run
> > result = self.command()
> > File
> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> > line 114, in command
> > self.run_harvester()
> > File
> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> > line 265, in run_harvester
> > jobs = get_action('harvest_jobs_run')(context,{})
> > File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line
> 356, in
> > wrapped
> > result = _action(context, data_dict, **kw)
> > File
> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/logic/action/update.py",
> > line 332, in harvest_jobs_run
> > raise Exception('There are no new harvesting jobs')
> > Exception: There are no new harvesting jobs
> >
> > Am I missing something?
> >
> >
> > 2013/8/12 Ryan Maine <balrogmi at msn.com>
> >>
> >> This is an example from the USA catalog
> >>
> >> http://catalog.data.gov/harvest/about/aasg-geothermal-data-csw
> >>
> >> There you can find more
> >>
> >> El 12/08/2013 15:57, "Tiago Ribeiro" <tribeiro.tb at gmail.com> escribió:
> >>
> >>> Thanks David, that did the trick!
> >>>
> >>> Does have/Can anyone provide me an example of a configuration object
> for
> >>> a CSW harvester?
> >>>
> >>> Cheers,
> >>> Tiago
> >>>
> >>> 2013/8/12 David Read <david.read at hackneyworkshop.com>
> >>>>
> >>>> Tiago,
> >>>>
> >>>> The README is wrong on this one. Try this:
> >>>>
> >>>> ckan.harvest.mq.type = ampq
> >>>>
> >>>> (And yes, this is AMQP with a typo)
> >>>>
> >>>> David
> >>>>
> >>>> On 12 August 2013 10:24, Tiago Ribeiro <tribeiro.tb at gmail.com> wrote:
> >>>> > Err... thank you for both your answers but now it's working... don't
> >>>> > know
> >>>> > why!
> >>>> > Just for reference I'm using CKAN 2.2a (latest).
> >>>> >
> >>>> > Now I have a different problem. When I run the command line to run
> the
> >>>> > Harvest jobs I get this message:
> >>>> > Traceback (most recent call last):
> >>>> > File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
> >>>> > load_entry_point('PasteScript==1.7.5', 'console_scripts',
> >>>> > 'paster')()
> >>>> > File
> >>>> >
> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >>>> > line 104, in run
> >>>> > invoke(command, command_name, options, args[1:])
> >>>> > File
> >>>> >
> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >>>> > line 143, in invoke
> >>>> > exit_code = runner.run(args)
> >>>> > File
> >>>> >
> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >>>> > line 238, in run
> >>>> > result = self.command()
> >>>> > File
> >>>> >
> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> >>>> > line 126, in command
> >>>> > consumer = get_fetch_consumer()
> >>>> > File
> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >>>> > line 340, in get_fetch_consumer
> >>>> > consumer =
> get_consumer('ckan.harvest.fetch','harvest_object_id')
> >>>> > File
> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >>>> > line 174, in get_consumer
> >>>> > connection = get_connection()
> >>>> > File
> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >>>> > line 39, in get_connection
> >>>> > raise Exception('not a valid queue type %s' % backend)
> >>>> > Exception: not a valid queue type rabbitmq
> >>>> >
> >>>> > In my CKAN config file I have:
> >>>> > ckan.plugins = stats text_preview recline_preview datastore
> >>>> > ckan.plugins = harvest ckan_harvester csw_harvester
> >>>> > ckan.harvest.mq.type = rabbitmq
> >>>> >
> >>>> > I've installed RabbitMQ as the backend.
> >>>> >
> >>>> > NB: I'm pretty new with CKAN, so I'm sorry if I've missed something
> >>>> > "basic"!
> >>>> >
> >>>> > Thanks!
> >>>> >
> >>>> > Tiago
> >>>> >
> >>>> >
> >>>> > 2013/8/8 Ryan Maine <balrogmi at msn.com>
> >>>> >>
> >>>> >> Are you sure you installed harvester extension before?
> >>>> >>
> >>>> >> El 08/08/2013 17:59, "Adrià Mercader" <adria.mercader at okfn.org>
> >>>> >> escribió:
> >>>> >>
> >>>> >>> Hi Tiago,
> >>>> >>>
> >>>> >>> This is bizarre, I can not reproduce it so we will need more
> >>>> >>> details.
> >>>> >>> Is there any message or stacktrace available (eg what is not
> found)?
> >>>> >>> Does this happen when you navigate to http://yourinstance/harvest
> ?
> >>>> >>>
> >>>> >>> What versions/branches of ckan/spatial/harvest are you using?
> >>>> >>>
> >>>> >>> Adrià
> >>>> >>>
> >>>> >>> On 7 August 2013 17:20, Tiago Ribeiro <tribeiro.tb at gmail.com>
> wrote:
> >>>> >>> > Hi all,
> >>>> >>> >
> >>>> >>> > I'm having an issues with spatial harvester plugin. As soon as I
> >>>> >>> > add
> >>>> >>> > the
> >>>> >>> > "csw_harvester" to ckan.plugins the and navigate to the harvest
> >>>> >>> > domain
> >>>> >>> > I get
> >>>> >>> > a 404.
> >>>> >>> > ckan.plugins = harvest ckan_harvester - works
> >>>> >>> > ckan.plugins = harvest ckan_harvester csw_harvester - get a 404.
> >>>> >>> >
> >>>> >>> > Am I doing something wrong?
> >>>> >>> > I have CKAN in a Ubuntu 12.04 Amazon box. I've used the "Install
> >>>> >>> > from
> >>>> >>> > source" guide.
> >>>> >>> >
> >>>> >>> > If you could help me I would be much appreciated. If you need
> more
> >>>> >>> > details
> >>>> >>> > let me know.
> >>>> >>> >
> >>>> >>> > Tiago Ribeiro
> >>>> >>> >
> >>>> >>> >
> >>>> >>> > _______________________________________________
> >>>> >>> > ckan-dev mailing list
> >>>> >>> > ckan-dev at lists.okfn.org
> >>>> >>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >>> >
> >>>> >>>
> >>>> >>> _______________________________________________
> >>>> >>> ckan-dev mailing list
> >>>> >>> ckan-dev at lists.okfn.org
> >>>> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >>>
> >>>> >>
> >>>> >> _______________________________________________
> >>>> >> ckan-dev mailing list
> >>>> >> ckan-dev at lists.okfn.org
> >>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >>
> >>>> >
> >>>> >
> >>>> > _______________________________________________
> >>>> > ckan-dev mailing list
> >>>> > ckan-dev at lists.okfn.org
> >>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >
> >>>>
> >>>> _______________________________________________
> >>>> ckan-dev mailing list
> >>>> ckan-dev at lists.okfn.org
> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130813/5cbfe2a4/attachment-0001.html>
More information about the ckan-dev
mailing list