[ckan-dev] Fwd: Issues using Geo plugins

Tue Aug 13 10:43:08 UTC 2013

David,

I'm currently working with 2.2a (latest) version of CKAN. Is there a way to
import just the validators from DGU?
I'm sorry but I'm new to most of this and I'm learning as I go...

2013/8/13 David Read <david.read at hackneyworkshop.com>

> Tiago,
>
> Great to hear that helped.
>
> The data.gov.uk harvester is currently set to:
>
> ckan.spatial.validator.profiles = iso19139eden,constraints-1.4,gemini2-1.3
> ckan.spatial.validator.reject = true
>
> If you're not on the dgu branch of ckanext-spatial already then you'll
> need to switch to it to get these validators and the second option.
>
> However that still gives us 48 validation errors for spatialni.gov.uk
> data. This is not uncommon - producing XML that is valid Gemini2
> appears to be tripping up a lot of publishers. UKLP makes some effort
> to help, but at the end of the day it is NI's responsibility.
>
> It sounds like the datasets you see on data.gov.uk were harvested
> before we switched on the 'rejection', which was April. Previous to
> this, datasets were still harvested into CKAN even if they didn't
> fully validate. You can do this with this config:
>
> ckan.spatial.validator.reject = false
>
> David
>
> On 13 August 2013 10:32, Tiago Ribeiro <tribeiro.tb at gmail.com> wrote:
> > Thank you all for your help. I have finally the harvester job running...
> but
> > now I'm having another problem.
> >
> > I'm harvesting the catalog from an INSPIRE portal, namely this one:
> >
> http://www.spatialni.gov.uk/geoportal/csw/discovery?Request=GetCapabilities&Service=CSW
> >
> > I've tried to set the validator like this
> "ckan.spatial.validator.profiles =
> > inspire" in the config file but it doesn't work (KeyError: 'INSPIRE'),
> and
> > using "ckan.spatial.validator.profiles = iso19139,gemini2,constraints" it
> > gives validation error when reading the items...
> >
> > The data.gov.uk portal have harvested from the same source so I was
> > wondering if there's any other profile validator that I should use?
> >
> > Thanks!
> >
> > Tiago
> >
> > 2013/8/12 David Read <david.read at hackneyworkshop.com>
> >>
> >> You need to create a Harvest Job for the particular Harvest Source
> >> that you want to harvest from. You can do this with paster or in the
> >> web interface.
> >>
> >> David
> >>
> >> On 12 August 2013 16:22, Tiago Ribeiro <tribeiro.tb at gmail.com> wrote:
> >> > Thanks Ryan.
> >> > I'm doing exactly what this links says:
> >> > https://github.com/okfn/ckanext-harvest#running-the-harvest-jobs, but
> >> > when I
> >> > execute the run command on the third window it throws an exception:
> >> > Exception: There are no new harvesting jobs.
> >> >
> >> > In one console I have:
> >> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> >> > --plugin=ckanext-harvest harvester gather_consumer
> >> > --config=production.ini
> >> > 2013-08-12 15:18:29,140 DEBUG [ckanext.harvest.model] Harvest tables
> >> > defined
> >> > in memory
> >> > 2013-08-12 15:18:29,143 DEBUG [ckanext.harvest.model] Harvest tables
> >> > already
> >> > exist
> >> > 2013-08-12 15:18:29,182 DEBUG [ckanext.harvest.queue] pika connection
> >> > using
> >> > {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale':
> >> > 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> >> > 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> >> > <pika.credentials.PlainCredentials object at 0x40a6f10>,
> >> > 'backpressure_detection': False, 'port': 5672, 'connection_attempts':
> 1}
> >> > 2013-08-12 15:18:30,229 DEBUG [ckanext.harvest.queue] Gather queue
> >> > consumer
> >> > registered
> >> >
> >> > Then in other console:
> >> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> >> > --plugin=ckanext-harvest harvester fetch_consumer
> >> > --config=production.ini
> >> > 2013-08-12 15:19:16,846 DEBUG [ckanext.harvest.model] Harvest tables
> >> > defined
> >> > in memory
> >> > 2013-08-12 15:19:16,850 DEBUG [ckanext.harvest.model] Harvest tables
> >> > already
> >> > exist
> >> > 2013-08-12 15:19:16,892 DEBUG [ckanext.harvest.queue] pika connection
> >> > using
> >> > {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale':
> >> > 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> >> > 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> >> > <pika.credentials.PlainCredentials object at 0x5e26c50>,
> >> > 'backpressure_detection': False, 'port': 5672, 'connection_attempts':
> 1}
> >> > 2013-08-12 15:19:17,938 DEBUG [ckanext.harvest.queue] Fetch queue
> >> > consumer
> >> > registered
> >> >
> >> > Finally in other console:
> >> > (default)ubuntu at ip-172-31-31-141:/etc/ckan/default$ paster
> >> > --plugin=ckanext-harvest harvester run --config=production.ini
> >> > 2013-08-12
> >> > 15:19:34,221 DEBUG [ckanext.harvest.model] Harvest tables defined in
> >> > memory
> >> > 2013-08-12 15:19:34,225 DEBUG [ckanext.harvest.model] Harvest tables
> >> > already
> >> > exist
> >> > 2013-08-12 15:19:34,264 INFO  [ckanext.harvest.logic.action.update]
> >> > Harvest
> >> > job run: {}
> >> > 2013-08-12 15:19:34,284 INFO  [ckanext.harvest.logic.action.update] No
> >> > new
> >> > harvest jobs.
> >> > Traceback (most recent call last):
> >> >   File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
> >> >     load_entry_point('PasteScript==1.7.5', 'console_scripts',
> >> > 'paster')()
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> > line 104, in run
> >> >     invoke(command, command_name, options, args[1:])
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> > line 143, in invoke
> >> >     exit_code = runner.run(args)
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> > line 238, in run
> >> >     result = self.command()
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> >> > line 114, in command
> >> >     self.run_harvester()
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> >> > line 265, in run_harvester
> >> >     jobs = get_action('harvest_jobs_run')(context,{})
> >> >   File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line
> >> > 356, in
> >> > wrapped
> >> >     result = _action(context, data_dict, **kw)
> >> >   File
> >> >
> >> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/logic/action/update.py",
> >> > line 332, in harvest_jobs_run
> >> >     raise Exception('There are no new harvesting jobs')
> >> > Exception: There are no new harvesting jobs
> >> >
> >> > Am I missing something?
> >> >
> >> >
> >> > 2013/8/12 Ryan Maine <balrogmi at msn.com>
> >> >>
> >> >> This is an example from the USA catalog
> >> >>
> >> >> http://catalog.data.gov/harvest/about/aasg-geothermal-data-csw
> >> >>
> >> >> There you can find more
> >> >>
> >> >> El 12/08/2013 15:57, "Tiago Ribeiro" <tribeiro.tb at gmail.com>
> escribió:
> >> >>
> >> >>> Thanks David, that did the trick!
> >> >>>
> >> >>> Does have/Can anyone provide me an example of a configuration object
> >> >>> for
> >> >>> a CSW harvester?
> >> >>>
> >> >>> Cheers,
> >> >>> Tiago
> >> >>>
> >> >>> 2013/8/12 David Read <david.read at hackneyworkshop.com>
> >> >>>>
> >> >>>> Tiago,
> >> >>>>
> >> >>>> The README is wrong on this one. Try this:
> >> >>>>
> >> >>>> ckan.harvest.mq.type = ampq
> >> >>>>
> >> >>>> (And yes, this is AMQP with a typo)
> >> >>>>
> >> >>>> David
> >> >>>>
> >> >>>> On 12 August 2013 10:24, Tiago Ribeiro <tribeiro.tb at gmail.com>
> wrote:
> >> >>>> > Err... thank you for both your answers but now it's working...
> >> >>>> > don't
> >> >>>> > know
> >> >>>> > why!
> >> >>>> > Just for reference I'm using CKAN 2.2a (latest).
> >> >>>> >
> >> >>>> > Now I have a different problem. When I run the command line to
> run
> >> >>>> > the
> >> >>>> > Harvest jobs I get this message:
> >> >>>> > Traceback (most recent call last):
> >> >>>> >   File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
> >> >>>> >     load_entry_point('PasteScript==1.7.5', 'console_scripts',
> >> >>>> > 'paster')()
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> >>>> > line 104, in run
> >> >>>> >     invoke(command, command_name, options, args[1:])
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> >>>> > line 143, in invoke
> >> >>>> >     exit_code = runner.run(args)
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> >> >>>> > line 238, in run
> >> >>>> >     result = self.command()
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> >> >>>> > line 126, in command
> >> >>>> >     consumer = get_fetch_consumer()
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >> >>>> > line 340, in get_fetch_consumer
> >> >>>> >     consumer =
> >> >>>> > get_consumer('ckan.harvest.fetch','harvest_object_id')
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >> >>>> > line 174, in get_consumer
> >> >>>> >     connection = get_connection()
> >> >>>> >   File
> >> >>>> >
> >> >>>> >
> "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py",
> >> >>>> > line 39, in get_connection
> >> >>>> >     raise Exception('not a valid queue type %s' % backend)
> >> >>>> > Exception: not a valid queue type rabbitmq
> >> >>>> >
> >> >>>> > In my CKAN config file I have:
> >> >>>> > ckan.plugins = stats text_preview recline_preview datastore
> >> >>>> > ckan.plugins = harvest ckan_harvester csw_harvester
> >> >>>> > ckan.harvest.mq.type = rabbitmq
> >> >>>> >
> >> >>>> > I've installed RabbitMQ as the backend.
> >> >>>> >
> >> >>>> > NB: I'm pretty new with CKAN, so I'm sorry if I've missed
> something
> >> >>>> > "basic"!
> >> >>>> >
> >> >>>> > Thanks!
> >> >>>> >
> >> >>>> > Tiago
> >> >>>> >
> >> >>>> >
> >> >>>> > 2013/8/8 Ryan Maine <balrogmi at msn.com>
> >> >>>> >>
> >> >>>> >> Are you sure you installed harvester extension before?
> >> >>>> >>
> >> >>>> >> El 08/08/2013 17:59, "Adrià Mercader" <adria.mercader at okfn.org>
> >> >>>> >> escribió:
> >> >>>> >>
> >> >>>> >>> Hi Tiago,
> >> >>>> >>>
> >> >>>> >>> This is bizarre, I can not reproduce it so we will need more
> >> >>>> >>> details.
> >> >>>> >>> Is there any message or stacktrace available (eg what is not
> >> >>>> >>> found)?
> >> >>>> >>> Does this happen when you navigate to
> >> >>>> >>> http://yourinstance/harvest?
> >> >>>> >>>
> >> >>>> >>> What versions/branches of ckan/spatial/harvest are you using?
> >> >>>> >>>
> >> >>>> >>> Adrià
> >> >>>> >>>
> >> >>>> >>> On 7 August 2013 17:20, Tiago Ribeiro <tribeiro.tb at gmail.com>
> >> >>>> >>> wrote:
> >> >>>> >>> > Hi all,
> >> >>>> >>> >
> >> >>>> >>> > I'm having an issues with spatial harvester plugin. As soon
> as
> >> >>>> >>> > I
> >> >>>> >>> > add
> >> >>>> >>> > the
> >> >>>> >>> > "csw_harvester" to ckan.plugins the and navigate to the
> harvest
> >> >>>> >>> > domain
> >> >>>> >>> > I get
> >> >>>> >>> > a 404.
> >> >>>> >>> > ckan.plugins = harvest ckan_harvester - works
> >> >>>> >>> > ckan.plugins = harvest ckan_harvester csw_harvester - get a
> >> >>>> >>> > 404.
> >> >>>> >>> >
> >> >>>> >>> > Am I doing something wrong?
> >> >>>> >>> > I have CKAN in a Ubuntu 12.04 Amazon box. I've used the
> >> >>>> >>> > "Install
> >> >>>> >>> > from
> >> >>>> >>> > source" guide.
> >> >>>> >>> >
> >> >>>> >>> > If you could help me I would be much appreciated. If you need
> >> >>>> >>> > more
> >> >>>> >>> > details
> >> >>>> >>> > let me know.
> >> >>>> >>> >
> >> >>>> >>> > Tiago Ribeiro
> >> >>>> >>> >
> >> >>>> >>> >
> >> >>>> >>> > _______________________________________________
> >> >>>> >>> > ckan-dev mailing list
> >> >>>> >>> > ckan-dev at lists.okfn.org
> >> >>>> >>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> >>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>> >>> >
> >> >>>> >>>
> >> >>>> >>> _______________________________________________
> >> >>>> >>> ckan-dev mailing list
> >> >>>> >>> ckan-dev at lists.okfn.org
> >> >>>> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>> >>>
> >> >>>> >>
> >> >>>> >> _______________________________________________
> >> >>>> >> ckan-dev mailing list
> >> >>>> >> ckan-dev at lists.okfn.org
> >> >>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>> >>
> >> >>>> >
> >> >>>> >
> >> >>>> > _______________________________________________
> >> >>>> > ckan-dev mailing list
> >> >>>> > ckan-dev at lists.okfn.org
> >> >>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>> >
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> ckan-dev mailing list
> >> >>>> ckan-dev at lists.okfn.org
> >> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> ckan-dev mailing list
> >> >>> ckan-dev at lists.okfn.org
> >> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>>
> >> >>
> >> >> _______________________________________________
> >> >> ckan-dev mailing list
> >> >> ckan-dev at lists.okfn.org
> >> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > ckan-dev mailing list
> >> > ckan-dev at lists.okfn.org
> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >> >
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130813/7b4d5e80/attachment-0001.html>