[ckan-dev] Harvestor extension for CKAN
Toby Dacre
toby.okfn at gmail.com
Tue Jan 8 09:29:57 UTC 2013
On 7 January 2013 14:45, Adrià Mercader <adria.mercader at okfn.org> wrote:
> Thanks Lasse,
>
> These logs are really helpful, I'll need to investigate a bit more.
>
>
It looks like we are trying to create a qualified url ie the entire url
http://host/path it looks like routes is looking in the config .ini file
and we need to supply this information, for example in the ini template we
have this
[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5000
Adrià
>
> On 7 January 2013 14:11, Lasse Vestergaard <ibbernik at gmail.com> wrote:
> > Hi again again :-)
> >
> > I can see that I am able to harvest datasets without resources attached.
> > Furthermore, I have tried to make a clean install of CKAN and I still get
> > the same result: AttributeError: 'thread._local' object has no attribute
> > 'host'
> >
> > /Lasse
> >
> >
> > 2013/1/7 Lasse Vestergaard <ibbernik at gmail.com>
> >>
> >> Hi again.
> >>
> >> I have been looking in the log files, and I got the following:
> >>
> >> 2013-01-04 15:45:06,336 INFO [ckanext.harvest.queue] Received harvest
> >> object id: 8741a64e-1673-4f29-a713-f9a41d8c9732
> >> 2013-01-04 15:45:06,348 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> >> In CKANHarvester fetch_stage
> >> 2013-01-04 15:45:06,353 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
> >> u'api_version': u'1'}
> >> 2013-01-04 15:45:06,577 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> >> In CKANHarvester import_stage
> >> 2013-01-04 15:45:06,579 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
> >> u'api_version': u'1'}
> >> 2013-01-04 15:45:06,600 INFO [ckanext.harvest.harvesters.base] Package
> >> with GUID test does not exist, let's create it
> >> 2013-01-04 15:45:06,852 ERROR [ckan.model.modification] 'thread._local'
> >> object has no attribute 'host'
> >> Traceback (most recent call last):
> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
> 68,
> >> in notify
> >> observer.notify(entity, operation)
> >> File
> >>
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> >> line 30, in notify
> >> self._create_datastorer_task(entity)
> >> File
> >>
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> >> line 42, in _create_datastorer_task
> >> 'site_url': h.url_for_static('/', qualified=True),
> >> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
> >> url_for_static
> >> my_url = _routes_default_url_for(*args, **kw)
> >> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
> >> url_for
> >> host = config.host
> >> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14,
> in
> >> __getattr__
> >> return getattr(self.__shared_state, name)
> >> AttributeError: 'thread._local' object has no attribute 'host'
> >> 2013-01-04 15:45:06,854 ERROR [ckanext.harvest.harvesters.base]
> >> 'thread._local' object has no attribute 'host'
> >> Traceback (most recent call last):
> >> File "/home/ckan1/ckanext-harvest/ckanext/harvest/harvesters/base.py",
> >> line 165, in _create_or_update_package
> >> new_package = get_action('package_create_rest')(context,
> package_dict)
> >> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
> >> 723, in package_create_rest
> >> dictized_after = _get_action('package_create')(context,
> >> dictized_package)
> >> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
> >> 159, in package_create
> >> model.repo.commit()
> >> File "/usr/lib/pymodules/python2.6/vdm/sqlalchemy/tools.py", line 102,
> >> in commit
> >> self.session.commit()
> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/scoping.py",
> line
> >> 114, in do
> >> return getattr(self.registry(), name)(*args, **kwargs)
> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
> line
> >> 656, in commit
> >> self.transaction.commit()
> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
> line
> >> 314, in commit
> >> self._prepare_impl()
> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
> line
> >> 290, in _prepare_impl
> >> self.session.dispatch.before_commit(self.session)
> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/event.py", line
> 291,
> >> in __call__
> >> fn(*args, **kw)
> >> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 103,
> >> in before_commit
> >> methodcaller('before_commit', session)
> >> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 38,
> in
> >> notify_observers
> >> func(observer)
> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
> 37,
> >> in before_commit
> >> self.notify(obj, domain_object.DomainObjectOperation.new)
> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
> 68,
> >> in notify
> >> observer.notify(entity, operation)
> >> File
> >>
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> >> line 30, in notify
> >> self._create_datastorer_task(entity)
> >> File
> >>
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> >> line 42, in _create_datastorer_task
> >> 'site_url': h.url_for_static('/', qualified=True),
> >> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
> >> url_for_static
> >> my_url = _routes_default_url_for(*args, **kw)
> >> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
> >> url_for
> >> host = config.host
> >> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14,
> in
> >> __getattr__
> >> return getattr(self.__shared_state, name)
> >> AttributeError: 'thread._local' object has no attribute 'host'
> >> 2013-01-04 15:45:06,881 ERROR [ckanext.harvest.harvesters.base]
> >> AttributeError("'thread._local' object has no attribute 'host'",)
> >>
> >> Does this make sense to anyone :-)
> >>
> >> /Lasse
> >>
> >>
> >> 2013/1/3 Lasse Vestergaard <ibbernik at gmail.com>
> >>>
> >>> Hi Adriá Mercader.
> >>>
> >>> My version of CKAN is 1.8 and the ckanext-harvest is the one I get
> when I
> >>> follow the tutorial at this page:
> https://github.com/okfn/ckanext-harvest. I
> >>> cloned the repository today (3. january).
> >>>
> >>> /Lasse
> >>>
> >>>
> >>> 2013/1/3 Adrià Mercader <adria.mercader at okfn.org>
> >>>>
> >>>> Hi Lesse,
> >>>>
> >>>> You are right, you don't need to specify the json config option unless
> >>>> you need it.
> >>>> What versions of CKAN core and ckanext-harvest are you using?
> >>>>
> >>>>
> >>>> Adrià
> >>>>
> >>>> On 3 January 2013 14:49, Lasse Vestergaard <ibbernik at gmail.com>
> wrote:
> >>>> > Hej Peder :-)
> >>>> >
> >>>> > I wrote this in the configuration field:
> >>>> >
> >>>> > {
> >>>> > "api_version":"1",
> >>>> > "default_groups":["mygroup"],
> >>>> > "user":"my-username"
> >>>> > }
> >>>> >
> >>>> > The group exists and I can easily create the harvest source. The
> error
> >>>> > occurs when the harvest process runs. As I read the documentation, I
> >>>> > don't
> >>>> > need to put in all configuration options. Is that correct?
> >>>> >
> >>>> > /Lasse
> >>>> >
> >>>> >
> >>>> >
> >>>> > 2013/1/3 Peder Jakobsen | gmail <pjakobsen at gmail.com>
> >>>> >>
> >>>> >> HI Lasse,
> >>>> >>
> >>>> >> Did you enter your configuration details in the form of json in the
> >>>> >> UI?
> >>>> >>
> >>>> >> Venlig hilsen,
> >>>> >>
> >>>> >> Peder :)
> >>>> >>
> >>>> >> On Thu, Jan 3, 2013 at 3:56 AM, Lasse Vestergaard
> >>>> >> <ibbernik at gmail.com>
> >>>> >> wrote:
> >>>> >> > Hi all.
> >>>> >> >
> >>>> >> > I am trying to get the harvestor extension (ckanext-harvest-)
> work
> >>>> >> > for
> >>>> >> > CKAN.
> >>>> >> > All in all it seems like, I have managed to install it. I am
> >>>> >> > currently
> >>>> >> > trying to harvest data form an other CKAN instance (remote
> computer
> >>>> >> > on
> >>>> >> > an
> >>>> >> > other network). I only provided the url to the CKAN frontend:
> >>>> >> > ckan.odaa.dk.
> >>>> >> > When i run the harvesting process I get the following error:
> >>>> >> >
> >>>> >> > Last Harvest Errors: 12
> >>>> >> > Object errors
> >>>> >> >
> >>>> >> > GUID fredede-bygninger-og-fortidsminder
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID blog-p-bibliotekernes-hjemmeside
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID arrangementer-p-bibliotekerne
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID ejendomsfortegnelse
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID vejnavne-og-vejkoder
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID parkering
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID bibliotekstransaktioner
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID beholdning-ved-aarhus-kommunes-biblioteker
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID koordinats-tning-af-aarhus-adresser
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID cvr-produktionsenheder
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID bevaringsv-rdige-bygninger-i-aarhus-kommune
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> > GUID bbr-ejendomme-og-bbr-bygninger
> >>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
> >>>> >> >
> >>>> >> > There are only 12 datasets in the CKAN instance I try to harvest
> >>>> >> > from
> >>>> >> > (it's
> >>>> >> > a test instance). I have tried to Google the error, but I don't
> get
> >>>> >> > any
> >>>> >> > relevant suggestions, so I hope you guys can point me in some
> >>>> >> > direction.
> >>>> >> >
> >>>> >> > Regards
> >>>> >> >
> >>>> >> > Lasse Vestergaard
> >>>> >> >
> >>>> >> > _______________________________________________
> >>>> >> > ckan-dev mailing list
> >>>> >> > ckan-dev at lists.okfn.org
> >>>> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >> >
> >>>> >>
> >>>> >> _______________________________________________
> >>>> >> ckan-dev mailing list
> >>>> >> ckan-dev at lists.okfn.org
> >>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >
> >>>> >
> >>>> >
> >>>> > _______________________________________________
> >>>> > ckan-dev mailing list
> >>>> > ckan-dev at lists.okfn.org
> >>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>> >
> >>>>
> >>>> _______________________________________________
> >>>> ckan-dev mailing list
> >>>> ckan-dev at lists.okfn.org
> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>>
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130108/73ebe89a/attachment-0001.html>
More information about the ckan-dev
mailing list