[ckan-dev] Harvestor extension for CKAN
Lasse Vestergaard
ibbernik at gmail.com
Fri Jan 11 09:57:46 UTC 2013
Hi CKAN developers.
Have you been looking in to the : AttributeError: 'thread._local' object
has no attribute 'host' error?
I'm working on creating a distributed CKAN network, and I want a central
computer to work as an index for the complete network. I have pretty much
setup everything, but I only need to get this "little" error fixed to be
able to make it all work.
Best regards
Lasse Vestergaard
2013/1/8 Toby Dacre <toby.okfn at gmail.com>
>
>
> On 7 January 2013 14:45, Adrià Mercader <adria.mercader at okfn.org> wrote:
>
>> Thanks Lasse,
>>
>> These logs are really helpful, I'll need to investigate a bit more.
>>
>>
> It looks like we are trying to create a qualified url ie the entire url
> http://host/path it looks like routes is looking in the config .ini file
> and we need to supply this information, for example in the ini template we
> have this
>
> [server:main]
> use = egg:Paste#http
> host = 0.0.0.0
> port = 5000
>
>
> Adrià
>>
>> On 7 January 2013 14:11, Lasse Vestergaard <ibbernik at gmail.com> wrote:
>> > Hi again again :-)
>> >
>> > I can see that I am able to harvest datasets without resources attached.
>> > Furthermore, I have tried to make a clean install of CKAN and I still
>> get
>> > the same result: AttributeError: 'thread._local' object has no attribute
>> > 'host'
>> >
>> > /Lasse
>> >
>> >
>> > 2013/1/7 Lasse Vestergaard <ibbernik at gmail.com>
>> >>
>> >> Hi again.
>> >>
>> >> I have been looking in the log files, and I got the following:
>> >>
>> >> 2013-01-04 15:45:06,336 INFO [ckanext.harvest.queue] Received harvest
>> >> object id: 8741a64e-1673-4f29-a713-f9a41d8c9732
>> >> 2013-01-04 15:45:06,348 DEBUG
>> [ckanext.harvest.harvesters.ckanharvester]
>> >> In CKANHarvester fetch_stage
>> >> 2013-01-04 15:45:06,353 DEBUG
>> [ckanext.harvest.harvesters.ckanharvester]
>> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>> >> u'api_version': u'1'}
>> >> 2013-01-04 15:45:06,577 DEBUG
>> [ckanext.harvest.harvesters.ckanharvester]
>> >> In CKANHarvester import_stage
>> >> 2013-01-04 15:45:06,579 DEBUG
>> [ckanext.harvest.harvesters.ckanharvester]
>> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>> >> u'api_version': u'1'}
>> >> 2013-01-04 15:45:06,600 INFO [ckanext.harvest.harvesters.base] Package
>> >> with GUID test does not exist, let's create it
>> >> 2013-01-04 15:45:06,852 ERROR [ckan.model.modification] 'thread._local'
>> >> object has no attribute 'host'
>> >> Traceback (most recent call last):
>> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>> 68,
>> >> in notify
>> >> observer.notify(entity, operation)
>> >> File
>> >>
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> >> line 30, in notify
>> >> self._create_datastorer_task(entity)
>> >> File
>> >>
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> >> line 42, in _create_datastorer_task
>> >> 'site_url': h.url_for_static('/', qualified=True),
>> >> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>> >> url_for_static
>> >> my_url = _routes_default_url_for(*args, **kw)
>> >> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
>> >> url_for
>> >> host = config.host
>> >> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line
>> 14, in
>> >> __getattr__
>> >> return getattr(self.__shared_state, name)
>> >> AttributeError: 'thread._local' object has no attribute 'host'
>> >> 2013-01-04 15:45:06,854 ERROR [ckanext.harvest.harvesters.base]
>> >> 'thread._local' object has no attribute 'host'
>> >> Traceback (most recent call last):
>> >> File
>> "/home/ckan1/ckanext-harvest/ckanext/harvest/harvesters/base.py",
>> >> line 165, in _create_or_update_package
>> >> new_package = get_action('package_create_rest')(context,
>> package_dict)
>> >> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
>> >> 723, in package_create_rest
>> >> dictized_after = _get_action('package_create')(context,
>> >> dictized_package)
>> >> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
>> >> 159, in package_create
>> >> model.repo.commit()
>> >> File "/usr/lib/pymodules/python2.6/vdm/sqlalchemy/tools.py", line
>> 102,
>> >> in commit
>> >> self.session.commit()
>> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/scoping.py",
>> line
>> >> 114, in do
>> >> return getattr(self.registry(), name)(*args, **kwargs)
>> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>> line
>> >> 656, in commit
>> >> self.transaction.commit()
>> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>> line
>> >> 314, in commit
>> >> self._prepare_impl()
>> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>> line
>> >> 290, in _prepare_impl
>> >> self.session.dispatch.before_commit(self.session)
>> >> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/event.py", line
>> 291,
>> >> in __call__
>> >> fn(*args, **kw)
>> >> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line
>> 103,
>> >> in before_commit
>> >> methodcaller('before_commit', session)
>> >> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line
>> 38, in
>> >> notify_observers
>> >> func(observer)
>> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>> 37,
>> >> in before_commit
>> >> self.notify(obj, domain_object.DomainObjectOperation.new)
>> >> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>> 68,
>> >> in notify
>> >> observer.notify(entity, operation)
>> >> File
>> >>
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> >> line 30, in notify
>> >> self._create_datastorer_task(entity)
>> >> File
>> >>
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> >> line 42, in _create_datastorer_task
>> >> 'site_url': h.url_for_static('/', qualified=True),
>> >> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>> >> url_for_static
>> >> my_url = _routes_default_url_for(*args, **kw)
>> >> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
>> >> url_for
>> >> host = config.host
>> >> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line
>> 14, in
>> >> __getattr__
>> >> return getattr(self.__shared_state, name)
>> >> AttributeError: 'thread._local' object has no attribute 'host'
>> >> 2013-01-04 15:45:06,881 ERROR [ckanext.harvest.harvesters.base]
>> >> AttributeError("'thread._local' object has no attribute 'host'",)
>> >>
>> >> Does this make sense to anyone :-)
>> >>
>> >> /Lasse
>> >>
>> >>
>> >> 2013/1/3 Lasse Vestergaard <ibbernik at gmail.com>
>> >>>
>> >>> Hi Adriá Mercader.
>> >>>
>> >>> My version of CKAN is 1.8 and the ckanext-harvest is the one I get
>> when I
>> >>> follow the tutorial at this page:
>> https://github.com/okfn/ckanext-harvest. I
>> >>> cloned the repository today (3. january).
>> >>>
>> >>> /Lasse
>> >>>
>> >>>
>> >>> 2013/1/3 Adrià Mercader <adria.mercader at okfn.org>
>> >>>>
>> >>>> Hi Lesse,
>> >>>>
>> >>>> You are right, you don't need to specify the json config option
>> unless
>> >>>> you need it.
>> >>>> What versions of CKAN core and ckanext-harvest are you using?
>> >>>>
>> >>>>
>> >>>> Adrià
>> >>>>
>> >>>> On 3 January 2013 14:49, Lasse Vestergaard <ibbernik at gmail.com>
>> wrote:
>> >>>> > Hej Peder :-)
>> >>>> >
>> >>>> > I wrote this in the configuration field:
>> >>>> >
>> >>>> > {
>> >>>> > "api_version":"1",
>> >>>> > "default_groups":["mygroup"],
>> >>>> > "user":"my-username"
>> >>>> > }
>> >>>> >
>> >>>> > The group exists and I can easily create the harvest source. The
>> error
>> >>>> > occurs when the harvest process runs. As I read the documentation,
>> I
>> >>>> > don't
>> >>>> > need to put in all configuration options. Is that correct?
>> >>>> >
>> >>>> > /Lasse
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > 2013/1/3 Peder Jakobsen | gmail <pjakobsen at gmail.com>
>> >>>> >>
>> >>>> >> HI Lasse,
>> >>>> >>
>> >>>> >> Did you enter your configuration details in the form of json in
>> the
>> >>>> >> UI?
>> >>>> >>
>> >>>> >> Venlig hilsen,
>> >>>> >>
>> >>>> >> Peder :)
>> >>>> >>
>> >>>> >> On Thu, Jan 3, 2013 at 3:56 AM, Lasse Vestergaard
>> >>>> >> <ibbernik at gmail.com>
>> >>>> >> wrote:
>> >>>> >> > Hi all.
>> >>>> >> >
>> >>>> >> > I am trying to get the harvestor extension (ckanext-harvest-)
>> work
>> >>>> >> > for
>> >>>> >> > CKAN.
>> >>>> >> > All in all it seems like, I have managed to install it. I am
>> >>>> >> > currently
>> >>>> >> > trying to harvest data form an other CKAN instance (remote
>> computer
>> >>>> >> > on
>> >>>> >> > an
>> >>>> >> > other network). I only provided the url to the CKAN frontend:
>> >>>> >> > ckan.odaa.dk.
>> >>>> >> > When i run the harvesting process I get the following error:
>> >>>> >> >
>> >>>> >> > Last Harvest Errors: 12
>> >>>> >> > Object errors
>> >>>> >> >
>> >>>> >> > GUID fredede-bygninger-og-fortidsminder
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID blog-p-bibliotekernes-hjemmeside
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID arrangementer-p-bibliotekerne
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID ejendomsfortegnelse
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID vejnavne-og-vejkoder
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID parkering
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID bibliotekstransaktioner
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID beholdning-ved-aarhus-kommunes-biblioteker
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID koordinats-tning-af-aarhus-adresser
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID cvr-produktionsenheder
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID bevaringsv-rdige-bygninger-i-aarhus-kommune
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> > GUID bbr-ejendomme-og-bbr-bygninger
>> >>>> >> > AttributeError("'thread._local' object has no attribute
>> 'host'",)
>> >>>> >> >
>> >>>> >> > There are only 12 datasets in the CKAN instance I try to harvest
>> >>>> >> > from
>> >>>> >> > (it's
>> >>>> >> > a test instance). I have tried to Google the error, but I don't
>> get
>> >>>> >> > any
>> >>>> >> > relevant suggestions, so I hope you guys can point me in some
>> >>>> >> > direction.
>> >>>> >> >
>> >>>> >> > Regards
>> >>>> >> >
>> >>>> >> > Lasse Vestergaard
>> >>>> >> >
>> >>>> >> > _______________________________________________
>> >>>> >> > ckan-dev mailing list
>> >>>> >> > ckan-dev at lists.okfn.org
>> >>>> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >> >
>> >>>> >>
>> >>>> >> _______________________________________________
>> >>>> >> ckan-dev mailing list
>> >>>> >> ckan-dev at lists.okfn.org
>> >>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > _______________________________________________
>> >>>> > ckan-dev mailing list
>> >>>> > ckan-dev at lists.okfn.org
>> >>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> >>>> >
>> >>>>
>> >>>> _______________________________________________
>> >>>> ckan-dev mailing list
>> >>>> ckan-dev at lists.okfn.org
>> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> >>>
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> >
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130111/615a670a/attachment-0001.html>
More information about the ckan-dev
mailing list