[ckan-dev] Harvestor extension for CKAN

Fri Jan 11 13:57:55 UTC 2013

Hi Lasse,

Did you check the configuration options that Toby suggested on the
previous email?

If that does not work could you paste the output of pip freeze with
the virtualenv activated?

Adrià

On 11 January 2013 09:57, Lasse Vestergaard <ibbernik at gmail.com> wrote:
> Hi CKAN developers.
>
> Have you been looking in to the :  AttributeError: 'thread._local' object
> has no attribute 'host' error?
>
> I'm working on creating a distributed CKAN network, and I want a central
> computer to work as an index for the complete network. I have pretty much
> setup everything, but I only need to get this "little" error fixed to be
> able to make it all work.
>
> Best regards
>
> Lasse Vestergaard
>
>
> 2013/1/8 Toby Dacre <toby.okfn at gmail.com>
>>
>>
>>
>> On 7 January 2013 14:45, Adrià Mercader <adria.mercader at okfn.org> wrote:
>>>
>>> Thanks Lasse,
>>>
>>> These logs are really helpful, I'll need to investigate a bit more.
>>>
>>
>> It looks like we are trying to create a qualified url ie the entire url
>> http://host/path it looks like routes is looking in the config .ini file and
>> we need to supply this information, for example in the ini template we have
>> this
>>
>> [server:main]
>> use = egg:Paste#http
>> host = 0.0.0.0
>> port = 5000
>>
>>
>>> Adrià
>>>
>>> On 7 January 2013 14:11, Lasse Vestergaard <ibbernik at gmail.com> wrote:
>>> > Hi again again :-)
>>> >
>>> > I can see that I am able to harvest datasets without resources
>>> > attached.
>>> > Furthermore, I have tried to make a clean install of CKAN and I still
>>> > get
>>> > the same result: AttributeError: 'thread._local' object has no
>>> > attribute
>>> > 'host'
>>> >
>>> > /Lasse
>>> >
>>> >
>>> > 2013/1/7 Lasse Vestergaard <ibbernik at gmail.com>
>>> >>
>>> >> Hi again.
>>> >>
>>> >> I have been looking in the log files, and I got the following:
>>> >>
>>> >> 2013-01-04 15:45:06,336 INFO  [ckanext.harvest.queue] Received harvest
>>> >> object id: 8741a64e-1673-4f29-a713-f9a41d8c9732
>>> >> 2013-01-04 15:45:06,348 DEBUG
>>> >> [ckanext.harvest.harvesters.ckanharvester]
>>> >> In CKANHarvester fetch_stage
>>> >> 2013-01-04 15:45:06,353 DEBUG
>>> >> [ckanext.harvest.harvesters.ckanharvester]
>>> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>>> >> u'api_version': u'1'}
>>> >> 2013-01-04 15:45:06,577 DEBUG
>>> >> [ckanext.harvest.harvesters.ckanharvester]
>>> >> In CKANHarvester import_stage
>>> >> 2013-01-04 15:45:06,579 DEBUG
>>> >> [ckanext.harvest.harvesters.ckanharvester]
>>> >> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>>> >> u'api_version': u'1'}
>>> >> 2013-01-04 15:45:06,600 INFO  [ckanext.harvest.harvesters.base]
>>> >> Package
>>> >> with GUID test does not exist, let's create it
>>> >> 2013-01-04 15:45:06,852 ERROR [ckan.model.modification]
>>> >> 'thread._local'
>>> >> object has no attribute 'host'
>>> >> Traceback (most recent call last):
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>>> >> 68,
>>> >> in notify
>>> >>     observer.notify(entity, operation)
>>> >>   File
>>> >>
>>> >> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>>> >> line 30, in notify
>>> >>     self._create_datastorer_task(entity)
>>> >>   File
>>> >>
>>> >> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>>> >> line 42, in _create_datastorer_task
>>> >>     'site_url': h.url_for_static('/', qualified=True),
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>>> >> url_for_static
>>> >>     my_url = _routes_default_url_for(*args, **kw)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257,
>>> >> in
>>> >> url_for
>>> >>     host = config.host
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line
>>> >> 14, in
>>> >> __getattr__
>>> >>     return getattr(self.__shared_state, name)
>>> >> AttributeError: 'thread._local' object has no attribute 'host'
>>> >> 2013-01-04 15:45:06,854 ERROR [ckanext.harvest.harvesters.base]
>>> >> 'thread._local' object has no attribute 'host'
>>> >> Traceback (most recent call last):
>>> >>   File
>>> >> "/home/ckan1/ckanext-harvest/ckanext/harvest/harvesters/base.py",
>>> >> line 165, in _create_or_update_package
>>> >>     new_package = get_action('package_create_rest')(context,
>>> >> package_dict)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py",
>>> >> line
>>> >> 723, in package_create_rest
>>> >>     dictized_after = _get_action('package_create')(context,
>>> >> dictized_package)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py",
>>> >> line
>>> >> 159, in package_create
>>> >>     model.repo.commit()
>>> >>   File "/usr/lib/pymodules/python2.6/vdm/sqlalchemy/tools.py", line
>>> >> 102,
>>> >> in commit
>>> >>     self.session.commit()
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/scoping.py",
>>> >> line
>>> >> 114, in do
>>> >>     return getattr(self.registry(), name)(*args, **kwargs)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>>> >> line
>>> >> 656, in commit
>>> >>     self.transaction.commit()
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>>> >> line
>>> >> 314, in commit
>>> >>     self._prepare_impl()
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py",
>>> >> line
>>> >> 290, in _prepare_impl
>>> >>     self.session.dispatch.before_commit(self.session)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/event.py", line
>>> >> 291,
>>> >> in __call__
>>> >>     fn(*args, **kw)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line
>>> >> 103,
>>> >> in before_commit
>>> >>     methodcaller('before_commit', session)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line
>>> >> 38, in
>>> >> notify_observers
>>> >>     func(observer)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>>> >> 37,
>>> >> in before_commit
>>> >>     self.notify(obj, domain_object.DomainObjectOperation.new)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line
>>> >> 68,
>>> >> in notify
>>> >>     observer.notify(entity, operation)
>>> >>   File
>>> >>
>>> >> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>>> >> line 30, in notify
>>> >>     self._create_datastorer_task(entity)
>>> >>   File
>>> >>
>>> >> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>>> >> line 42, in _create_datastorer_task
>>> >>     'site_url': h.url_for_static('/', qualified=True),
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>>> >> url_for_static
>>> >>     my_url = _routes_default_url_for(*args, **kw)
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257,
>>> >> in
>>> >> url_for
>>> >>     host = config.host
>>> >>   File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line
>>> >> 14, in
>>> >> __getattr__
>>> >>     return getattr(self.__shared_state, name)
>>> >> AttributeError: 'thread._local' object has no attribute 'host'
>>> >> 2013-01-04 15:45:06,881 ERROR [ckanext.harvest.harvesters.base]
>>> >> AttributeError("'thread._local' object has no attribute 'host'",)
>>> >>
>>> >> Does this make sense to anyone :-)
>>> >>
>>> >> /Lasse
>>> >>
>>> >>
>>> >> 2013/1/3 Lasse Vestergaard <ibbernik at gmail.com>
>>> >>>
>>> >>> Hi Adriá Mercader.
>>> >>>
>>> >>> My version of CKAN is 1.8 and the ckanext-harvest is the one I get
>>> >>> when I
>>> >>> follow the tutorial at this page:
>>> >>> https://github.com/okfn/ckanext-harvest. I
>>> >>> cloned the repository today (3. january).
>>> >>>
>>> >>> /Lasse
>>> >>>
>>> >>>
>>> >>> 2013/1/3 Adrià Mercader <adria.mercader at okfn.org>
>>> >>>>
>>> >>>> Hi Lesse,
>>> >>>>
>>> >>>> You are right, you don't need to specify the json config option
>>> >>>> unless
>>> >>>> you need it.
>>> >>>> What versions of CKAN core and ckanext-harvest are you using?
>>> >>>>
>>> >>>>
>>> >>>> Adrià
>>> >>>>
>>> >>>> On 3 January 2013 14:49, Lasse Vestergaard <ibbernik at gmail.com>
>>> >>>> wrote:
>>> >>>> > Hej Peder :-)
>>> >>>> >
>>> >>>> > I wrote this in the configuration field:
>>> >>>> >
>>> >>>> > {
>>> >>>> >  "api_version":"1",
>>> >>>> >  "default_groups":["mygroup"],
>>> >>>> >  "user":"my-username"
>>> >>>> > }
>>> >>>> >
>>> >>>> > The group exists and I can easily create the harvest source. The
>>> >>>> > error
>>> >>>> > occurs when the harvest process runs. As I read the documentation,
>>> >>>> > I
>>> >>>> > don't
>>> >>>> > need to put in all configuration options. Is that correct?
>>> >>>> >
>>> >>>> > /Lasse
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > 2013/1/3 Peder Jakobsen | gmail <pjakobsen at gmail.com>
>>> >>>> >>
>>> >>>> >> HI Lasse,
>>> >>>> >>
>>> >>>> >> Did you enter your configuration details in the form of json in
>>> >>>> >> the
>>> >>>> >> UI?
>>> >>>> >>
>>> >>>> >> Venlig hilsen,
>>> >>>> >>
>>> >>>> >> Peder :)
>>> >>>> >>
>>> >>>> >> On Thu, Jan 3, 2013 at 3:56 AM, Lasse Vestergaard
>>> >>>> >> <ibbernik at gmail.com>
>>> >>>> >> wrote:
>>> >>>> >> > Hi all.
>>> >>>> >> >
>>> >>>> >> > I am trying to get the harvestor extension (ckanext-harvest-)
>>> >>>> >> > work
>>> >>>> >> > for
>>> >>>> >> > CKAN.
>>> >>>> >> > All in all it seems like, I have managed to install it. I am
>>> >>>> >> > currently
>>> >>>> >> > trying to harvest data form an other CKAN instance (remote
>>> >>>> >> > computer
>>> >>>> >> > on
>>> >>>> >> > an
>>> >>>> >> > other network). I only provided the url to the CKAN frontend:
>>> >>>> >> > ckan.odaa.dk.
>>> >>>> >> > When i run the harvesting process I get the following error:
>>> >>>> >> >
>>> >>>> >> > Last Harvest Errors: 12
>>> >>>> >> > Object errors
>>> >>>> >> >
>>> >>>> >> > GUID fredede-bygninger-og-fortidsminder
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID blog-p-bibliotekernes-hjemmeside
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID arrangementer-p-bibliotekerne
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID ejendomsfortegnelse
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID vejnavne-og-vejkoder
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID parkering
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID bibliotekstransaktioner
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID beholdning-ved-aarhus-kommunes-biblioteker
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID koordinats-tning-af-aarhus-adresser
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID cvr-produktionsenheder
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID bevaringsv-rdige-bygninger-i-aarhus-kommune
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> > GUID bbr-ejendomme-og-bbr-bygninger
>>> >>>> >> > AttributeError("'thread._local' object has no attribute
>>> >>>> >> > 'host'",)
>>> >>>> >> >
>>> >>>> >> > There are only 12 datasets in the CKAN instance I try to
>>> >>>> >> > harvest
>>> >>>> >> > from
>>> >>>> >> > (it's
>>> >>>> >> > a test instance). I have tried to Google the error, but I don't
>>> >>>> >> > get
>>> >>>> >> > any
>>> >>>> >> > relevant suggestions, so I hope you guys can point me in some
>>> >>>> >> > direction.
>>> >>>> >> >
>>> >>>> >> > Regards
>>> >>>> >> >
>>> >>>> >> > Lasse Vestergaard
>>> >>>> >> >
>>> >>>> >> > _______________________________________________
>>> >>>> >> > ckan-dev mailing list
>>> >>>> >> > ckan-dev at lists.okfn.org
>>> >>>> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >>>> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >>>> >> >
>>> >>>> >>
>>> >>>> >> _______________________________________________
>>> >>>> >> ckan-dev mailing list
>>> >>>> >> ckan-dev at lists.okfn.org
>>> >>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > _______________________________________________
>>> >>>> > ckan-dev mailing list
>>> >>>> > ckan-dev at lists.okfn.org
>>> >>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >>>> >
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> ckan-dev mailing list
>>> >>>> ckan-dev at lists.okfn.org
>>> >>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >>>
>>> >>>
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>