[ckan-dev] Harvestor extension for CKAN
Lasse Vestergaard
ibbernik at gmail.com
Mon Jan 7 14:11:56 UTC 2013
Hi again again :-)
I can see that I am able to harvest datasets without resources attached.
Furthermore, I have tried to make a clean install of CKAN and I still get
the same result: AttributeError: 'thread._local' object has no attribute
'host'
/Lasse
2013/1/7 Lasse Vestergaard <ibbernik at gmail.com>
> Hi again.
>
> I have been looking in the log files, and I got the following:
>
> 2013-01-04 15:45:06,336 INFO [ckanext.harvest.queue] Received harvest
> object id: 8741a64e-1673-4f29-a713-f9a41d8c9732
> 2013-01-04 15:45:06,348 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> In CKANHarvester fetch_stage
> 2013-01-04 15:45:06,353 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
> u'api_version': u'1'}
> 2013-01-04 15:45:06,577 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> In CKANHarvester import_stage
> 2013-01-04 15:45:06,579 DEBUG [ckanext.harvest.harvesters.ckanharvester]
> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
> u'api_version': u'1'}
> 2013-01-04 15:45:06,600 INFO [ckanext.harvest.harvesters.base] Package
> with GUID test does not exist, let's create it
> 2013-01-04 15:45:06,852 ERROR [ckan.model.modification] 'thread._local'
> object has no attribute 'host'
> Traceback (most recent call last):
> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 68,
> in notify
> observer.notify(entity, operation)
> File
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> line 30, in notify
> self._create_datastorer_task(entity)
> File
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> line 42, in _create_datastorer_task
> 'site_url': h.url_for_static('/', qualified=True),
> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
> url_for_static
> my_url = _routes_default_url_for(*args, **kw)
> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
> url_for
> host = config.host
> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14, in
> __getattr__
> return getattr(self.__shared_state, name)
> AttributeError: 'thread._local' object has no attribute 'host'
> 2013-01-04 15:45:06,854 ERROR [ckanext.harvest.harvesters.base]
> 'thread._local' object has no attribute 'host'
> Traceback (most recent call last):
> File "/home/ckan1/ckanext-harvest/ckanext/harvest/harvesters/base.py",
> line 165, in _create_or_update_package
> new_package = get_action('package_create_rest')(context, package_dict)
> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
> 723, in package_create_rest
> dictized_after = _get_action('package_create')(context,
> dictized_package)
> File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
> 159, in package_create
> model.repo.commit()
> File "/usr/lib/pymodules/python2.6/vdm/sqlalchemy/tools.py", line 102,
> in commit
> self.session.commit()
> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/scoping.py", line
> 114, in do
> return getattr(self.registry(), name)(*args, **kwargs)
> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
> 656, in commit
> self.transaction.commit()
> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
> 314, in commit
> self._prepare_impl()
> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
> 290, in _prepare_impl
> self.session.dispatch.before_commit(self.session)
> File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/event.py", line 291,
> in __call__
> fn(*args, **kw)
> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 103,
> in before_commit
> methodcaller('before_commit', session)
> File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 38, in
> notify_observers
> func(observer)
> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 37,
> in before_commit
> self.notify(obj, domain_object.DomainObjectOperation.new)
> File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 68,
> in notify
> observer.notify(entity, operation)
> File
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> line 30, in notify
> self._create_datastorer_task(entity)
> File
> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
> line 42, in _create_datastorer_task
> 'site_url': h.url_for_static('/', qualified=True),
> File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
> url_for_static
> my_url = _routes_default_url_for(*args, **kw)
> File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
> url_for
> host = config.host
> File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14, in
> __getattr__
> return getattr(self.__shared_state, name)
> AttributeError: 'thread._local' object has no attribute 'host'
> 2013-01-04 15:45:06,881 ERROR [ckanext.harvest.harvesters.base]
> AttributeError("'thread._local' object has no attribute 'host'",)
>
> Does this make sense to anyone :-)
>
> /Lasse
>
>
> 2013/1/3 Lasse Vestergaard <ibbernik at gmail.com>
>
>> Hi Adriá Mercader.
>>
>> My version of CKAN is 1.8 and the ckanext-harvest is the one I get when I
>> follow the tutorial at this page: https://github.com/okfn/ckanext-harvest.
>> I cloned the repository today (3. january).
>>
>> /Lasse
>>
>>
>> 2013/1/3 Adrià Mercader <adria.mercader at okfn.org>
>>
>>> Hi Lesse,
>>>
>>> You are right, you don't need to specify the json config option unless
>>> you need it.
>>> What versions of CKAN core and ckanext-harvest are you using?
>>>
>>>
>>> Adrià
>>>
>>> On 3 January 2013 14:49, Lasse Vestergaard <ibbernik at gmail.com> wrote:
>>> > Hej Peder :-)
>>> >
>>> > I wrote this in the configuration field:
>>> >
>>> > {
>>> > "api_version":"1",
>>> > "default_groups":["mygroup"],
>>> > "user":"my-username"
>>> > }
>>> >
>>> > The group exists and I can easily create the harvest source. The error
>>> > occurs when the harvest process runs. As I read the documentation, I
>>> don't
>>> > need to put in all configuration options. Is that correct?
>>> >
>>> > /Lasse
>>> >
>>> >
>>> >
>>> > 2013/1/3 Peder Jakobsen | gmail <pjakobsen at gmail.com>
>>> >>
>>> >> HI Lasse,
>>> >>
>>> >> Did you enter your configuration details in the form of json in the
>>> UI?
>>> >>
>>> >> Venlig hilsen,
>>> >>
>>> >> Peder :)
>>> >>
>>> >> On Thu, Jan 3, 2013 at 3:56 AM, Lasse Vestergaard <ibbernik at gmail.com
>>> >
>>> >> wrote:
>>> >> > Hi all.
>>> >> >
>>> >> > I am trying to get the harvestor extension (ckanext-harvest-) work
>>> for
>>> >> > CKAN.
>>> >> > All in all it seems like, I have managed to install it. I am
>>> currently
>>> >> > trying to harvest data form an other CKAN instance (remote computer
>>> on
>>> >> > an
>>> >> > other network). I only provided the url to the CKAN frontend:
>>> >> > ckan.odaa.dk.
>>> >> > When i run the harvesting process I get the following error:
>>> >> >
>>> >> > Last Harvest Errors: 12
>>> >> > Object errors
>>> >> >
>>> >> > GUID fredede-bygninger-og-fortidsminder
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID blog-p-bibliotekernes-hjemmeside
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID arrangementer-p-bibliotekerne
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID ejendomsfortegnelse
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID vejnavne-og-vejkoder
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID parkering
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID bibliotekstransaktioner
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID beholdning-ved-aarhus-kommunes-biblioteker
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID koordinats-tning-af-aarhus-adresser
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID cvr-produktionsenheder
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID bevaringsv-rdige-bygninger-i-aarhus-kommune
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> > GUID bbr-ejendomme-og-bbr-bygninger
>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>> >> >
>>> >> > There are only 12 datasets in the CKAN instance I try to harvest
>>> from
>>> >> > (it's
>>> >> > a test instance). I have tried to Google the error, but I don't get
>>> any
>>> >> > relevant suggestions, so I hope you guys can point me in some
>>> direction.
>>> >> >
>>> >> > Regards
>>> >> >
>>> >> > Lasse Vestergaard
>>> >> >
>>> >> > _______________________________________________
>>> >> > ckan-dev mailing list
>>> >> > ckan-dev at lists.okfn.org
>>> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >> >
>>> >>
>>> >> _______________________________________________
>>> >> ckan-dev mailing list
>>> >> ckan-dev at lists.okfn.org
>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130107/43c49634/attachment-0001.html>
More information about the ckan-dev
mailing list