[ckan-dev] Harvestor extension for CKAN

• Previous message: [ckan-dev] Harvestor extension for CKAN
• Next message: [ckan-dev] Harvestor extension for CKAN
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Thanks Lasse,

These logs are really helpful, I'll need to investigate a bit more.

Adrià

On 7 January 2013 14:11, Lasse Vestergaard <ibbernik at gmail.com> wrote:
> Hi again again :-)
>
> I can see that I am able to harvest datasets without resources attached.
> Furthermore, I have tried to make a clean install of CKAN and I still get
> the same result: AttributeError: 'thread._local' object has no attribute
> 'host'
>
> /Lasse
>
>
> 2013/1/7 Lasse Vestergaard <ibbernik at gmail.com>
>>
>> Hi again.
>>
>> I have been looking in the log files, and I got the following:
>>
>> 2013-01-04 15:45:06,336 INFO  [ckanext.harvest.queue] Received harvest
>> object id: 8741a64e-1673-4f29-a713-f9a41d8c9732
>> 2013-01-04 15:45:06,348 DEBUG [ckanext.harvest.harvesters.ckanharvester]
>> In CKANHarvester fetch_stage
>> 2013-01-04 15:45:06,353 DEBUG [ckanext.harvest.harvesters.ckanharvester]
>> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>> u'api_version': u'1'}
>> 2013-01-04 15:45:06,577 DEBUG [ckanext.harvest.harvesters.ckanharvester]
>> In CKANHarvester import_stage
>> 2013-01-04 15:45:06,579 DEBUG [ckanext.harvest.harvesters.ckanharvester]
>> Using config: {u'default_groups': [u'mygroup'], u'user': u'admin',
>> u'api_version': u'1'}
>> 2013-01-04 15:45:06,600 INFO  [ckanext.harvest.harvesters.base] Package
>> with GUID test does not exist, let's create it
>> 2013-01-04 15:45:06,852 ERROR [ckan.model.modification] 'thread._local'
>> object has no attribute 'host'
>> Traceback (most recent call last):
>>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 68,
>> in notify
>>     observer.notify(entity, operation)
>>   File
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> line 30, in notify
>>     self._create_datastorer_task(entity)
>>   File
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> line 42, in _create_datastorer_task
>>     'site_url': h.url_for_static('/', qualified=True),
>>   File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>> url_for_static
>>     my_url = _routes_default_url_for(*args, **kw)
>>   File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
>> url_for
>>     host = config.host
>>   File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14, in
>> __getattr__
>>     return getattr(self.__shared_state, name)
>> AttributeError: 'thread._local' object has no attribute 'host'
>> 2013-01-04 15:45:06,854 ERROR [ckanext.harvest.harvesters.base]
>> 'thread._local' object has no attribute 'host'
>> Traceback (most recent call last):
>>   File "/home/ckan1/ckanext-harvest/ckanext/harvest/harvesters/base.py",
>> line 165, in _create_or_update_package
>>     new_package = get_action('package_create_rest')(context, package_dict)
>>   File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
>> 723, in package_create_rest
>>     dictized_after = _get_action('package_create')(context,
>> dictized_package)
>>   File "/usr/lib/pymodules/python2.6/ckan/logic/action/create.py", line
>> 159, in package_create
>>     model.repo.commit()
>>   File "/usr/lib/pymodules/python2.6/vdm/sqlalchemy/tools.py", line 102,
>> in commit
>>     self.session.commit()
>>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/scoping.py", line
>> 114, in do
>>     return getattr(self.registry(), name)(*args, **kwargs)
>>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
>> 656, in commit
>>     self.transaction.commit()
>>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
>> 314, in commit
>>     self._prepare_impl()
>>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/orm/session.py", line
>> 290, in _prepare_impl
>>     self.session.dispatch.before_commit(self.session)
>>   File "/usr/lib/pymodules/python2.6/ckan/sqlalchemy/event.py", line 291,
>> in __call__
>>     fn(*args, **kw)
>>   File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 103,
>> in before_commit
>>     methodcaller('before_commit', session)
>>   File "/usr/lib/pymodules/python2.6/ckan/model/extension.py", line 38, in
>> notify_observers
>>     func(observer)
>>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 37,
>> in before_commit
>>     self.notify(obj, domain_object.DomainObjectOperation.new)
>>   File "/usr/lib/pymodules/python2.6/ckan/model/modification.py", line 68,
>> in notify
>>     observer.notify(entity, operation)
>>   File
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> line 30, in notify
>>     self._create_datastorer_task(entity)
>>   File
>> "/var/lib/ckan/std/pyenv/src/ckanext-datastorer/ckanext/datastorer/plugin.py",
>> line 42, in _create_datastorer_task
>>     'site_url': h.url_for_static('/', qualified=True),
>>   File "/usr/lib/pymodules/python2.6/ckan/lib/helpers.py", line 87, in
>> url_for_static
>>     my_url = _routes_default_url_for(*args, **kw)
>>   File "/usr/lib/pymodules/python2.6/ckan/routes/util.py", line 257, in
>> url_for
>>     host = config.host
>>   File "/usr/lib/pymodules/python2.6/ckan/routes/__init__.py", line 14, in
>> __getattr__
>>     return getattr(self.__shared_state, name)
>> AttributeError: 'thread._local' object has no attribute 'host'
>> 2013-01-04 15:45:06,881 ERROR [ckanext.harvest.harvesters.base]
>> AttributeError("'thread._local' object has no attribute 'host'",)
>>
>> Does this make sense to anyone :-)
>>
>> /Lasse
>>
>>
>> 2013/1/3 Lasse Vestergaard <ibbernik at gmail.com>
>>>
>>> Hi Adriá Mercader.
>>>
>>> My version of CKAN is 1.8 and the ckanext-harvest is the one I get when I
>>> follow the tutorial at this page: https://github.com/okfn/ckanext-harvest. I
>>> cloned the repository today (3. january).
>>>
>>> /Lasse
>>>
>>>
>>> 2013/1/3 Adrià Mercader <adria.mercader at okfn.org>
>>>>
>>>> Hi Lesse,
>>>>
>>>> You are right, you don't need to specify the json config option unless
>>>> you need it.
>>>> What versions of CKAN core and ckanext-harvest are you using?
>>>>
>>>>
>>>> Adrià
>>>>
>>>> On 3 January 2013 14:49, Lasse Vestergaard <ibbernik at gmail.com> wrote:
>>>> > Hej Peder :-)
>>>> >
>>>> > I wrote this in the configuration field:
>>>> >
>>>> > {
>>>> >  "api_version":"1",
>>>> >  "default_groups":["mygroup"],
>>>> >  "user":"my-username"
>>>> > }
>>>> >
>>>> > The group exists and I can easily create the harvest source. The error
>>>> > occurs when the harvest process runs. As I read the documentation, I
>>>> > don't
>>>> > need to put in all configuration options. Is that correct?
>>>> >
>>>> > /Lasse
>>>> >
>>>> >
>>>> >
>>>> > 2013/1/3 Peder Jakobsen | gmail <pjakobsen at gmail.com>
>>>> >>
>>>> >> HI Lasse,
>>>> >>
>>>> >> Did you enter your configuration details in the form of json in the
>>>> >> UI?
>>>> >>
>>>> >> Venlig hilsen,
>>>> >>
>>>> >> Peder :)
>>>> >>
>>>> >> On Thu, Jan 3, 2013 at 3:56 AM, Lasse Vestergaard
>>>> >> <ibbernik at gmail.com>
>>>> >> wrote:
>>>> >> > Hi all.
>>>> >> >
>>>> >> > I am trying to get the harvestor extension (ckanext-harvest-) work
>>>> >> > for
>>>> >> > CKAN.
>>>> >> > All in all it seems like, I have managed to install it. I am
>>>> >> > currently
>>>> >> > trying to harvest data form an other CKAN instance (remote computer
>>>> >> > on
>>>> >> > an
>>>> >> > other network). I only provided the url to the CKAN frontend:
>>>> >> > ckan.odaa.dk.
>>>> >> > When i run the harvesting process I get the following error:
>>>> >> >
>>>> >> > Last Harvest Errors: 12
>>>> >> > Object errors
>>>> >> >
>>>> >> > GUID fredede-bygninger-og-fortidsminder
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID blog-p-bibliotekernes-hjemmeside
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID arrangementer-p-bibliotekerne
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID ejendomsfortegnelse
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID vejnavne-og-vejkoder
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID parkering
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID bibliotekstransaktioner
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID beholdning-ved-aarhus-kommunes-biblioteker
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID koordinats-tning-af-aarhus-adresser
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID cvr-produktionsenheder
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID bevaringsv-rdige-bygninger-i-aarhus-kommune
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> > GUID bbr-ejendomme-og-bbr-bygninger
>>>> >> > AttributeError("'thread._local' object has no attribute 'host'",)
>>>> >> >
>>>> >> > There are only 12 datasets in the CKAN instance I try to harvest
>>>> >> > from
>>>> >> > (it's
>>>> >> > a test instance). I have tried to Google the error, but I don't get
>>>> >> > any
>>>> >> > relevant suggestions, so I hope you guys can point me in some
>>>> >> > direction.
>>>> >> >
>>>> >> > Regards
>>>> >> >
>>>> >> > Lasse Vestergaard
>>>> >> >
>>>> >> > _______________________________________________
>>>> >> > ckan-dev mailing list
>>>> >> > ckan-dev at lists.okfn.org
>>>> >> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>> >> >
>>>> >>
>>>> >> _______________________________________________
>>>> >> ckan-dev mailing list
>>>> >> ckan-dev at lists.okfn.org
>>>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > ckan-dev mailing list
>>>> > ckan-dev at lists.okfn.org
>>>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>> >
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>

• Previous message: [ckan-dev] Harvestor extension for CKAN
• Next message: [ckan-dev] Harvestor extension for CKAN
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]