[ckan-dev] delete a harvesting resource

Elena Camossi elena.camossi at ext.jrc.ec.europa.eu
Tue Nov 12 15:50:56 UTC 2013


Hi everyone,

we have found out how to partially delete a harvest source. 

While command 

	paster --plugin=ckanext-harvest harvester rmsource [harvest source
id] 

just disactivates the harvest source, this still exists as dataset. 
So harvest sources can be deleted using  command 

	paster dataset purge [harvest source id]   

which removes dataset from db. The "deleted" harvest source still exists
somewhere, because it is still visible when running 

	paster --plugin=ckanext-harvest harvester  harvester sources 

but at least it can  be re-inserted from the web interface as a new source,
which wasn't the case after running just rmsource.

However, even if the new source is visible from http://myckansite/harvest,
and a new harvesting request can be inserted in the queue, when launching
harvester run we have again an error of Harvest source not found

(pyenv)[ckan at drdsi-data ckan]$ paster --plugin=ckanext-harvest harvester run
-c /etc/ckan/production.ini

Traceback (most recent call last):
   File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
   File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 104, in run
     invoke(command, command_name, options, args[1:])
   File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 143, in invoke
     exit_code = runner.run(args)
   File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 238, in run
     result = self.command()
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/commands/harveste
r.py",
line 113, in command
     self.run_harvester()
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/commands/harveste
r.py",
line 278, in run_harvester
     jobs = get_action('harvest_jobs_run')(context,{})
   File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 329,
in wrapped
     return _action(context, data_dict, **kw)
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/upda
te.py",
line 291, in harvest_jobs_run
     jobs =
harvest_job_list(context,{'source_id':source_id,'status':u'Running'})
   File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 386,
in wrapper
     return action(context, data_dict)
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/get.
py",
line 226, in harvest_job_list
     check_access('harvest_job_list',context,data_dict)
   File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 207,
in check_access
     logic_authorization = new_authz.is_authorized(action, context,
data_dict)
   File "/usr/local/ckan/pyenv/src/ckan/ckan/new_authz.py", line 80, in
is_authorized
     return auth_function(context, data_dict)
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/get.py
",
line 85, in harvest_job_list
     {'id': source_id})
   File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 207,
in check_access
     logic_authorization = new_authz.is_authorized(action, context,
data_dict)
   File "/usr/local/ckan/pyenv/src/ckan/ckan/new_authz.py", line 80, in
is_authorized
     return auth_function(context, data_dict)
   File
"/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/update
.py",
line 19, in harvest_source_update
     raise pt.ObjectNotFound(pt._('Harvest source not found'))
ckan.logic.NotFound: Harvest source not found (pyenv)[ckan at drdsi-data ckan]$
gedit /usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/

Does someone have any idea on what went wrong? 


Thanks.
Regards,
-Elena 






> -----Original Message-----
> From: ckan-dev-bounces at lists.okfn.org [mailto:ckan-dev-
> bounces at lists.okfn.org] On Behalf Of Hildegard Gerlach
> Sent: lunedì 11 novembre 2013 14:29
> To: CKAN Development Discussions
> Subject: [ckan-dev] delete a harvesting resource
> 
> Hello,
> 
> we have installed CKAN 2.1 on Redhat6 and enabled the datastore
> extension. This caused errors when trying to use the harvester.
> 
> IntegrityError: (IntegrityError) duplicate key value violates unique
constraint
> "pg_type_typname_nsp_index", refere jrc.it/harvest [Mon Nov 11 13:58:14
> 2013] [error] [client 139.191.16.105] DETAIL: Key (typname,
> typnamespace)=(_foo, 2200) already exists., referer:
> http://drdsi-data.ies.jrc.it/harvest
> [Mon Nov 11 13:58:14 2013] [error] [client 139.191.16.105]  'CREATE TABLE
> _foo ()' {}, referer: http://drdsi-data.ies.jrc.it/harvest
> [Mon Nov 11 13:58:14 2013] [error] [client 139.191.16.105]
> cursor.execute(statement, parameters)
> 
> I disabled the datastore again and have no harvesting sources visible in
the
> interface (hostname/harvest) but they are still there.
> 
> (pyenv)[ckan at drdsi-data ckan]$ paster --plugin=ckanext-harvest harvester
> sources all  -c /etc/ckan/production.ini
> 
> Source id: 92ff0ade-0892-4d84-9e3b-ed8bd2612aff
>        url:
> http://rdsi-portal.jrc.it:8080/geonetwork/srv/eng/csw-
> drdsi?request=GetCapabilities&service=CSW&version=2.0.2
>       type: csw
>     active: False
> frequency: MANUAL
>       jobs: 0
> 
> Source id: 58f12a89-1525-43d7-a822-cdc07d648101
>        url:
> http://drdsi-
> portal.ies.jrc.it/GeoportalProxyWebServices/resources/OGCCSW202?service
> =CSW&version=2.0.2&request=GetCapabilities
>       type: csw
>     active: False
> frequency: MANUAL
>       jobs: 0
> 
> Source id: 4a0b4385-4483-4bc3-9a1c-c0d238b90808
>        url:
> http://cat.ccss.cz/csw/?request=GetCapabilities&service=CSW&version=2.0.
> 2
>       type: csw
>     active: False
> frequency: MANUAL
>       jobs: 0
> 
> There are 3 harvest sources
> 
> How can I delete them ? I tried with user ckan in the virtual environment
but
> the user ckan doesn't seem to be authorized.
> 
> (pyenv)[ckan at drdsi-data ckan]$ paster --plugin=ckanext-harvest harvester
> rmsource 92ff0ade-0892-4d84-9e3b-ed8bd2612aff  -c
> /etc/ckan/production.ini
> 
> Traceback (most recent call last):
>    File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
>      load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
>    File
> "/usr/local/ckan/pyenv/lib/python2.6/site-
> packages/paste/script/command.py",
> line 104, in run
>      invoke(command, command_name, options, args[1:])
>    File
> "/usr/local/ckan/pyenv/lib/python2.6/site-
> packages/paste/script/command.py",
> line 143, in invoke
>      exit_code = runner.run(args)
>    File
> "/usr/local/ckan/pyenv/lib/python2.6/site-
> packages/paste/script/command.py",
> line 238, in run
>      result = self.command()
>    File
> "/usr/local/ckan/pyenv/src/ckanext-
> harvest/ckanext/harvest/commands/harvester.py",
> line 105, in command
>      self.remove_harvest_source()
>    File
> "/usr/local/ckan/pyenv/src/ckanext-
> harvest/ckanext/harvest/commands/harvester.py",
> line 239, in remove_harvest_source
>      get_action('harvest_source_delete')(context,{'id':source_id})
>    File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 329,
in
> wrapped
>      return _action(context, data_dict, **kw)
>    File
> "/usr/local/ckan/pyenv/src/ckanext-
> harvest/ckanext/harvest/logic/action/delete.py",
> line 25, in harvest_source_delete
>      p.toolkit.check_access('harvest_source_delete', context, data_dict)
>    File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 210,
in
> check_access
>      raise NotAuthorized(msg)
> ckan.logic.NotAuthorized: User  not authorized to delete harvest source
> 92ff0ade-0892-4d84-9e3b-ed8bd2612aff
> 
> Thanks
> 
> Hilde
> 
> 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev





More information about the ckan-dev mailing list