[ckan-dev] harvesting user and deleting a user, PostgreSQL version ?
Joe Tsoi
joe.tsoi at okfn.org
Wed Nov 27 07:50:31 UTC 2013
Hi,
The harvest source not found error looks like it might not be the
harvest user as it could be bailing out before it gets to the point
where it uses it.
I think there are two seperate things that might, help. Either you
harvest extension isn't up to date, could you do a git show on
ckanext-harvest directory? Just so I can know which version of the
code we're dealing with here.
Or you've got jobs left over from a previous harvest job before you
wiped the database. In which case could you do a paster harvester
purge_queues and then start again with the empty database
On 26 November 2013 17:45, Hildegard Gerlach
<hildegard.gerlach at jrc.ec.europa.eu> wrote:
> Hi again,
>
> I did another test. I created a new database, initialized it for ckan, the
> harvest and spatial extension and created the sysadmin user admin.
> Anyway, an empty user (name '') is created again when initializing the
> database for CKAN (version 2.1.1)
> When I try to harvest I get the same error
>
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/update.py",
> line 28, in harvest_source_update
> raise pt.ObjectNotFound(pt._('Harvest source not found'))
> ckan.logic.NotFound: Harvest source not found
>
> and in the logfile I have the following:
>
> 2013-11-26 16:19:35,913 DEBUG [ckanext.harvest.model] Harvest tables defined
> in memory
> 2013-11-26 16:19:35,917 DEBUG [ckanext.harvest.model] Harvest tables already
> exist
> 2013-11-26 16:19:35,938 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables defined in memory
> 2013-11-26 16:19:35,944 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables already exist
> 2013-11-26 16:19:35,960 DEBUG [ckanext.harvest.queue] pika connection using
> {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale' :
> 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> <pika.credentials.PlainCredentials object at 0x3949a50>,
> 'backpressure_detection': False, 'port': 5672, 'connection_attempts': 1}
> 2013-11-26 16:19:37,008 DEBUG [ckanext.harvest.queue] Gather queue consumer
> registered
> 2013-11-26 16:20:17,507 DEBUG [ckanext.harvest.model] Harvest tables defined
> in memory
> 2013-11-26 16:20:17,511 DEBUG [ckanext.harvest.model] Harvest tables already
> exist
> 2013-11-26 16:20:17,532 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables defined in memory
> 2013-11-26 16:20:17,540 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables already exist
> 2013-11-26 16:20:17,555 DEBUG [ckanext.harvest.queue] pika connection using
> {'retry_delay': 2.0, 'frame_max': 10000, 'channel_max': 0, 'locale' :
> 'en_US', 'socket_timeout': 0.25, 'ssl': False, 'host': 'localhost',
> 'ssl_options': {}, 'virtual_host': '/', 'heartbeat': 0, 'credentials':
> <pika.credentials.PlainCredentials object at 0x39be990>,
> 'backpressure_detection': False, 'port': 5672, 'connection_attempts': 1}
> 2013-11-26 16:20:18,602 DEBUG [ckanext.harvest.queue] Fetch queue consumer
> registered
> 2013-11-26 16:20:32,927 DEBUG [ckanext.harvest.model] Harvest tables defined
> in memory
> 2013-11-26 16:20:32,931 DEBUG [ckanext.harvest.model] Harvest tables already
> exist
> 2013-11-26 16:20:32,952 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables defined in memory
> 2013-11-26 16:20:32,959 DEBUG [ckanext.spatial.model.package_extent] Spatial
> tables already exist
> 2013-11-26 16:20:32,974 INFO [ckanext.harvest.logic.action.update] Harvest
> job run: {}
> 2013-11-26 16:21:03,016 WARNI [ckan.lib.maintain] Function nav_named_link()
> in module ckan.lib.helpers has been deprecated and will be removed in a
> later release of ckan. h.nav_named_link is deprecated please use h.nav_link
> NOTE: you will need to pass the route_name as a named parameter
>
>
> We are using solr as a backend
> ckanext.spatial.search_backend = solr
>
> and have PostgreSQL 9.2 with PostGIS 2.1.0
>
> ckandb=> SELECT PostGIS_full_version();
> postgis_full_version
> ----------------------------------------------------------------------
> POSTGIS="2.1.0 r11822" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6
> March 2012" GDAL="GDAL 1.9.2, released 2012/10/08"
> LIBXML="2.7.6" LIBJSON="UNKNOWN" TOPOLOGY RASTER
> (1 row)
>
> and the table package_extent is slightly different as to an older version of
> PostgreSQL/PostGIS
>
> ckandb=> \d package_extent
> Table "public.package_extent"
> Column | Type | Modifiers
> ------------+-------------------------+-----------
> package_id | text | not null
> the_geom | geometry(Geometry,4326) |
> Indexes:
> "package_extent_pkey" PRIMARY KEY, btree (package_id)
> "idx_package_extent_the_geom" gist (the_geom)
>
>
> I also cannot clean the database, I get the following error:
>
> paster db clean -c /etc/ckan/production.ini
> /usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47:
> SAWarning: Did not recognize type 'geometry(Geometry,4326)' of column
> 'the_geom'
> ret = fn(self, con, *args, **kw)
>
> Is this version of PostgreSQL/PostGIS not yet supported by CKAN ?
>
> I really don't know what to try any more.
>
> Thanks
>
> Hilde
>
>
>
> On 26.11.2013, at 12:09, Hildegard Gerlach
> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>
> Hi everyone,
>
>
> http://docs.ckan.org/projects/ckanext-spatial/en/latest/harvesters.html says
>
> By default the harvesting actions (eg creating or updating datasets) will be
> performed by the internal site admin user. This is the recommended setting,
> but if necessary, it can be overridden with the
> ckanext.spatial.harvest.user_name config option, eg to support the old
> hardcoded ‘harvest’ user:
>
> ckanext.spatial.harvest.user_name = harvest
>
> We have another instance of CKAN (both on Redhat) on another machine where
> the harvester works without having a user harvest.
>
> I am working with Elena and we have obviously an authorization problem
> running the harvesting.
>
> paster --plugin=ckanext-harvest harvester run -c /etc/ckan/production.ini
>
> Traceback (most recent call last):
> File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
> load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
> File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 104, in run
> invoke(command, command_name, options, args[1:])
> File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 143, in invoke
> exit_code = runner.run(args)
> File
> "/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 238, in run
> result = self.command()
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> line 113, in command
> self.run_harvester()
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/commands/harvester.py",
> line 278, in run_harvester
> jobs = get_action('harvest_jobs_run')(context,{})
> File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 329, in
> wrapped
> return _action(context, data_dict, **kw)
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/update.py",
> line 291, in harvest_jobs_run
> jobs = harvest_job_list(context,{'source_id':source_id,'status':u'Running'})
> File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 386, in
> wrapper
> return action(context, data_dict)
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/action/get.py",
> line 226, in harvest_job_list
> check_access('harvest_job_list',context,data_dict)
> File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 207, in
> check_access
> logic_authorization = new_authz.is_authorized(action, context, data_dict)
> File "/usr/local/ckan/pyenv/src/ckan/ckan/new_authz.py", line 82, in
> is_authorized
> return auth_function(context, data_dict)
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/get.py",
> line 85, in harvest_job_list
> {'id': source_id})
> File "/usr/local/ckan/pyenv/src/ckan/ckan/logic/__init__.py", line 207, in
> check_access
> logic_authorization = new_authz.is_authorized(action, context, data_dict)
> File "/usr/local/ckan/pyenv/src/ckan/ckan/new_authz.py", line 82, in
> is_authorized
> return auth_function(context, data_dict)
> File
> "/usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/update.py",
> line 28, in harvest_source_update
> raise pt.ObjectNotFound(pt._('Harvest source not found'))
> ckan.logic.NotFound: Harvest source not found
>
> I changed the
> /usr/local/ckan/pyenv/src/ckanext-harvest/ckanext/harvest/logic/auth/update.py
> to output some data, and I get an empty user (no value)
> and source_id: None
>
> Now I tried to set
>
> ckanext.spatial.harvest.user_name = admin
>
> but this didn't change anything.
>
> Then we have this empty user (name= ) which might cause a problem, but we
> don't know hot to get rid of it (as already described).
>
> paster --plugin=ckan user list -c /etc/ckan/production.ini
> Users:
> count = 9
> name=
> name=admin
> name=annafan
> name=joeadmin
> name=logged_in
> name=russianfan
> name=tester
> name=testsysadmin
> name=visitor
>
> Any help would be appreciated.
>
> Otherwise is there a way to uninstall and reinstall the harvesting extension
> ? We did an upgrade to 2.1.1 of CKAN and extensions to stable, but nothing
> changed.
>
> Kind regards
>
> Hilde
>
>
>
> On 11/25/2013 9:05 PM, Vitor Baptista wrote:
>
> Hi Elena,
>
> By default, ckanext-harvest use the user "harvest". You'll have some poblems
> if it doesn't exist (or you haven't changed it to another username). To
> solve it, you can simply add this user as a sysadmin. The password doesn't
> matter.
>
> Cheers,
>
>
> 2013/11/25 Elena Camossi <elena.camossi at ext.jrc.ec.europa.eu
> <mailto:elena.camossi at ext.jrc.ec.europa.eu>>
>
> Hi everyone,
>
> we are experiencing some problem with harvesting (CKAN 2.1, on
> RedHat),
> doing some debugging we discovered that harvesting seems to be
> runned by a
> user with no name.
>
> Getting the list of users with "paster user list", we discovered
> that one of
> the users has indeed no name, and table user within Postgres has
> indeed a
> row
> column name having value ' '.
>
> We have tried to delete this user with "paster user remove". Command
> succeeds but user still exists. We have tried deleting it from the
> Postgresql database, the row is removed but it reappears after
> running again
> "paster user list"...
>
> Is there any other way to delete a CKAN user?
>
> Does harvesting use some default CKAN user?
>
> Thanks a lot for your help.
>
> Kind regards,
> -Elen a
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org <mailto:ckan-dev at lists.okfn.org>
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
>
> --
>
> Vítor Baptista
>
> Developer |http://vitorbaptista.com | LinkedIn
> <http://www.linkedin.com/in/vitorbaptista> | @vitorbaptista
> <http://twitter.com/vitorbaptista>
>
> The Open Knowledge Foundation <http://okfn.org>
>
> /Empowering through Open Knowledge/
>
> http://okfn.org/ | @okfn <http://twitter.com/okfn> | OKF on Facebook
> <https://www.facebook.com/OKFNetwork> | Blog <http://blog.okfn.org/> |
> Newsletter <http://okfn.org/about/newsletter/>
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
More information about the ckan-dev
mailing list