[ckan-dev] Connection slots error

• Previous message: [ckan-dev] Connection slots error
• Next message: [ckan-dev] Connection slots error
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi Luis,

We've successfully used pgbouncer to solve this problem, it pre-creates a set number of connection of Postgres and 
multiplexes queries and re-uses those connections.  I believe others may use pgpool2. It has been very stable and very fast for us.

It's also worth checking (as well as pgbouncer/pgpool) whether this is caused by someone harvesting at a particularly unhealthy rate.  You could always rate limit those harvesters in your nginx config. We also had to do this.

Hope this helps.

Ross



> On 10 Feb 2016, at 14:53, Luis <eltipicoemail at gmail.com> wrote:
> 
> Hi all,
> 
> from time to time, we are getting this error on our CKAN installation:
> 
> [Mon Feb 01 07:17:03.284918 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438] mod_wsgi (pid=26305): Target WSGI script '/etc/ckan/default/apache.wsgi' cannot be loaded as Python module., referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.284974 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438] mod_wsgi (pid=26305): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'., referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.285036 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438] Traceback (most recent call last):, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286007 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/etc/ckan/default/apache.wsgi", line 10, in <module>, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286090 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     application = loadapp('config:%s' % config_filepath), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286120 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286164 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return loadobj(APP, uri, name=name, **kw), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286189 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286245 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return context.create(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286271 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286304 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return self.object_type.invoke(self), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286327 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286460 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return fix_call(context.object, context.global_conf, **context.local_conf), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286491 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286529 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     val = callable(*args, **kw), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286553 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware.py", line 58, in make_app, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286590 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     load_environment(conf, app_conf), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286613 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 232, in load_environment, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286647 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     p.load_all(config), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286670 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 134, in load_all, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286704 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     load(*plugins), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286727 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 166, in load, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286757 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     plugins_update(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286779 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 116, in plugins_update, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286824 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     environment.update_config(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286848 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 403, in update_config, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286880 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     plugin.configure(config), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286903 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/plugin.py", line 202, in configure, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286938 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     model_setup(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286961 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 50, in setup, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.286995 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     if not model.package_table.exists():, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287018 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 631, in exists, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287052 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     self.name, schema=self.schema), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287074 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1670, in run_callable, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287109 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     with self.contextual_connect() as conn:, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287132 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1737, in contextual_connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287164 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     self.pool.connect(),, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287187 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 332, in connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287219 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return _ConnectionFairy._checkout(self), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287241 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 630, in _checkout, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287284 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     fairy = _ConnectionRecord.checkout(pool), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287308 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 433, in checkout, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287339 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     rec = pool._do_get(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287361 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 949, in _do_get, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287392 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return self._create_connection(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287414 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 278, in _create_connection, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287445 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return _ConnectionRecord(self), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287467 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 404, in __init__, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287499 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     self.connection = self.__connect(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287522 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py", line 530, in __connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287553 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     connection = self.__pool._creator(), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287576 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 95, in connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287608 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     connection_invalidated=invalidated, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287631 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287665 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     reraise(type(exception), exception, tb=exc_tb), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287696 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 89, in connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287730 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return dialect.connect(*cargs, **cparams), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287753 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 376, in connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287786 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     return self.dbapi.connect(*cargs, **cparams), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287808 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287842 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438]     connection_factory=connection_factory, async=async), referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> [Mon Feb 01 07:17:03.287942 2016] [:error] [pid 26305:tid 139727432808192] [client 127.0.0.1:56438] OperationalError: (OperationalError) FATAL:  remaining connection slots are reserved for non-replication superuser connections, referer: http://taro-des.fg.ull.es/datos/sl/dataset/0faa5f08-dd15-41f7-a51e-8648174c35de
> 
> We have tried to increase the max_connections setting in our postgresql.conf but the error still occurs after a while. 
> 
> Any help to fix this would be appreciated.
> 
> Regards,
> Luis
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

• Previous message: [ckan-dev] Connection slots error
• Next message: [ckan-dev] Connection slots error
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]