[ckan-dev] Database connections per worker

Fri May 2 10:15:19 UTC 2014

On 01/05/14 16:20, Alice Heaton wrote:
> On 01/05/14 14:33, Alice Heaton wrote:
>> I was wondering how ckan handles it's PostgreSQL connections.
>
> So to answer my question, and as far as I can tell, the pooling is 
> handled by SqlAlchemy.
>
> ckan creates one [*] SqlAlchemy engine that is used for querying 
> models, and passes it the main config file option. This means that the 
> pool configuration can be set by setting sqlalchemy.pool_size, 
> sqlalchemy.max_overflow, etc. This pool is shared across threads.

To continue on the topic:

The default setting for the SqlAlchemy pool is to have up to 15 
PostgreSQL connections open (pool_size=5,max_overflow=10).
The default suggested configuration for production is to have 15 threads 
per process.

This suggests that the expected behaviour is that a single thread will 
not use more than one connection at a time (per SqlAlchemy engine).

Is this correct?

In that case, given that we have three engines (with the datastore 
extension) then we shouldn't expect more than 3 connections per thread. 
So, sticking to the defaults, we could have up 3 * 15 * 2 = 90 
concurrent connections to the PostgreSql server.

Thanks,
Alice