[ckan-dev] Database connections per worker

Thu May 1 15:20:19 UTC 2014

On 01/05/14 14:33, Alice Heaton wrote:
> I was wondering how ckan handles it's PostgreSQL connections.

So to answer my question, and as far as I can tell, the pooling is 
handled by SqlAlchemy.

ckan creates one [*] SqlAlchemy engine that is used for querying models, 
and passes it the main config file option. This means that the pool 
configuration can be set by setting sqlalchemy.pool_size, 
sqlalchemy.max_overflow, etc. This pool is shared across threads.

The datastore extention creates three engines (one for ckan db, one for 
datastore read db, and one for datastore write db). Unfortunately the 
datastore extention does not pass any settings other than the connection 
URL, so these connections use SqlAlchemy default pool settings 
(pool_size=5, max_overflow=10). The pools are shared across threads.

Is this correct? That would mean (with default setup) at most 15 
connections (from ckan) and (regardless of setup) 45 connections (from 
datastore) per process.

Any ideas on how to change the Datastore pool settings?  Any other 
thoughts/info on the topic is welcome :-)

Thanks,
Alice

[*] Actually in my setup it creates two engines, but the second one 
isn't actually used. Not sure if this is an issue in my setup or otherwise.