[ckan-dev] Improving CKAN performance

Fri Sep 26 20:30:11 UTC 2014

Thanks Lothar,

I'll try some of your recommendations, specially the Apache part :)

For PostgreSQL I've been checking this resource:
http://thebuild.com/blog/2012/06/04/postgresql-when-its-not-your-job-at-djangocon-europe/
hope you find it useful

I'll start measuring things with JMeter and see how it goes.

Greetings.

On Mon, Sep 22, 2014 at 8:39 AM, hotz <hotz at informatik.uni-hamburg.de>
wrote:

>  Hi,
>
> Please find our setting below. Feel free to comment on our settings, I'm
> not sure
> if they are optimal, however, running...
>
> uWSGI + Nginx see:
> http://blog.talatchaudhri.net/2014/03/27/uwsgi-nginx-ckan/
> Looks in general better applicable for heavy-python applications (instead
> of mod_wsgi
> and Apache). However, we have no experience.
>
> Postgres comes with (only) 100 connections, see enhancements below.
>
> Experiences with Varnish can be reported in the next weeks...
>
> Other readings:
>
> http://engineering.hackerearth.com/2013/11/21/scaling-python-django-application-apache-mod_wsgi/
> https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives
>
> Best wishes,
> Lothar
>
>
> Configuration for two servers for CKAN
> A) Ubuntu with 4 GB memory, 2 Cores  (development)
> B) Ubuntu with 16 GB memory, 8 Cores  (production,
> suche.transparenz.hamburg.de)
> Both: Apache with mod_wsgi
>
>
> Configuration options:
> 1) Jetty
> 2) Apache
> 3) VHost ckan_default
> 4) DB-Connections in pool.py
>    (I don't know how to configure parameter pool_size, thus, changing the
> default value).
> 5) Postgres max_connections
>
> ===================
> A)
> 1) Jetty:
> /etc/init.d/jetty
>   JAVA_OPTIONS="-Xms1g -Xmx3g -Djava.awt.headless=true
> -DentityExpansionLimit=100000"
>
> 2) Apache:
> /etc/apache2/apache2.conf
>   Worker MPM:
> <IfModule mpm_worker_module>
>     StartServers          6
>     MaxClients          500
>     ServerLimit         10
>     MinSpareThreads      25
>     MaxSpareThreads      450
>     ThreadLimit          500
>     ThreadsPerChild      50
>     MaxRequestsPerChild   0
> </IfModule>
>
> 3) VHost ckan_default
> /etc/apache2/sites-available/ckan_default
>    WSGIDaemonProcess ckan_default display-name=ckan_default processes=2
> threads=120
>    #WSGIProcessGroup ckan_default
>
> --> Yes, "#", because embedded mode was faster.
>
> 4) DB-Connections in pool.py
> /usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py
> pool_size increased from 5 to 50.
>
> class QueuePool(Pool):
> ...
> def __init__(self, creator, pool_size=50, max_overflow=10, timeout=300,
> **kw):
>
>
> 5) Postgres max_connection
> /etc/postgresql/9.1/main/postgresql.conf
> max_connections = 120
>
> If the configuration of the connections is too high for the machine,
> starting postgres brings up an error related to "shared Memory".
>
> 6) Postgres shared memory :
> 33554432 bytes
> cat /proc/meminfo
> sysctl -a | grep shm
> $ sysctl -w kernel.shmmax=33554432     ;;Bytes
>
> see: http://leopard.in.ua/2013/09/05/postgresql-sessting-shared-memory/
>
>
> B)
> 1) Jetty:
> /etc/init.d/jetty
>   JAVA_OPTIONS="-Xms1g -Xmx5g -Djava.awt.headless=true
> -DentityExpansionLimit=100000"
>
> 2) Apache:
> StartServer 50
> MinSpareThreads 2
> MaxSpareThreads 2500
> ThreadLimit 2500
> ThreadsPerChild 50
> ServerLimit 1300
> MaxClients 1300
>
> 3) VHost ckan_default
>    WSGIDaemonProcess ckan_default display-name=ckan_default processes=8
> threads=500
>    WSGIProcessGroup ckan_default
>
> --> Yes, without "#". Daemon-mode was faster. Python processes work in
> separate process from apache
>
> 4) DB-Connections in pool.py
> /usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py
> pool_size increase from 5 to 800
>
> 5) Postgres max_connection
> /etc/postgresql/9.1/main/postgresql.conf
> max_connections = 1000
>
> 6) Postgres shared memory:
> cat /proc/meminfo
> sysctl -a | grep shm
> $ sysctl -w kernel.shmmax=17179869184   ;;Bytes
>
> cat /proc/sys/kernel/shmmni
> Default: autovacuum_max_workers = 3
> /etc/sysctl.conf
> kernel.sem = 250 32000 32 6000
>
>
> Am 18.09.2014 22:40, schrieb Julio Acuña:
>
>  Hi all,
>
>  Are there common practices to improve the performance of a CKAN instance?
>
>  I'm trying with memcached right now and probably experimenting with
> Varnish and tunig PostgreSQL, but I wish to know if someone has experience
> in this matter.
>
>  Does anyone have seen a difference in performance using uWSGI + Nginx
> for example instead of using mod_wsgi?
>
>
>   Cheers.
>
>      Julio.
>
>
> _______________________________________________
> ckan-dev mailing listckan-dev at lists.okfn.orghttps://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> --
> Dr. Lothar Hotz
> HITeC e.V., University of Hamburg
> Vogt-Kölln-Str. 30, 22527 Hamburg
> Tel: 040/42883-2605; Fax: 040/42883-2572
> E-Mail: hotz at informatik.uni-hamburg.de
> WWW: www.hitec-hh.de
> Private page: kogs-www.informatik.uni-hamburg.de/~hotz
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140926/fa5edb4f/attachment-0003.html>