[ckan-dev] Improving CKAN performance

Mon Sep 22 13:39:01 UTC 2014

Hi,

Please find our setting below. Feel free to comment on our settings, I'm 
not sure
if they are optimal, however, running...

uWSGI + Nginx see: 
http://blog.talatchaudhri.net/2014/03/27/uwsgi-nginx-ckan/
Looks in general better applicable for heavy-python applications 
(instead of mod_wsgi
and Apache). However, we have no experience.

Postgres comes with (only) 100 connections, see enhancements below.

Experiences with Varnish can be reported in the next weeks...

Other readings:
http://engineering.hackerearth.com/2013/11/21/scaling-python-django-application-apache-mod_wsgi/
https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives

Best wishes,
Lothar

Configuration for two servers for CKAN
A) Ubuntu with 4 GB memory, 2 Cores(development)
B) Ubuntu with 16 GB memory, 8 Cores  (production, 
suche.transparenz.hamburg.de)
Both: Apache with mod_wsgi

Configuration options:
1) Jetty
2) Apache
3) VHost ckan_default
4) DB-Connections in pool.py
    (I don't know how to configure parameter pool_size, thus, changing 
the default value).
5) Postgres max_connections

===================
A)
1) Jetty:
/etc/init.d/jetty
   JAVA_OPTIONS="-Xms1g -Xmx3g -Djava.awt.headless=true 
-DentityExpansionLimit=100000"

2) Apache:
/etc/apache2/apache2.conf
   Worker MPM:
<IfModule mpm_worker_module>
     StartServers          6
     MaxClients          500
     ServerLimit         10
     MinSpareThreads      25
     MaxSpareThreads      450
     ThreadLimit          500
     ThreadsPerChild      50
     MaxRequestsPerChild   0
</IfModule>

3) VHost ckan_default
/etc/apache2/sites-available/ckan_default
    WSGIDaemonProcess ckan_default display-name=ckan_default processes=2 
threads=120
    #WSGIProcessGroup ckan_default

--> Yes, "#", because embedded mode was faster.

4) DB-Connections in pool.py
/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py
pool_size increased from 5 to 50.

class QueuePool(Pool):
...
def __init__(self, creator, pool_size=50, max_overflow=10, timeout=300, 
**kw):

5) Postgres max_connection
/etc/postgresql/9.1/main/postgresql.conf
max_connections = 120

If the configuration of the connections is too high for the machine,
starting postgres brings up an error related to "shared Memory".

6) Postgres shared memory :
33554432 bytes
cat /proc/meminfo
sysctl -a | grep shm
$ sysctl -w kernel.shmmax=33554432     ;;Bytes

see: http://leopard.in.ua/2013/09/05/postgresql-sessting-shared-memory/

B)
1) Jetty:
/etc/init.d/jetty
   JAVA_OPTIONS="-Xms1g -Xmx5g -Djava.awt.headless=true 
-DentityExpansionLimit=100000"

2) Apache:
StartServer 50
MinSpareThreads 2
MaxSpareThreads 2500
ThreadLimit 2500
ThreadsPerChild 50
ServerLimit 1300
MaxClients 1300

3) VHost ckan_default
    WSGIDaemonProcess ckan_default display-name=ckan_default processes=8 
threads=500
    WSGIProcessGroup ckan_default

--> Yes, without "#". Daemon-mode was faster. Python processes work in 
separate process from apache

4) DB-Connections in pool.py
/usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/pool.py
pool_size increase from 5 to 800

5) Postgres max_connection
/etc/postgresql/9.1/main/postgresql.conf
max_connections = 1000

6) Postgres shared memory:
cat /proc/meminfo
sysctl -a | grep shm
$ sysctl -w kernel.shmmax=17179869184   ;;Bytes

cat /proc/sys/kernel/shmmni
Default: autovacuum_max_workers = 3
/etc/sysctl.conf
kernel.sem = 250 32000 32 6000

Am 18.09.2014 22:40, schrieb Julio Acuña:
> Hi all,
>
> Are there common practices to improve the performance of a CKAN instance?
>
> I'm trying with memcached right now and probably experimenting with 
> Varnish and tunig PostgreSQL, but I wish to know if someone has 
> experience in this matter.
>
> Does anyone have seen a difference in performance using uWSGI + Nginx 
> for example instead of using mod_wsgi?
>
>
> Cheers.
>
>     Julio.
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 
Dr. Lothar Hotz
HITeC e.V., University of Hamburg
Vogt-Kölln-Str. 30, 22527 Hamburg
Tel: 040/42883-2605; Fax: 040/42883-2572
E-Mail: hotz at informatik.uni-hamburg.de
WWW: www.hitec-hh.de
Private page: kogs-www.informatik.uni-hamburg.de/~hotz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140922/a21d7431/attachment-0003.html>