[wdmmg-dev] Where we are with OpenSpending

[Stefan Wehrmeyer]

On 15.06.2011, at 15:38 , Martin Keegan wrote:

>> The more serious issue is that we are having trouble with the task queue, "celery". An option has been added to bypass celery, but this only really works for small datasets; users trying to load large datasets will get a timeout.
> 
> I originally setup celery for openspending. As you noticed yourself, asynchronous processing is necessary and circumventing it can't possibly be a viable solution.
> Can you describe the problem you are having with celery?
> 
> Indeed: circumventing it isn't viable for most cases.
> 
> The problem we're having is that celery seems to get into a state where it stops processing new tasks.
> 
> It looks like the tools for checking celery's status (cli or python) don't work terribly well: ergo we can't pass this information on to end users.

Setting up celery is a bit of a pain. I had a look at the current setup, some pointers:

 - There is an /etc/init.d/celeryd script that reads configuration from /etc/default/celeryd, this might be preferred way to start celery
 - The python module name of the celery configuration file can be set in the pylons ini file (currently "celeryconfig_production"). Make sure it get's the right one, there could be more than one on the Python Path.
 - the loader for celery is "celerypylons.loader.PylonsLoader"
 - There are ways to monitor celery described here: http://celeryproject.org/docs/userguide/monitoring.html (but couldn't get them to work when I just tried)
 - Celery's Pylons integration is not official, I had to write my own fork, but there are also some others: https://bitbucket.org/ianschenck/celery-pylons/descendants
 - IANASA (I am not a sysadmin) and getting the configuration (incl. permissions, virtualenvs etc.) right was difficult. I got it working some time ago, but I guess lots changed.

I put Nils (the sysadmin) in CC. Are uat... and sandbox... the same? Which of them is actually powering the Sandbox now?

I will try to help solve this problem, but can't invest lots of debugging time. We should also put this in the docs, when it's done.

Cheers
Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4261 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20110615/95df2c09/attachment-0001.bin>