[openspending-dev] Version 0.14 - Celery queues

Thu Jun 19 15:53:36 UTC 2014

Hi all,

I just wanted to tell you about a new (small) version of OpenSpending I
rolled out today, version 0.14.
It included just one new, small but quite an important change (which
won't be deployed until next Monday).

The celery setup now routes tasks to three different queues:

* celery
* analysis
* loading

The celery queue is the default queue in celery which we keep around for
generic tasks like the ping resource (which outputs a pong in the celery
logs). For now that's the only task bound to that queue.

The analysis queue takes care of the source analysis (grabbing the
headers from the csv files)

The loading queue takes care of the actual loading of the csv files into
the database.

The reason for this split is that sometimes when big csv files were
being loaded they hogged the queue and so the faster analysis tasks
would have to wait for hours and hours (ok I'm exaggerating a bit). This
split allows us to create dedicated analysis workers and separate
loading workers so even if a lot of loading of csv is happening the
analysis jobs just go elsewhere and get taken care of by other backend
workers.

This is therefore backwards incompatible since we have to start the
workers differently and name the queues they should pick tasks from (and
the reason we won't deploy until next Monday since that's the time when
I and the sysadmins have time to sit down and reconfigure the backend
accordingly).

-- 

Tryggvi Björgvinsson

Technical Lead, OpenSpending

The Open Knowledge Foundation <http://okfn.org>

/Empowering through Open Knowledge/

http://okfn.org/ | @okfn <http://twitter.com/OKFN> | OKF on Facebook
<https://facebook.com/OKFNetwork> | Blog <http://blog.okfn.org/> |
Newsletter <http://okfn.org/about/newsletter>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20140619/e8c16f8c/attachment.html>