[ckan-dev] key value store, caching and redis

Wed Feb 2 02:35:05 UTC 2011

] Varnish could be a good solution. I would argue adding varnish adds to the
] complexity of the system more.

I agree with this assessent of varnish. If we were to use its edge-include
stuff we would effectively be moving a good part of the application display
logic into to varnish config. This would greatly complicate what is 
necessary for a testing/development setup (how would we emulate this with
paster and unittests?).

One thing that we learned from our last excursion into caching was that
accellerators like squid in reverse-proxy mode can be easily added in
or taken out of the data flow as long as we pay some attention to the 
HTTP headers -- they are session layer beasts. Whereas varnish is part
of the presentation and application layer and is very tightly coupled
to whatever the back-end service is supposed to be doing.

If we are to go to partial page or query result caching, which can
be very helpful, I would advocate using something like redis (over
memcached since it is more flexible, and over varnish since it is
less complicated and the logic is in python code which we are all
familiar with). Redis is cheap enough that I would also replace the
existing beaker-based cache, or make sure there is a beaker back-end
that will use it.

] Cache invalidation is the age old difficult problem.

Yes.

-w
-- 
William Waites                <mailto:ww at styx.org>
http://eris.okfn.org/ww/         <sip:ww at styx.org>
F4B3 39BF E775 CF42 0BAB  3DF0 BE40 A6DF B06F FD45