[ckan-dev] key value store, caching and redis

Fri Jan 28 23:42:11 UTC 2011

Hello

There is a need for ckan plugins to have a place to store things.
http://ckan.org/ticket/934

There are 3 ideas on how to achieve this:

 1.  Let plugins make their own sql tables.

Pros.
     Greatest flexibility.   Give power over completely to the plugin.
Cons
     Migration issues.  Different instances will have different schemas.
Will definitely need lot of manual work every db upgrade.
     There is nothing to stop a plugin from doing this already in its own
database/key value store of choosing.  It will have to handle its own db
upgrades.

 2.  Make a key value table.  This table will have essentially 3 columns.
Namespace, key, value.  The value being a serialised json object.  The
namespace will denote what plugin owns that particular row.

Pros
     Flexible enough for most needs.
     Simple to make.
Cons
     Serializing json in dbs is not great practice.
     Data would be messy to handle

3.  Use redis as key value store.  Keys can have their own namespace above.
This can be optional as a config setting (but obviously needed if a plugin
required it)

Pros
     Simple.
     Data store suited to task.
     Everything it did would be fast.
     Atomic operations easy i.e counters, queues
     Plugins could do many more things, without the need to manage own
database. i.e there could be a caching plugin, a pubsub plugin, a plugin
that stored the last 10 packages a user viewed.
Cons
    All stored in memory
    Another daemon process to run.  (even though Ubuntu has an upto date
version in its repositories)
    If used for caching and persistent data at the same time we will have to
deal with durability/speed compromises. see
http://redis.io/topics/persistence

I personally would go with redis.  I think we need a rethink how we do
caching at the moment and this could be the way to do it.

However, I could well be biased, so I want as much feedback as possible
before I go ahead with it.

Thanks

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110128/b70f4e98/attachment-0001.html>