[ckan-dev] key value store, caching and redis

Sat Jan 29 11:22:06 UTC 2011

Hi all,

On Sat, Jan 29, 2011 at 12:42 AM, David Raznick <kindly at gmail.com> wrote:
>  2.  Make a key value table.  This table will have essentially 3 columns.
> Namespace, key, value.  The value being a serialised json object.  The
> namespace will denote what plugin owns that particular row.

Whatever we do, I'm against making this a special thing for plugins.
If we do it, it should also hold resource extras, package extras etc.
Again, imo, everything but package_id, package_name should be an extra
(i.e. drop the schema).

FWIW, there is a variant 4: Actually doing SQL. If the forms
refactoring ends up giving us a description of the fields in a
specified form, we could go and do ALTER based on that. Given that we
would want to abstract this and the run-time manipulation/generation
of SQLA objects, this could then also serve as a framework for plugins
to hook into. Such a mechanism could, for example, maintain a
structure log, i.e. an SQL dump of all structural changes which could
be used to migrate instances. It's not a simple solution, but its a
very clean way to get the job done.

>  1.  Let plugins make their own sql tables.

While I think we all agree that this is not a nice way to do it; I
wonder if (2) wouldn't end up with a fairly similar result. My
understanding of the k,v-tables is that they would likely end up to
take the form (obj_type, obj_id, key_name, value [, value_type])
(namespace would otherwise be some mix of the first two, e.g.
package:uuid). So given that I want to create an apps catalogue
plugin, I'd most likely still create a table "application" just to
establish integrity for the first two columns. Given that, we might as
well model things properly.

> 3.  Use redis as key value store.  Keys can have their own namespace above.

I'm sorry, but once we add another storage mechanism, we're in the
middle of the backend debate. We need to have this anyway and perhaps
now is a good time to have it. Just piling another system on top of
the current stack seems like an extreme move, given the complexity in
operations that it introduces.

In this, I'd like to make an argument for mongo, while I'm sure Will
would want to speak for Virtuoso. Shall we go there?

 Friedrich