[ckan-dev] key value store, caching and redis

[Rufus Pollock]

On 29 January 2011 11:22, Friedrich Lindenberg <friedrich at pudo.org> wrote:
> Hi all,

I'll also reply to David's mail, but want to respond on specific points here.

> On Sat, Jan 29, 2011 at 12:42 AM, David Raznick <kindly at gmail.com> wrote:
>>  2.  Make a key value table.  This table will have essentially 3 columns.
>> Namespace, key, value.  The value being a serialised json object.  The
>> namespace will denote what plugin owns that particular row.
>
> Whatever we do, I'm against making this a special thing for plugins.
> If we do it, it should also hold resource extras, package extras etc.
> Again, imo, everything but package_id, package_name should be an extra
> (i.e. drop the schema).

I'll come back to use cases in main response to David's mail. Suffice
to say this does not have to be a special thing for plugins but
generally useful (e.g. it could, possibly, be useful for config). I
also think it could be more oriented to things that were key value
like (e.g. download counts, 'watchers') than a generic solution for
every plugin.

> FWIW, there is a variant 4: Actually doing SQL. If the forms
> refactoring ends up giving us a description of the fields in a
> specified form, we could go and do ALTER based on that. Given that we
> would want to abstract this and the run-time manipulation/generation
> of SQLA objects, this could then also serve as a framework for plugins
> to hook into. Such a mechanism could, for example, maintain a
> structure log, i.e. an SQL dump of all structural changes which could
> be used to migrate instances. It's not a simple solution, but its a
> very clean way to get the job done.

I'm a bit lost here. Please clarify :)

>>  1.  Let plugins make their own sql tables.
>
> While I think we all agree that this is not a nice way to do it; I
> wonder if (2) wouldn't end up with a fairly similar result. My
> understanding of the k,v-tables is that they would likely end up to
> take the form (obj_type, obj_id, key_name, value [, value_type])
> (namespace would otherwise be some mix of the first two, e.g.
> package:uuid). So given that I want to create an apps catalogue
> plugin, I'd most likely still create a table "application" just to
> establish integrity for the first two columns. Given that, we might as
> well model things properly.

I agree and think we may have a mix of the two solution. I don't think
we need one exclusively (though happy to be argued with). I do think
the k/v solution is a quick way to get started (I think it is
important to not be too worried about the perfect solution esp when
this is orthogonal to core and it is likely to get refactored at least
once).

>> 3.  Use redis as key value store.  Keys can have their own namespace above.
>
> I'm sorry, but once we add another storage mechanism, we're in the
> middle of the backend debate. We need to have this anyway and perhaps
> now is a good time to have it. Just piling another system on top of
> the current stack seems like an extreme move, given the complexity in
> operations that it introduces.

I agree but will respond on this in main email.

> In this, I'd like to make an argument for mongo, while I'm sure Will
> would want to speak for Virtuoso. Shall we go there?

No, I think that is a separate discussion :)

Rufus