[ckan-dev] key value store, caching and redis

[Seb Bacon]

On 30 January 2011 11:53, David Raznick <kindly at gmail.com> wrote:
>> Seb said
>>As a general point I am no fan of SQL databases
>
> I funny enough am a big fan sql databases.  I just do not like them
> abused.   I like the the way the schema gives you an implicit model of your
> data, that its got rock solid durability and that they can be queried easily
> with a well established standard.  I think this is very important for
> valuable data.

Careful there with my context :)  I said "no fan... *in our webby world*".

In a context where rock-solid durability and high levels of querying
are required, I think they're great :)  And arguably this is the case
for our package catalogue.  But not really for "I like this"
applications.  That's what I was trying to say.  Kind of agreeing with
you, but asking if it's worth introducing a new database for.

> The two questions for me are.
>
> 1. Will this increase complexity of the system or simplify it?
>
> For me it simplifies it.  Redis is no harder to set up than say memcached.
> Its *much* easier than something like rabbinmq.

Just because we have something quite hard to set up already in our
system, doesn't mean adding another thing will simplify it, however
easy it is to set up.

I take the general point that we already have loads of dependencies.
And I don't think that a concern to reduce them should be a limiting
factor in a decision on what technology to use.  But I do think we
need to be a little bit wary about ensuring our software is easy to
understand and deploy.

> 2.  Do we need a new solution to caching or storing semi-valuable data in a
> fast way?
>
> I think we do.  I do not see this as a new database, I see it as memcached
> with some persistence.

One thought: will we need to join data across redis and postgres?

Seb