[ckan-dev] key value store, caching and redis

[David Raznick]

On Mon, Jan 31, 2011 at 9:00 AM, Seb Bacon <seb.bacon at gmail.com> wrote:

> On 30 January 2011 11:53, David Raznick <kindly at gmail.com> wrote:
> >> Seb said
> >>As a general point I am no fan of SQL databases
> >
> > I funny enough am a big fan sql databases.  I just do not like them
> > abused.   I like the the way the schema gives you an implicit model of
> your
> > data, that its got rock solid durability and that they can be queried
> easily
> > with a well established standard.  I think this is very important for
> > valuable data.
>
> Careful there with my context :)  I said "no fan... *in our webby world*".
>
> In a context where rock-solid durability and high levels of querying
> are required, I think they're great :)  And arguably this is the case
> for our package catalogue.  But not really for "I like this"
> applications.  That's what I was trying to say.  Kind of agreeing with
> you, but asking if it's worth introducing a new database for.
>
> I was agreeing with you too :).  Just wanted to make sure that I did not
come across as having a secret plan to try and move everything over to
redis.


> > The two questions for me are.
> >
> > 1. Will this increase complexity of the system or simplify it?
> >
> > For me it simplifies it.  Redis is no harder to set up than say
> memcached.
> > Its *much* easier than something like rabbinmq.
>
> Just because we have something quite hard to set up already in our
> system, doesn't mean adding another thing will simplify it, however
> easy it is to set up.
>
> I take the general point that we already have loads of dependencies.
> And I don't think that a concern to reduce them should be a limiting
> factor in a decision on what technology to use.  But I do think we
> need to be a little bit wary about ensuring our software is easy to
> understand and deploy.
>
> > 2.  Do we need a new solution to caching or storing semi-valuable data in
> a
> > fast way?
> >
> > I think we do.  I do not see this as a new database, I see it as
> memcached
> > with some persistence.
>
> One thought: will we need to join data across redis and postgres?
>
> Yes but they can be emulated simply.

Redis stores lists/sets against keys. i.e   key: [package1_id, package2_id,
package3_id].

So I cant imagine a case where a simple   "select * from package where
package_id in (package1_id, package2_id, package3_id)"  will not suffice to
emulate a join with no performance issues.


> Seb
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110131/b55c9573/attachment-0001.html>