[ckan-dev] key value store, caching and redis

Tue Feb 1 11:43:14 UTC 2011

Hi all,

I suspect this thread is now a bit stale and the decision has been
taken, but just in the spirit of being devil's advocate...  It seems
that the principle justification for introducing another dependency
into the system is:

> We do not want to be hitting the database for every resource

Why not?  It's not like CKAN instances are sites that need to
massively scale.  I am sure we can easily accommodate the levels we
typically need with postgres.

The other justification is that it's somehow simpler not to use SQL,
of which I'm not convinced, at least not when we already have to deal
with it anyway.

That said, just to be clear: I'm not really bothered either way :)

Seb

On 31 January 2011 09:41, David Raznick <kindly at gmail.com> wrote:
> On Mon, Jan 31, 2011 at 9:00 AM, Seb Bacon <seb.bacon at gmail.com> wrote:
>>
>> On 30 January 2011 11:53, David Raznick <kindly at gmail.com> wrote:
>> >> Seb said
>> >>As a general point I am no fan of SQL databases
>> >
>> > I funny enough am a big fan sql databases.  I just do not like them
>> > abused.   I like the the way the schema gives you an implicit model of
>> > your
>> > data, that its got rock solid durability and that they can be queried
>> > easily
>> > with a well established standard.  I think this is very important for
>> > valuable data.
>>
>> Careful there with my context :)  I said "no fan... *in our webby world*".
>>
>> In a context where rock-solid durability and high levels of querying
>> are required, I think they're great :)  And arguably this is the case
>> for our package catalogue.  But not really for "I like this"
>> applications.  That's what I was trying to say.  Kind of agreeing with
>> you, but asking if it's worth introducing a new database for.
>>
> I was agreeing with you too :).  Just wanted to make sure that I did not
> come across as having a secret plan to try and move everything over to
> redis.
>
>>
>> > The two questions for me are.
>> >
>> > 1. Will this increase complexity of the system or simplify it?
>> >
>> > For me it simplifies it.  Redis is no harder to set up than say
>> > memcached.
>> > Its *much* easier than something like rabbinmq.
>>
>> Just because we have something quite hard to set up already in our
>> system, doesn't mean adding another thing will simplify it, however
>> easy it is to set up.
>>
>> I take the general point that we already have loads of dependencies.
>> And I don't think that a concern to reduce them should be a limiting
>> factor in a decision on what technology to use.  But I do think we
>> need to be a little bit wary about ensuring our software is easy to
>> understand and deploy.
>>
>> > 2.  Do we need a new solution to caching or storing semi-valuable data
>> > in a
>> > fast way?
>> >
>> > I think we do.  I do not see this as a new database, I see it as
>> > memcached
>> > with some persistence.
>>
>> One thought: will we need to join data across redis and postgres?
>>
> Yes but they can be emulated simply.
>
> Redis stores lists/sets against keys. i.e   key: [package1_id, package2_id,
> package3_id].
> So I cant imagine a case where a simple   "select * from package where
> package_id in (package1_id, package2_id, package3_id)"  will not suffice to
> emulate a join with no performance issues.
>
>>
>> Seb
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>

-- 
skype: seb.bacon
mobile: 07790 939224
land: 0207 183 9618
web: http://baconconsulting.co.uk