[okfn-discuss] [okfn-coord] question from the Twitter gallery about Open Data Grid

[Rufus Pollock]

On 4 October 2010 15:43, Schuyler Erle <schuyler at nocat.net> wrote:
>
>>>> I think investigating these other alternatives would be a better use
>>>> of people's time right now [than any more work on/with Tahoe].
>
> What is wrong with using Tahoe-LAFS for OKF's purposes, out of curiosity? I pored over the relevant wiki pages, and found no account of what became of the existing pilot, nor why it was discontinued. The lead developer of Tahoe-LAFS works at SimpleGeo, and (obviously) advocates for it pretty strongly.

Don't get me wrong: I'm really impressed with Tahoe and think they are
a great team and doing a great job at producing a *security and
privacy*-oriented distributed file system.

However, we aren't after privacy or cryptographic security (features
we want are listed at [1]).

[1]:<http://wiki.okfn.org/p/Distributed_Storage#WhatWeWant.28orWhatWeMeanbyDistributed.29>

Some of the issues for us with Tahoe are listed at the top of
<http://wiki.okfn.org/p/Distributed_Storage/Plan>. To expand and
summarize:

* Lack of storage accounting: i.e. a way to see who is using what
amount of storage (and more specifically who is 'owning' which files).
This makes it very difficult to control usage, and more importantly,
ensure the grid is not being 'abused'

More: <http://tahoe-lafs.org/pipermail/tahoe-dev/2009-June/001985.html>

NB: I should say it is more than 9 months since i last looked at Tahoe
in detail but the roadmap for 2.0 (not yet done) still has storage
accounting in it so it doesn't look like it is done yet

* (Related) Difficult to do access-control and operations like file
deletion (how do you deal with someone uploading something
unacceptable to your grid ...?)

More:
  * Access control:
<http://tahoe-lafs.org/pipermail/tahoe-dev/2009-June/001985.html>
  * File deletion:
<http://tahoe-lafs.org/pipermail/tahoe-dev/2009-June/001985.html>

* Encryption: we don't need it and it a) makes installs harder (more
crypto requirements) b) makes other things (like storage accounting)
much harder.

> My ulterior interest in this project, of coursem is that the OpenAerialMap project is going to need some amount of distributed, scalable, eventually consistent blahblahblah as well. Obviously, our objectives are nearly identical to those of the OKFN Data Grid.

Right. I think this -- i.e. blob storage on a scale that inevitably
means 'distributed' -- is a common core piece of infrastructure that
everyone in the 'open' (knowledge/data/content/...) community will
need.

Rufus
-- 
Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/