[okfn-discuss] Distributed Storage: Suggestions?

Thu May 7 15:13:52 UTC 2009

2009/5/7 Lukasz Szybalski <szybalski at gmail.com>:
> On Wed, May 6, 2009 at 6:42 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>> 2009/5/1 Lukasz Szybalski <szybalski at gmail.com>:
[...]
>>> 2. kfs(kosmos) seems to be similar to google file system, where you
>>> have main server (metaserver), and run chunk servers (how much space
>>> you have available) , data gets divided into 64mb and send to chunk
>>> servers. You always have 3 copies of the chunk.  The filesystem (at
>>> metaserver) seems like one big file server. (+python bindings)
>>
>> OK. Useful info.
>
> What is the structure of the tahoe fs? how is it different from kfs?

I haven't looked at kfs in detail but:

  * Seems to be "cluster-oriented" and GFS-like (it says it is built
on ideas from GFS and can be dropped in to replaces HadoopFS).
  * It is built in C++ with some high-level bindings for python.
  * v0.3

Detailed overview of Tahoe can be found in
<http://allmydata.org/source/tahoe/trunk/docs/architecture.txt>

  * Very much about distributed *file storage*.
  * Focused storing and nodes are expected to highly distributed (so
no cluster/cloud computing here).
  * Data is encrypted (so you can put private data on it)
  * Almost entirely in python
  * v1.4

I should reiterate that what we're looking for is a storage solution
not a cluster (cloud computing) solution. That's why, at the moment,
Tahoe's looking like a much better match.

Rufus