[okfn-discuss] Distributed Storage: Suggestions?

Fri May 1 18:22:43 UTC 2009

On Fri, May 1, 2009 at 11:22 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> 2009/4/28 Lukasz Szybalski <szybalski at gmail.com>:
>> a follow up:
>> 1. kosmosfs  www.linux-magazine.com/w3/issue/90/048-051_kosmos.pdf
>> 2. sector http://sector.sourceforge.net/doc.html
>
> Thanks for the info. This one looks interesting but not sure what it
> offers over e.g. Hadoop which seems more mature and widely used.
>
>> 3. hadoop fs
>
> Hadoop looks very promising but seems more "cluster" oriented rather
> than distributed storage oriented. I've collected some more info on
> Hadoop in <http://wiki.okfn.org/projects/Distributed_Storage/Research/>.
>
> BTW: I've now started hacking with Julian to try out what seem to be
> the 2 best options at the moment: Tahoe and Hadoop. We're keeping our
> code and instructions in this public mercurial repo (under dfs):
>
> <http://knowledgeforge.net/okfn/okfncc>
> <http://knowledgeforge.net/okfn/okfncc?file/tip/dfs/>
>
> The main focus is on Tahoe as this looks the most promising option
> (plus it is python based!). We've already got a basic install script
> and are working on laying a more human-friendly interface over the
> basic tahoe DFS structure.

1. sector is for highly "read" intensive apps, optimized for extremely
fast read.
2. kfs(kosmos) seems to be similar to google file system, where you
have main server (metaserver), and run chunk servers (how much space
you have available) , data gets divided into 64mb and send to chunk
servers. You always have 3 copies of the chunk.  The filesystem (at
metaserver) seems like one big file server. (+python bindings)

Lucas

>
> Anyone wanting to get involved in hacking or wanting to volunteer
> storage node space please let me know.

>
> Rufus
>

-- 
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
DataHub - create a package that gets, parses, loads, visualizes data
http://lucasmanual.com/mywiki/DataHub