[okfn-discuss] Distributed Storage: Suggestions?

Thu Apr 23 13:41:31 UTC 2009

On Thu, Apr 23, 2009 at 5:43 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> 2009/4/22 Michael Chelen <michael.chelen at gmail.com>:
>> tahoe is good for distributed filesystem, all the nodes must be trusted
>> though
>
> I don't think that would be big problem for us -- we would only be
> hosting open material and my understanding of tahoe was that it can be
> configured to replicate chunks a given number of times in classic p2p
> fashion.
>
>> what order of magnitude are the storage requirements?
>
> Somewhere in the region 100 GB - 5 TM at the moment -- the large
> uncertainty is due to the fact that we are currently not doing a whole
> bunch of things because we don't have the capacity and it is not
> certain in advance how much capacity they will exactly require (e.g.
> hosting datasets from CKAN0.
>
>> another good option might be public p2p distribution, through bittorrent
>> hosting sites like vipeers.com or other software derivatives like wuala.com
>
> We though about a bittorrent option originally. Does the BT option
> guarantee good persistence (we want stuff to stay around), and how
> does it deal with replication, chunking etc
>
>> anyone can contribute storage space & bw by seeding the OKFN torrents, which
>> are easy to share or post on sites
>
> Sounds interesting. How does the system control what a given node will store?

What are you mostly storing right now?

Are we talking about "static data", data with updates that you put into svn?
Do you need a mirror of svn as well?

If it is a static huge files, then bittorrent would sound interesting.
You could host it on your side, and allow mirrors to join in. If we
are talking about small files <1mb then torrent might not be a
solution, but http/ftp mirrors might.

I guess the question would be: Could you describe the type of data you
currently have. (percentage of space, downloads, changes)

Thanks,
Lucas