[okfn-discuss] Distributed Storage: Suggestions?

[Rufus Pollock]

2009/4/22 Michael Chelen <michael.chelen at gmail.com>:
> tahoe is good for distributed filesystem, all the nodes must be trusted
> though

I don't think that would be big problem for us -- we would only be
hosting open material and my understanding of tahoe was that it can be
configured to replicate chunks a given number of times in classic p2p
fashion.

> what order of magnitude are the storage requirements?

Somewhere in the region 100 GB - 5 TM at the moment -- the large
uncertainty is due to the fact that we are currently not doing a whole
bunch of things because we don't have the capacity and it is not
certain in advance how much capacity they will exactly require (e.g.
hosting datasets from CKAN0.

> another good option might be public p2p distribution, through bittorrent
> hosting sites like vipeers.com or other software derivatives like wuala.com

We though about a bittorrent option originally. Does the BT option
guarantee good persistence (we want stuff to stay around), and how
does it deal with replication, chunking etc

> anyone can contribute storage space & bw by seeding the OKFN torrents, which
> are easy to share or post on sites

Sounds interesting. How does the system control what a given node will store?

Rufus