[okfn-discuss] Distributed Storage: Suggestions?

Wed Apr 22 12:27:44 BST 2009

Anyone out there have ideas or suggestions about distributed storage.
We have an ever growing need to store sizeable amounts of open
material in a replicated and distributed fashion.

We're particularly interested in methods that would allow others to
"donate" some degree of storage space (e.g. by running storage nodes).

Rufus

## The Background

Ever since we started up Knowledgeforge we've been looking for ways to
have a distributed storage infrastructure which could scale with our
needs. Unfortunately a solution has proved elusive and with several
other projects (CKAN especially) in need of the same thing the
limitations this imposes are becoming increasingly constrictive -- for
example, the bulk of the materials in Knowledgeforge reside in
subversion repositories some of which have now grown to be multi-GB in
size (not a great situation for performance, backup or anything else).

When we lasted look at this issue a couple of years ago there seemed a
dearth of (simple-to-use) tools with which to address this problem.
However, I get the feeling things have been advancing significantly in
the last few years and the time might be right to take another look.