[okfn-discuss] Distributed Storage: Suggestions?

[Rufus Pollock]

2009/4/23  <julian at goatchurch.org.uk>:
> Quoting Lukasz Szybalski <szybalski at gmail.com>:
>
>> I guess the question would be: Could you describe the
> type of data you
>> currently have. (percentage of space, downloads, changes)
>>
>
> This is the directory that has broken the system (watch
> out-- it may break your browser):
>
> http://ukparse.kforge.net/svn/undata/pdf/
>
> It's several thousand large PDFs of UN documents.  The
> same would apply to scanned images, archived pages from
> Hansard, etc.

To add to this: we're also looking to have a way to store CKAN listed
materials. These range from a few 100k to 100s of MBs or GBs.

To answer your specific question:

Space: (for a start) 50-500GB.

Downloads: probably fairly light -- at least to start with this is
more of a caching service (We're not releasing the latest linux
distro)

Changes: Again initially fairly light (we'll mostly be adding
immutable files). In the long run it would be really nice to have
versioning support but I think this will be implemented on top of the
storage system rather than part of it (at least initially).

Rufus