[okfn-discuss] data distribution

Thu Jan 12 19:29:27 UTC 2006

What specific data have you got that is so large, and who needs access
to it?

I'm doubtful that Bittorrent will be helpful - I believe it only works
if lots of people want to download the same data, which is unlikely
with the sort of sets we'll be working on.

I suspect traditional download mechanisms are fine. Broadband is
getting much faster now, and you can download the amount of data on a
CD relatively quickly. Certainly, download speed hasn't been a
limiting factor for anything in relation to parlparse - processor 
speed and RAM more important, and that only because our parsing
scripts aren't even remotely optimised.

Francis

On Thu, Jan 12, 2006 at 07:23:18PM +0000, Rufus Pollock wrote:
> Recently I've been thinking about distributing data, especially large 
> amounts of it (I've started an incubator project on OKFN [1]). Recent 
> trends in distributing software seem to be towards using a p2p protocol 
> such as bittorrent, though it is noticeable that most software download 
> is still fairly traditional in being straight file transfer (either via 
> http or ftp).
> 
> Given that for 'knowledge' we are often talking about chunks of data 
> that are sizeable compared to normal software (perhaps equivalent to a 
> linux distro or bigger) it seems sensible to go down this route. I don't 
> yet know much about this and my experience with BT only extends to 
> downloading a few distros -- I have never distributed using it -- so I 
> would welcome any comments people had about this (I have read [2]). Some 
> of the things i'd particularly like information on are:
> 
>   1. Mirroring. By the nature of BT I assume you can't explicitly 
> select a mirror as you often do when downloading in the traditional 
> manner so to what extent does BT do mirroring for you?
> 
>   2. Is there any way to do fallback in BT so that clients which can't 
> support BT can download normally? If not is this an issue?
> 
>   3. Any tips on setting up a tracker (or should one use trackerless 
> torrents)?
> 
> [1] http://www.okfn.org/wiki/DataDistribution
> [1] http://www.bittorrent.com/guide.html
> 
> _______________________________________________
> okfn-discuss mailing list
> okfn-discuss at lists.okfn.org
> http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
> 

-- 
Form a long term relationship with your MP... http://www.hearfromyourmp.com