[okfn-discuss] data distribution

[Rufus Pollock]

Recently I've been thinking about distributing data, especially large 
amounts of it (I've started an incubator project on OKFN [1]). Recent 
trends in distributing software seem to be towards using a p2p protocol 
such as bittorrent, though it is noticeable that most software download 
is still fairly traditional in being straight file transfer (either via 
http or ftp).

Given that for 'knowledge' we are often talking about chunks of data 
that are sizeable compared to normal software (perhaps equivalent to a 
linux distro or bigger) it seems sensible to go down this route. I don't 
yet know much about this and my experience with BT only extends to 
downloading a few distros -- I have never distributed using it -- so I 
would welcome any comments people had about this (I have read [2]). Some 
of the things i'd particularly like information on are:

   1. Mirroring. By the nature of BT I assume you can't explicitly 
select a mirror as you often do when downloading in the traditional 
manner so to what extent does BT do mirroring for you?

   2. Is there any way to do fallback in BT so that clients which can't 
support BT can download normally? If not is this an issue?

   3. Any tips on setting up a tracker (or should one use trackerless 
torrents)?

[1] http://www.okfn.org/wiki/DataDistribution
[1] http://www.bittorrent.com/guide.html