Re: A success story with apt and rsync
On 6 Jul 2003, Goswin Brederlow wrote:
> Doogie is thinking about extending the Bittorrent protocol for use as
> apt-get method. I talked with him on irc about some design ideas and
> so far it looks realy good if he can get some mirrors to host it.
My plans are to require no additional software to be installed on any server.
This means all files will be pre-generated, and mirrored. This also means
that a tracker won't be available on that particular mirror, but the block
checksums will still be available.
> The bittorrent protocol organises multiple downloaders so that they
> also upload to each other and thereby reduces the traffic on the main
> server. The extension of the protocol should also utilise http/ftp
> mirrors as sources for the files thereby spreading the load over
> multiple servers evenly.
What this means is that clients will be able to fetch blocks from normal http
and ftp mirrors. This will be used to start fetching data before connections
have been opened with peers.
> Bittorrent calculates a hash for each block of a file very similar to
> what rsync needs to work. Via another small extension rolling
> checksums for each block could be included in the protocol and a
> client side rsync can be done. (I heard this variant of rsync would be
> patented in US but never saw real proof of it.)
>
> All together I think a extended bittorrent module for apt-get is by
> far the better sollution but it will take some more time and designing
> before it can be implemented.
Also, for better sharing, users will have the option of leaving a running
server on their machines.
Additionally, part of my work will include extensions to the tracker to
support tracker peers, and tracker clusters.
Another extension is which tracker to use. When fetching the .torrent
meta-data, my client will attempt to contact a tracker on the server the
.torrent resides on. If none is found, it'll fall back to the one encoded in
the .torrent. This provides for localization of connections, and better
latency.
Reply to: