Re: apt PARALLELISM
On Mon, Dec 12, 2005 at 12:52:09PM +0100, Goswin von Brederlow wrote:
> Wouter Verhelst <wouter@debian.org> writes:
> > True, but that's not what's being asked here. If multiple URLs could
> > serve requests for a single repository---i.e., if you've got both
> > deb http://ftp1.CC.debian.org/debian unstable main
> > and
> > deb http://ftp2.CC.debian.org/debian unstable main
> > in your sources.list, then apt will download everything from
> > ftp1.CC.debian.org (bar those files it gets an error on at that server;
> > in that case, it will download them from the second server).
> >
> > Which is, indeed, silly.
>
> Unless you have
>
> file://mnt/mirror/debian unstable main
> http://ftp.de.debian.org/debian unstable main
> http://ftp.debian.org/debian unstable main
>
> where you certainly want to get all files locally and only fall back
> to the web on errors and there also first use the german mirror and
> only fall back to ftp.d.o on error.
>
> If parallelism like this gets added then there has to be a way to
> enable/disbale it specifically. It is not generaly a good thing.
I'd rather think you'd only want to parallellize per protocol. I.e.,
only get stuff from http:// URIs if you can't get them from file://
URIs, etc.
On your point with one mirror being farther away than other mirrors, I
don't think that's such a problem; this could be handled by a smart
enough algorithm to distribute packages to the fetchers. For example,
first sort them so that large packages are at the end; then start
downloading one package from each mirror, measuring the time it takes to
perform that download; and when you have enough data to be sure, give
the larger packages to the faster mirrors, and the smaller packages to
the slower ones.
Such an algorithm should be more than enough for the common setup. There
will indeed be cases where that isn't going to be enough (i.e., you've
got two mirrors that are approximately as fast, but one is on the other
end of a line where you pay per downloaded byte while the other isn't),
so a way to disable parallellism will still be welcome; but I don't
think it needs to remain disabled by default.
That being said, I don't have either the time or teh skillz to implement
all this, so I'll shut up now.
--
.../ -/ ---/ .--./ / .--/ .-/ .../ -/ ../ -./ --./ / -.--/ ---/ ..-/ .-./ / -/
../ --/ ./ / .--/ ../ -/ ..../ / -../ ./ -.-./ ---/ -../ ../ -./ --./ / --/
-.--/ / .../ ../ --./ -./ .-/ -/ ..-/ .-./ ./ .-.-.-/ / --/ ---/ .-./ .../ ./ /
../ .../ / ---/ ..-/ -/ -../ .-/ -/ ./ -../ / -/ ./ -.-./ ..../ -./ ---/ .-../
---/ --./ -.--/ / .-/ -./ -.--/ .--/ .-/ -.--/ .-.-.-/ / ...-.-/
Reply to: