Hi,
I've got an idea for apt-get, which I discussed in #debian-devel. I want to know what you think about it. Also I need your guidance to implement that.
This is the IRC discussion log:
[16:40] <ebrahim> My idea about APT: let the user add both 'deb
http://ftp.de.debian.org/debian/ lenny main' and 'deb
http://ftp.fr.debian.org/debian/ lenny main', and download packages from both of them concurrently in a round-robin manner.
[16:43] <flightplan> ebrahim: you may want to have a look at apt's http method or at least how it is being called. Alternatively figure out how to use apt-transport-debtorrent package.
[16:46] <ebrahim> flightplan, I need to change how APT select between versions of some package 'pkg', and change how it behaves when some version Y of package 'pkg' is available from more than one repo.
[16:48] <ebrahim> There is no need both sources of the same package be of the same protocol. Because I want APT to select the source in a round-robin manner, and not to download the same file from both sources.
[16:49] <bremner> you could just hide this under a proxy
[16:50] <Clint> jamessan: load balancing?
[16:50] <Corsac> yeah but that means it's not a generic solution
[16:52] <ebrahim> bremner, why implement such a specific and complicated* proxy when I can presumably add it to APT by a small change? (complicated because it must be able to load-balance between http https rsync ftp ...)
[16:53] <bremner> well, apt is core to debian and getting your "simple" patch in might not be simple
[16:53] <ol> the flip side is why complicate apt with a feature of use to probably a small number of people
[16:54] <ol> not saying apt is or isn't the right place, but there are arguments against
[16:54] <ol> if the mirror's speed is actually the limiting factor, you have a much faster connection than I do!
[16:55] <ebrahim> Feasibility: It is currently possible to add some similar repo's to sources.list. Also APT knows how to download concurrently from different servers. I just need to tell APT not to always select the first source in case of more than one source for that version.
[16:56] <ol> does it currently always select the first?
[16:56] <ebrahim> ol, no. the problem is I can get more bandwith from my network by setting up more TCP connections.
[16:56] <ebrahim> ol, it did last time i checked.
[16:56] <ol> hmm, I've tried this (hoping for such fallback behaviour) and didn't get it
[16:57] <ol> because the fastest mirror for me is sometimes a bit unreliable at actually having all the packages...
[16:57] <ol> ah, perhaps if apt believes the contents of Packages...
[16:58] <bremner> ari: more than you wanted to know about the racket conversion process
http://www.cs.unb.ca/~bremner/blog/posts/convert-racket/
[17:00] <ebrahim> Now does anybody know where in APT source code I should implement that?
[17:01] <waldi> ol: 70MiB/s is no problem for several mirrors
[17:02] <Corsac> the thing is, the whole path has to be checked
[17:02] <ebrahim> ol, download acceleration through more TCP connections is a well-know technique. It is not just me.
[17:03] <ebrahim> I chose to connect to more servers rather than having more connections to the same server, for the sake of Debian mirror servers! :)
[17:31] <mrvn> ebrahim: If you patch it then please add an priority option (as in deb [pri=<N>] url suite component). Make it round-robin only between sources with equal priority and default to the line number (or something) so the old behaviour remains.
[17:31] <ebrahim> mrvn, nice idea :)
[17:32] <mrvn> ebrahim: FYI, the experimental / RCS version of apt already has the parser for the options in it. You just need to add the key and store the value somewhere.