Re: apt-get improvement idea
On Do, 2010-06-24 at 17:47 +0430, Mohammad Ebrahim Mohammadi Panah
wrote:
> Hi,
> I've got an idea for apt-get, which I discussed in #debian-devel. I
> want to know what you think about it. Also I need your guidance to
> implement that.
> This is the IRC discussion log:
>
> [16:40] <ebrahim> My idea about APT: let the user add both 'deb
> http://ftp.de.debian.org/debian/ lenny main' and 'deb
> http://ftp.fr.debian.org/debian/ lenny main', and download packages
> from both of them concurrently in a round-robin manner.
> [16:43] <flightplan> ebrahim: you may want to have a look at apt's
> http method or at least how it is being called. Alternatively figure
> out how to use apt-transport-debtorrent package.
> [16:46] <ebrahim> flightplan, I need to change how APT select between
> versions of some package 'pkg', and change how it behaves when some
> version Y of package 'pkg' is available from more than one repo.
> [16:48] <ebrahim> There is no need both sources of the same package be
> of the same protocol. Because I want APT to select the source in a
> round-robin manner, and not to download the same file from both
> sources.
> [16:49] <bremner> you could just hide this under a proxy
> [16:50] <Clint> jamessan: load balancing?
> [16:50] <Corsac> yeah but that means it's not a generic solution
> [16:52] <ebrahim> bremner, why implement such a specific and
> complicated* proxy when I can presumably add it to APT by a small
> change? (complicated because it must be able to load-balance between
> http https rsync ftp ...)
> [16:53] <bremner> well, apt is core to debian and getting your
> "simple" patch in might not be simple
> [16:53] <ol> the flip side is why complicate apt with a feature of use
> to probably a small number of people
> [16:54] <ol> not saying apt is or isn't the right place, but there are
> arguments against
> [16:54] <ol> if the mirror's speed is actually the limiting factor,
> you have a much faster connection than I do!
> [16:55] <ebrahim> Feasibility: It is currently possible to add some
> similar repo's to sources.list. Also APT knows how to download
> concurrently from different servers. I just need to tell APT not to
> always select the first source in case of more than one source for
> that version.
> [16:56] <ol> does it currently always select the first?
> [16:56] <ebrahim> ol, no. the problem is I can get more bandwith from
> my network by setting up more TCP connections.
> [16:56] <ebrahim> ol, it did last time i checked.
> [16:56] <ol> hmm, I've tried this (hoping for such fallback behaviour)
> and didn't get it
> [16:57] <ol> because the fastest mirror for me is sometimes a bit
> unreliable at actually having all the packages...
> [16:57] <ol> ah, perhaps if apt believes the contents of Packages...
> [16:58] <bremner> ari: more than you wanted to know about the racket
> conversion process
> http://www.cs.unb.ca/~bremner/blog/posts/convert-racket/
> [17:00] <ebrahim> Now does anybody know where in APT source code I
> should implement that?
> [17:01] <waldi> ol: 70MiB/s is no problem for several mirrors
> [17:02] <Corsac> the thing is, the whole path has to be checked
> [17:02] <ebrahim> ol, download acceleration through more TCP
> connections is a well-know technique. It is not just me.
> [17:03] <ebrahim> I chose to connect to more servers rather than
> having more connections to the same server, for the sake of Debian
> mirror servers! :)
> [17:31] <mrvn> ebrahim: If you patch it then please add an priority
> option (as in deb [pri=<N>] url suite component). Make it
> round-robin only between sources with equal priority and default to
> the line number (or something) so the old behaviour remains.
> [17:31] <ebrahim> mrvn, nice idea :)
> [17:32] <mrvn> ebrahim: FYI, the experimental / RCS version of apt
> already has the parser for the options in it. You just need to add the
> key and store the value somewhere.
>
If you can provide a patch that does not break API, sure. I guess you
would need to patch at least AcqArchive::QueueNext in
apt-pkg/acquire-item.cc and maybe pkgAcqArchive::pkgAcqArchive in the
same file to select a different source (AcqArchive::QueueNext seems to
select the source in the for loop, where it queries the version files,
i.e. the relation version<->packagefile).
You might need to query the items in the pkgAcquire object hold in the
member "Owner" to determine which URI to choose. For this, you can use
the iterator beginning at ItemsBegin() and ending at ItemsEnd():
for (pkgAcquire::ItemIterator I=Owner->ItemsBegin(); I != Owner->ItemsEnd(); I++) {
...
}
That should be all.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
Reply to: