Re: apt-fast
I don't think it matters as I understand why you prefer not to have
this in the wild.
1. Our mirror is located on Azure, which had a number of problems on
its own (some RFC that is not followed properly on the infrastructure
wrt to TCP connections, Acquire::http::Pipeline-Depth 0;).
2. The real reason I am using apt-fast is an effort to speed up builds
(also on Azure slaves): we have around 100 different systems where
nightly and production builds are done for complete file systems; part
of the build is fetching all the packages before loading them in the
Debian installer. All in all; this is only a very minimal part of the
entire build process (creating installer, installing, patching deb for
A/B installation; about part of 20% of the entire process). There was
some gain but not to the extent I hoped for. I wrote an article if you
want to know more about the setup [1].
I'll try your suggestions and see what the effect is.
[1] https://www.televic.com/en/business-units/televic-group/news/image-based-upgrades-with-debian
On Thu, 30 May 2024 at 20:48, Julian Andres Klode <jak@debian.org> wrote:
>
> On Thu, May 30, 2024 at 07:26:04PM +0200, Marc Leeman wrote:
> > Just for the record,
> >
> > We use apt-fast on our internal mirror that is sync'd with aptly on a
> > nightly basis.
>
> So my question here basically boils down to:
>
> What limits are you hitting and how can we fix them properly?
>
> I'm aware of 2 limitations:
>
> 1) You have extremely high RTT and extremely high bandwidth,
> I think it was something like 10 Gbit/s connection with
> multisecond RTT. So we produce a filled pipeline but get
> all answers back in the end.
>
> I believe that company wrote a custom method for their
> storage system in the end.
>
> 2) The mirror:// method has multi-RTT failure mode in the
> apt update use case. I don't yet quite know why exactly
> it tries to connect to the mirrors sequentially. But
> essentially it means we need to move DNS resolution
> and connect() calls into the main process or a factory
> process and then connect to all mirrors in happy eyeballs
> fashion and spawn new workers for mirrors as soon as they
> connected.
>
> This is a problem basically only in case your DNS or
> connect() for some mirrors time out; e.g. it's behind
> a VPN and connect() simply is not answered rather than
> rejected.
>
> In any case, you are in the prime position to apply a better
> workaround than apt-fast: You can use the mirror method for
> your internal mirror too.
>
> Configure a mirrors file on a location, e.g. on your file system
> (mirror+file:/path/to/file) or a web server (mirror://an-http-server/file;
> mirror+https://server/path), one URL per line (possibly with options,
> see apt-transport-mirror(1)), then just put that as the URL in
> your sources.list.
>
> A little known hack is that most web servers are perfectly fine
> answering requests with trailing dots, so
>
> http://foo/
> http://foo./
> http://foo../
>
> Would make apt open 3 connections to the server foo.
>
> Anyway, I'm curious to hear what the problems are and how we can
> fix them properly.
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer i speak de, en
--
g. Marc
GPG: 827C FD74 BA46 8152 A041 F3A0 7A6A 4F17 5995 A65B
Reply to: