Re: apt-fast

[Julian Andres Klode <jak@debian.org>]

On Thu, May 30, 2024 at 07:26:04PM +0200, Marc Leeman wrote:
> Just for the record,
> 
> We use apt-fast on our internal mirror that is sync'd with aptly on a
> nightly basis.

So my question here basically boils down to:

What limits are you hitting and how can we fix them properly?

I'm aware of 2 limitations:

1) You have extremely high RTT and extremely high bandwidth,
   I think it was something like 10 Gbit/s connection with
   multisecond RTT. So we produce a filled pipeline but get
   all answers back in the end.

   I believe that company wrote a custom method for their
   storage system in the end.

2) The mirror:// method has multi-RTT failure mode in the
   apt update use case. I don't yet quite know why exactly
   it tries to connect to the mirrors sequentially. But
   essentially it means we need to move DNS resolution
   and connect() calls into the main process or a factory
   process and then connect to all mirrors in happy eyeballs
   fashion and spawn new workers for mirrors as soon as they
   connected.

   This is a problem basically only in case your DNS or
   connect() for some mirrors time out; e.g. it's behind
   a VPN and connect() simply is not answered rather than
   rejected.

In any case, you are in the prime position to apply a better
workaround than apt-fast: You can use the mirror method for
your internal mirror too.

Configure a mirrors file on a location, e.g. on your file system
(mirror+file:/path/to/file) or a web server (mirror://an-http-server/file;
mirror+https://server/path), one URL per line (possibly with options,
see apt-transport-mirror(1)), then just put that as the URL in
your sources.list.

A little known hack is that most web servers are perfectly fine
answering requests with trailing dots, so

    http://foo/
    http://foo./
    http://foo../

Would make apt open 3 connections to the server foo.

Anyway, I'm curious to hear what the problems are and how we can
fix them properly.
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en