Re: ftp.debian.org: consider switching to merged pdiffs
On Wed, 13 Feb 2019 17:32:24 +0100 Julian Andres Klode <jak@debian.org>
wrote:
> Package: ftp.debian.org
> Severity: wishlist
>
>
> I'd like to reopen the discussion about the pdiff format on
> the archive. Currently a pdiff is generated for each generation
> of the archive, which means that apt has to fetch 4 pdiffs per
> day it has to catch up.
>
> This means that for a 10 day interval, we have to fetch 40 pdiffs
> per index. Assuming amd64+i386 with Contents files and Sources
> enabled, we are looking at 2*(1+1+1)*40=6*40=240 files to fetch.
>
> This is clearly suboptimal, as it makes the log output unreadable,
> and causes severe slowdowns on high-latency or non-persistent
> connections.
>
> It might make sense to consider switching to merged pdiffs, which generate
> one Pdiff from each generation to the latest one. This can be done either
> by preserving old index files and creating pdiffs from them, or simply by
> concatenating the new pdiff to the old ones.
>
> A point against it could be increased space requirements and time to
> compress the pdiffs, but I'd welcome more discussion on that subject.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer i speak de, en
>
>
On a related note, the last known blocker for #649882 is solved.
If/when #649882 is implemented, we will have even more PDiffs by default
for apt-file at the trade-off of making the Contents file smaller
(particular multi-arch setups will benefit there). I suspect it will
also off-set any grows in PDiffs being merged server side.
Thanks,
~Niels
Reply to: