Re: Bug#1023870: dpkg: Problems in buildds due to slow compression
Hi Guillem,
On 2022-11-13 11:04, Aurelien Jarno wrote:
> Hi,
>
> On 2022-11-13 02:00, Guillem Jover wrote:
> > Hi!
> >
> > On Sun, 2022-11-13 at 00:17:36 +0100, Aurelien Jarno wrote:
> > > On 2022-11-12 22:28, Guillem Jover wrote:
> > > > On Fri, 2022-11-11 at 19:15:59 +0100, Manuel A. Fernandez Montecelo wrote:
> > > > > Package: dpkg
> > > > > Version: 1.21.9
> > > > > Severity: normal
> > > > > X-Debbugs-Cc: mafm@debian.org, debian-wb-team@lists.debian.org
> > > >
> > > > > After some investigation by aurel32 and myself, this was traced back to the
> > > > > commit f8d254943051e085040367d689048c00f31514c3 [2], in which the calculation of
> > > > > the memory that can be used, to determine the number of threads to use, was
> > > > > changed from half of the physical mem to be based on the memory available.
> > > >
> > > > Ah, thanks for tracking this down! I think the problem is the usual
> > > > "available" memory does not really mean what people think it means. :/
> > > > And I unfortunately missed that (even thought I was aware of it) when
> > > > reviewing the patch.
> > > >
> > > > Attached is something I just quickly prepared, which I'll clean up and
> > > > merge for the upcoming 1.21.10. Let me know if that solves the issue
> > > > for you, otherwise we'd need to look for further changes.
> > >
> > > Thanks for providing a patch. I have not been able yet to try it for the
> > > case where we have found the issue, i.e. building linux. However I have
> > > tried to setup a similar environment:
> >
> > > - I took a just booted VM with 4 GB RAM, 4 GB swap and 4 GB tmpfs, and very few
> > > things running on it.
> > > - I filled the tmpfs with 4 GB of random data, which means that after
> > > moving the content of the tmpfs to the swap, 4 GB could still be used
> > > without issue.
> > > - I ended up with the following /proc/meminfo:
> > > MemTotal: 3951508 kB
> > > MemFree: 130976 kB
> > > MemAvailable: 10584 kB
> > > Buffers: 2448 kB
> > > Cached: 3694676 kB
> > > SwapCached: 12936 kB
> > > Active: 3111920 kB
> > > Inactive: 610376 kB
> > > Active(anon): 3102668 kB
> > > Inactive(anon): 606952 kB
> > > Active(file): 9252 kB
> > > Inactive(file): 3424 kB
> > > Unevictable: 0 kB
> > > Mlocked: 0 kB
> > > SwapTotal: 4194300 kB
> > > SwapFree: 3777400 kB
> > > Zswap: 0 kB
> > > Zswapped: 0 kB
> > > Dirty: 0 kB
> > > Writeback: 0 kB
> > > AnonPages: 12960 kB
> > > Mapped: 6700 kB
> > > Shmem: 3684416 kB
> > > KReclaimable: 27616 kB
> > > Slab: 54652 kB
> > > SReclaimable: 27616 kB
> > > SUnreclaim: 27036 kB
> > > KernelStack: 2496 kB
> > > PageTables: 1516 kB
> > > NFS_Unstable: 0 kB
> > > Bounce: 0 kB
> > > WritebackTmp: 0 kB
> > > CommitLimit: 6170052 kB
> > > Committed_AS: 4212940 kB
> > > VmallocTotal: 34359738367 kB
> > > VmallocUsed: 16116 kB
> > > VmallocChunk: 0 kB
> > > Percpu: 2288 kB
> > > HardwareCorrupted: 0 kB
> > > AnonHugePages: 0 kB
> > > ShmemHugePages: 0 kB
> > > ShmemPmdMapped: 0 kB
> > > FileHugePages: 0 kB
> > > FilePmdMapped: 0 kB
> > > HugePages_Total: 0
> > > HugePages_Free: 0
> > > HugePages_Rsvd: 0
> > > HugePages_Surp: 0
> > > Hugepagesize: 2048 kB
> > > Hugetlb: 0 kB
> > > DirectMap4k: 110452 kB
> > > DirectMap2M: 5132288 kB
> > > DirectMap1G: 5242880 kB
> >
> > > With the current version of dpkg, it means it consider 10584 kB are available
> > > (not however that there is 130976 kB of unused physical RAM). With your patch,
> > > it's a bit better, as it would be 123408 kB. Still far less that one the VM is
> > > capable of.
> >
> > Err sorry, the patch was computing the used memory and not the truly
> > available one! The updated patch should do better. :)
>
> Thanks for the updated patch. Computing the available memory manually
> from the above values sounds like it should work, so I'll give it a try.
I have run a build of the linux package on a buildd with your patch, and
I confirm it fixes the issue.
Thanks
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://www.aurel32.net
Reply to: