Re: consolidate linux-libc-dev headers
Hi Dimitri and Bastian,
On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:
> Currently in both Debian and Ubuntu we ship like close to 40 .deb packages
> that have linux-libc-dev (for native/multiarch, cross, ports, mipses).
>
> Each one of them is like 1.5MB, however they actually all have the same and
> repeated content.
>
> the bulk of headers are the same on all arches (and enforced via
> multi-arch:same), asm-generic is also the same for all of them, and then
> kernel-arch asm headers are unique - but only for given kernel arches (not
> all of the debian arch names).
>
> Currently there are only 13 kernel archs for unique asm-arch headers, but
> in Debian we have multiple reuses of the same kernel archs with different
> userspace abi properties (arm = armel armhf, mips = 12 mips archs, powerpc
> = 4, x86 = 3, and so on).
>
> It seems wasteful on disk, build-time, and derivative build time to package
> all of these separately. Especially since there is a chain of src:linux
> building native ones + linux-source as a .deb; which is then used by
> cross-toolchain-base{,-ports,-mips} to unpack and rebuild linux headers
> again, and publish. When in practice src:linux itself could build an
> efficient libc-dev packages for all arches as an arch:all package and
> Multi-Arch:foreign.
Thanks for working on this. I would have appreciated copying
debian-cross@lists.debian.org as I am not following d-kernel@l.d.o.
> The linux-libc-dev is native & multiarch uapi headers for all arches. The
> linux-libc-dev-alpha-cross is all debian arch crosses. It is implemented
> using hardlinks to maintain all the same paths that are currently being
> used by all the arch:any linux-libc-dev packages, and the
> linux-libc-dev-*-cross packages. In total they provide equivalent
> functionality as 40+ linux-libc-dev* current debs in either Debian or
> Ubuntu.
> Is this something that the debian kernel team could consider supporting /
> building as either standalone source package, or out of src:linux directly?
> as this would reduce 40+ .deb duplication in the mirror pools of the same
> content; and will eliminate build-depends on linux-source (and thus
> built-using) from src:cross-toolchain-base* and will actually ensure that
> all kernel headers for native / multiarch / cross arches are updated
> simultaneously, without need to wait for all arch:any builds to catch up
> first. It also gives ability to trivially create freestanding toolchains to
> any of the arches without actually building a full debian port for or
> having a working kernel for a given port.
I agree with this in principle, but I see a number of shortcomings and
hope you can address them.
When creating a new port, linux is one the pieces that comes first.
Hence building a new linux-lib-dev package was a natural with custom
changes was a very natural thing to do. Until now, we never needed to
build architecture-independent packages, so this is a noticeable
deviation. I'll be looking to support this in rebootstrap.
An example where this kinda fails is musl-linux-any. I'm still vaguely
trying to bootstrap this even though it kinda is stuck really hard on
musl vs systemd being unable to cooperate in any meaningful way. I'm not
sure it is reasonable to add these links the regular linux-libc-dev
package as 99% of people will never use them.
The same goes for esoteric architectures such as sh3, s390 (without x),
arc, loong64, and others.
I guess that the list of architectures will always be incomplete for
some, so we probably still need a process for building a modified
linux-libc-dev package only. This probably requires some build profiles.
Is pkg.linux.nokernel pkg.linux.notools pkg.linux.quick a sensible set
of profiles for this? Is there an easily patchable way to add an
architecture?
Let me also summarize how I see this change affecting various
infrastructure.
Bootstrapping will have to branch on whether the default linux-libc-dev
is usable and for many architectures it will be usable thus speeding up
bootstrap there.
For cross building we completely eliminate multi-arch skews of
linux-libc-dev that used to happen every so often. Cool.
For c-t-b, I guess that we can simply cut out linux-libc-dev and remove
all the -cross packages. I hope there is no devil in the detail.
So if the shortcomings end up having viable solutions, this seems like a
rather positive change to me.
Helmut
Reply to: