Re: Architecture variants for Debian / Ubuntu
On Wed, Sep 06, 2023 at 11:27:02AM +0200, Guillem Jover wrote:
> Hi!
>
> On Fri, 2023-09-01 at 08:43:55 +1200, Michael Hudson-Doyle wrote:
> > Recently the topic of exploiting newer instructions without dropping
> > support for older machines has come up several times inside Ubuntu
> > engineering. I understand this topic has come up several times in the past
> > for Debian as well, but nothing has really come of it to date.
>
> I also had a chat about this with Matthias Klose (CCed) around 2022-05.
>
> > I've spent a while thinking through the options and coming up with a design
> > and wrote some notes into a wiki page:
> > https://wiki.debian.org/ArchitectureVariants
>
> I think we are already doing 1, 2 and 3. I agree 4 is just wrong. And
> something like 5 is what I suggested to Matthias for Ubuntu when we
> last discussed it as the best way to go about this.
>
> I'm not sure I entirely agree with the requirements you set forth
> though:
>
> - I think such optimized builds might need to be done with "special
> toolchains" (these could simply be wrappers over the host compiler
> passing the appropriate flags via command-line or via specs or
> similar, not necessarily full blown toolchains), passing these via
> something like dpkg-buildflags seems currently unreliable, as I don't
> think we have full coverage in packages (neither for all compilers
> available)? Although it would be better as it would centralize the
> management. (For reference this is in part how rpm handles this:
> https://github.com/rpm-software-management/rpm/blob/master/rpmrc.in)
> - Perhaps that's a limitation from the archive software side, but
> requiring to place the binary packages in the same pool seems
> rather restrictive (it forces different filenames for example).
> - I guess it might be nice for the ISA to be passed down to the
> dpkg tools, but I don't think this is strictly necessary? A
> frontend like apt could also decide based on metadata in say the
> Release file, although not having the actual installed package
> metadata on whether it was a different ISA build or not would make
> its job more inconvenient. In any case I don't have a big issue
> with recording this via dpkg-gencontrol or similar if necessary.
>
> On the specific implementation details:
>
> - Changing the Architecture format (as in adding colons there) seems
> like a non-starter, and I expect that would break lots of things
> (I mean it could be done but I'm not sure it's worth it for this).
> Recording this mostly as a hint than anything else, via another
> field (if necessary at all) I think would be best.
> - As covered in previous discussions, dpkg could (but I don't think
> it's necessary) check whether the .deb is runnable on the current
> hw, but that's tricky as chrootless installs need to be taken
> into account, etc. It should certainly not be part of dependency
> resolution.
> - I'm not fond of having to change the binary package name format
> either for this (name_version_arch.deb) even if at least dpkg
> itself does not care (but I know other tools do care), and
> depending on the format I'd expect things to break (this goes
> back to the shared pool concern).
> - If dpkg-architecture needs to be aware of this, then this might need
> to be auto-detectable from just the current toolchain being used.
>
> Some of the above problems could perhaps be avoided if we introduced
> a concept of architecture aliases/ISAs (similar to what rpm has), which
> would side-step the pool sharing issue, the binary package renaming,
> etc. One big issue with this is that it requires for dpkg to have an
> exhaustive table of all such aliases, and if there's ever a new alias
> added, old dpkg versions need to be updated or they will not understand
> what they match with. So this does not seem ideal either. So I guess this
> is a variation over your proposal, but perhaps this could still be used
> in specific contexts, say only at build-time (but not for dependency
> relationships), for repo management (say binary-arm64v9/Packages.xz),
> or binary package names where the field would specify the actual name
> for the filename, say:
>
> Architecture: arm64
> ArchitectureIsa: arm64v9
>
> or maybe better:
>
> Architecture: arm64
> ArchitectureIsa: v9
>
> resulting in dpkg-deb generating:
>
> binpkg_1.0-1_arm64v9.deb
>
> but targeting arm64. I also think I prefer naming this explicitly as ISA
> variants, if you will, than just architecture variants as that gives
> way too much room (which perhaps we want, but then that has other
> implications over compatibility), and for the field perhaps just Isa is
> better, to avoid the implicit repetition of
> ArchitectureInstructionSetArchitecture :), but that makes it less easy
> to associate both as related.
I have thought more about this and I'm not particularly fond of the
ArchitectureIsa name. While *this specific use case* is a variant of
the architecture instruction set; you could just as well build other
variants such as "compiled with -O3", "compiled with frame pointers",
"compiled with -O0", or other shenanigans (I haven't thought about
others outside compiler flags)
Hence I prefer Architecture-Variant, Subarchitecture, or anything
like that rather than have to invent another field or abuse this
one the next time we want to build a special variant of an architecture
with special optimizations for a special customer or whatnot.
This has stalled a bit, but we really gotta get going on this, we
should really have this working in APT by end of year or something.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Reply to: