Hi!
On Thu, 2023-11-02 at 15:27:54 +0000, Michael Hudson-Doyle wrote:
> On Tue, 31 Oct 2023 at 09:21, Guillem Jover wrote:
> > This can be used among other things to set up foreign chroots, by
> > running the host tools, so disallowing installation could be
> > problematic. Even though I guess there could be a warning about this,
> > or maybe it could be controlled through a force option, although both
> > seems like they could be disruptive.
> Of course in such cases dpkg knows that something funny is going on and
> could suppress the warning itself.
Right, also true.
> I spent a few minutes trying to think hard about this and I honestly don't
> think I can predict whether trying to prevent installation of incompatible
> packages is worth it (after all one of the ways users could get into
> trouble would be moving an installed system to a different CPU and having
> binaries start to fail and obviously dpkg can't help there).
>
> One result of this thinking was: I had been thinking/assuming the issue of
> which variants to consider would be apt configuration, but maybe dpkg
> configuration would make more sense (after all, --add-architecture is a
> parameter to dpkg). And in this case, dpkg could object when installing a
> variant that has not been configured.
Yes, the "plan" has been to add support for run-time CPU attributes,
so that when adding a new arch, for example you can specify whether
that arch is runnable, which could help dpkg decide whether to allow
by default to install M-A:foreign packages.
Ah. Would this be a change to /var/lib/dpkg/arch or an additional file or ...?
I guess this is similar, so such future interface should probably take
this into account as something to support too. Will check where this
is tracked and add a note to it.
Did you find this place? :)
And of course that is fine as a guardrail, but if a user hit that out
of running a frontend, then that would already be too late, which to
me means that frontends need to be aware of this too (and not pass
packages that dpkg would/could/might refuse to install), when deciding
what to pass to dpkg.
Good point.
But in any case, as you say, this currently would not be worse than
configuring a foreign arch, installing some foreign package and trying
to run it, but it might make it potentially more common. And as
mentioned above the effecting layer this needs to be decided up seems
higher anyway (even if dpkg could provide the infra for it).
> > If the only change in the package filename format is in the <arch> part
> > where we'd use a name which would otherwise be valid as an arch name (so,
> > no weird symbols, or «-» separators that are not intended to split <os>
> > and <cpu> or similar), then using a name for the variant/ISA would be
> > fine.
> Right. I think that (when possible pretending e.g. "amd64v3" is a distinct
> architecture will generally make things easier. E.g. I think britney
> wouldn't need to know about the relationship between "amd64" and "amd64v3".
I guess that depends on whether the intention is to create a full
optimized archive, or just a partial overlay one. In the latter case
then it might need to know to be able to satisfy dependencies.
Maybe! Depends on details I think.
> > That would be one solution yes, which could give automatic bijective
> > mappings, although ideally with a machine-readable way to get at it,
> > which I'm not sure we have currently.
> I think "gcc -Q --help=target | grep -e '^\s*-march'" is about as machine
> readable as it gets currently, for better or worse (mostly worse).
That does not look very satisfactory, though.
Agreed!
And llvm/clang does not support it. :/
Ah I did not know that.
> > For example code in dpkg-dev
> > already runs «$CC -dumpmachine» to infer the host architecture to use
> > during builds.
> >
> > While using a triplet variation could be a way to do that, that would
> > require such triplet support for each variant/ISA, which tends to be
> > very painful to introduce if it's not there already, so I'd not
> > consider this specific way a viable option.
>
> I admit I'm not an expert on triplet intricacies but I think a new triplet
> is not appropriate here (a bit like a new Debian architecture for a
> variant/ISA choice is not the right concept).
We have i386 or arm (?) as (bad IMO) examples where the triplet can
define the arch baseline. The problem is that this requires updating
the GNU config.git upstream, and then getting that to trickle down into
every package that might be using autotools and not using autoreconf
at build time, or to even update triplet matches in configure scripts
and similar, which might be "acceptable" for a new arch, but seems
disproportionate for a new ISA, so yes, as mentioned I agree it's not
viable.
OK. Let's stop worrying about that then :)
> > On Thu, 2023-09-21 at 14:43:42 +1200, Michael Hudson-Doyle wrote:
> > > * Should the ISA influence the toolchain via toolchain defaults or
> > > dpkg-buildflags?
> > > * How is the default ISA for a buildd chroot selected?
> >
> > So the clear downsides of either modifying the default toolchain or
> > having to provide an additional one is that this seems pretty heavy
> > weight. Also because people might want to build optimized variants
> > locally w/o having to mess with their already existing toolchains.
> > (I'm not sure whether something going along the lines of
> > <https://git.hadrons.org/cgit/debian/fakecross.git> could be an
> > option, although as mentioned above, if that would imply new triplets,
> > then probably not.)
> >
> > So the easiest way might indeed be by controlling this via an envvar,
>
> DEB_HOST_ARCH_ISA?
Yeah, that works, and follows the current DPKG_*_ARCH_ABI lead for
example.
> > which dpkg-buildpackage could also setup internally via a new option,
> > say --arch-isa=amd64v3 or similar
> --host-arch-isa would be more coherent I think.
Ah absolutely! For some reason had --arch in mind as a valid option
(I only see it now in dpkg-scanpackages :D, or maybe I was thinking
about --host-isa :).
> I guess one could add support for --target-host-arch-isa to build a
> toolchain that defaults to a particular ISA. But well.
Yes, the ISA support in dpkg should be extensive enough (so that if
this needs to be supported in the toolchain, then it is possible).
> So to summarise, here are the generic changes that I think need to be made
> to src:dpkg to support variant ISAs as a thing:
>
> * add get_host_arch_isa() to Dpkg::Arch
Yes (perhaps as mentioned below also just get_host_isa()).
> * dpkg-gencontrol records DEB_HOST_ARCH_ISA into DEBIAN/control as
> ArchitectureIsa
Probably better Architecture-Isa, otherwise the current automatic
case folding would make it end up as Architectureisa.
Heh OK.
> * dpkg-architecture emits DEB_HOST_ARCH_ISA and grows --host-arch-isa flag
Also DEB_BUILD_ARCH_ISA and DEB_TARGET_ARCH_ISA, and also grows a
--target-arch-isa (but I'm thinking whether the shorter --host-isa would
be nicer, for example the --match-bits are not spelled --match-arch-bits,
which would seem also a bit redundant).
> * dpkg-buildpackage passes --host-arch-isa to dpkg-architecture
Yes, but only when not the baseline.
I think what I meant here was that if you pass one of the --*-arch-isa flags dpkg-buildpackage, it gets passed along to dpkg-architecture (as --host-arch etc are).
> * dpkg-genchanges should record the ISA in the changes file somehow I
> guess?
Yes, also dpkg-genbuildinfo.
Oh yeah. Although on a quick look dpkg-genbuildinfo gets the architecture from the filename...
This could be done either from the
envvars, or perhaps through the debian/files attributes support. But
given that this is supposedly build global (I think it would be rather
weird to end up with a .changes including say _amd64.deb and
_amd64v3.deb file references from the same build),
Ah yes that's true.
then probably using
the envvar might be the better way.
> * dpkg-deb records the ISA in the file name
Yes.
> Have I missed anything?
Nothing else comes to mind right now (except what I might have already
added).
Well of course I wrote this question in before going back and adding the stuff about having this all be driven by dpkg --add-archiitecture isa amd64v3 or anything like that. So those changes probably need to be hashed out too?
> (Hmm does anything need to reject unknown values
> found in DEB_HOST_ARCH_ISA / --host-arch-isa? Probably...)
I guess it indeed makes sense to define what ISAs are supported, and
either error out or warn and ignore such values. So there might be a
need to add something like a new data/isatable.
I notice that --add-architecture doesn't seem to do any checking.
> Conceptually slightly separately, it might make sense to add a build
> "feature" to Dpkg::Vendor::Debian to allow setting -march (and -mtune?)
>
> Then when we want to add support to an ISA, we add a little thing to
> set_build_features (in either Vendor::Debian or Vendor::Ubuntu or wherever)
> that maps get_host_arch_isa() to values for the march-influencing feature.
Hmm, right, how to hook this. I'm not sure the current interface is good
enough to describe this via build flags features, because such new feature
area would expose arch-specific features. I have been thinking through
the build flags and will try to send a proposal/RFC to revamp parts of
it during the weekend.
Did that happen? I didn't see if if so.
But I think the ISA stuff is better just handled
(at leas for now) directly by injecting whatever flags when the requested
ISA is different to the baseline.
Well that's easy enough too.
Cheers & thanks for the continued thoughts,
mwh