Re: rust-*/librust-* arch:any vs. arch:all and cross-compilation support
Hi Fabian,
On Wed, Feb 15, 2023 at 08:14:17PM +0100, Fabian Grünbichler wrote:
> > // This is the best but not ideal option for us.
> > //
> > // Currently Debian M-A spec has a deficiency where a package X that
> > // build-depends on a (M-A:foreign+arch:all) package that itself
> > // depends on an arch:any package Z, will pick up the BUILD_ARCH of
> > // package Z instead of the HOST_ARCH. This is because we currently
> > // have no way of telling dpkg to use HOST_ARCH when checking that the
> > // dependencies of Y are satisfied, which is done at install-time
> > // without any knowledge that we're about to do a cross-compile. It
> > // is also problematic to tell dpkg to "accept any arch" because of
> > // the presence of non-M-A:same packages in the archive, that are not
> > // co-installable - different arches of Z might be depended-upon by
> > // two conflicting chains. (dpkg has so far chosen not to add an
> > // exception for the case where package Z is M-A:same co-installable).
> > //
> > // The recommended work-around for now from the dpkg developers is to
> > // make our packages arch:any M-A:same even though this results in
> > // duplicate packages in the Debian archive. For very large crates we
> > // will eventually want to make debcargo generate -data packages that
> > // are arch:all and have the arch:any -dev packages depend on it.
> >
> > https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342
I guess that I was the one giving the advice that was recorded here
although I agree with Guillem that it is not recorded in a very
consumable way.
> > The chain is
> >
> > rust-lscolors (arch:any) --BD--> librust-tempfile-dev (arch:any)
> > librust-tempfile-dev (arch:any) --D--> librust-remove-dir-all-0.7+default-dev (arch:any) --D--> librust-log-0.4+default-dev (arch:any) --D--> librust-value-bag-1.0.0+serde-dev (arch:any) --D--> librust-serde-fmt-1+default-dev (arch:all) --D--> librust-serde-dev (arch:any, but wrongly resolved!)
> >
> > with librust-serde-fmt-dev being switched to arch:all, it will resolve to :amd64 instead of :i386 and in turn pull in its own dependencies librust-serde-1+std-dev & librust-serde-1-dev (both provided by librust-serde-dev) from amd64 (in addition to librust-serde-dev:i386, which is also part of the arch:any (build-)dependency tree of rust-lscolors itself, without the "hop" over librust-serde-fmt-dev).
The analysis is well done. Thank you. This is a real example of why we
do this annoying arch:any+m-a:same dance.
> > TL;DR: Is the switch to arch:all one that should be reverted in the face of it apparently breaking cross builds? Or is there another alternative (nowadays) that makes the "workaround" employed by debcargo no longer needed?
Yes, the switch to arch:all needs to be reverted at least partially.
No, there is no better way to achieve this. We extensively discussed
alternatives in Vaumarcus and reached the conclusion that no possible
interface was acceptable to dpkg. In essence, dpkg could automatically
track the architectures of packages (which would incur a recursive
dependency resolver onto dpkg) or architecture tracking could be a
manual thing, which would be far too annoying to end users. Therefore,
there is no sane alternative to the workaround.
> > Depending on feedback, the rust team would either ask Jonas to switch back his packages to arch:any, M-A:same (probably after bookworm, to prevent further fallout/need for RMs/.. during the freeze), or will evaluate whether switching to arch:all is an option for debcargo-managed packages as well, and which changes on the team tooling side are needed to avoid losing test coverage or increasing friction.
Technically, not all packages need to become arch:any+m-a:same. As you
correctly observed, the problem arises from non-rust dependencies in the
dependency tree. For instance, librust-typenum-dev has no dependencies.
As such, it could be changed to arch:all+m-a:foreign without incurring
problems to cross compilation. Likewise, librust-bitmaps-dev only
depends on librust-typenum-1+default-dev, which is provided by
librust-typenum-dev. Consequently, it could be converted to
arch:all+m-a:foreign as well. However, as soon as you reach C bindings,
such as librust-typenum-dev, the workaround to use arch:any+m-a:same is
needed. Then any library that transitively depends on any C binding also
needs the workaround. The problem here is that lower layers can
introduce bindings at a later time and thus break cross compilation
involving a higher level library. Whether a library can safely use
arch:all+m-a:foreign is dependent on its entire dependency tree. For
that reason, using arch:all+m-a:foreign is quite a good way to shoot
yourself in the foot.
Let me try to distill a rule of thumb from this: Whenever you require a
package to transfer an architecture constraint to downstream
dependencies (i.e. one of them likely is a C binding), arch:all package
cannot be used.
This actually is one of the m-a hinter rules. It suggests a conversion
from arch:any to arch:all+m-a:foreign whenever three conditions are met:
* The contents are equal on all release architectures.
* There are no maintainer scripts.
* All of the dependencies are marked m-a:foreign or annotated :any.
Could it be that Jonas picked up such hints and thus added m-a:foreign?
Then later a C binding was added to the mix, which broke the hint, but
the conversion wasn't reverted.
This suggests that the hinter should gain another condition to never
issue arch:all+m-a:foreign conversion hints for librust-*-dev packages.
Do you agree?
Helmut
Reply to: