Re: Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building
Control: tags -1 - pending
Control: reassign -1 libglib2.0-dev,sbuild
Hi Simon,
I'm sorry for the delayed reply. Even though I dumped significant effort
into this, things turned out getting more complicated.
For one thing, my attempts to fix this on the sbuild side have resulted
in tagpending tagging this bug. sbuild's git no longer performs such
tagging for feature branches.
On Fri, May 17, 2024 at 10:42:11AM +0100, Simon McVittie wrote:
> This is happening because upstream GLib 2.80.x has taken over
> the lower-level parts of the GObject-Introspection toolchain
> from src:gobject-introspection. As a result, providing its full
> functionality requires the ability to run host-architecture binaries
> (for at least gi-compile-repository, which will eventually replace
> gobject-introspection's g-ir-compiler).
>
> In older versions, src:gobject-introspection would likely have had
> the same issue; the only difference is that libglib2.0-dev affects
> more packages.
>
> As a reminder of the design here, g-ir-scanner from in the
> gobject-introspection source package examines shared libraries and outputs
> GIR XML, which is analogous to C headers: usually, but not always,
> architecture-independent. g-ir-compiler or gi-compile-repository takes the
> GIR XML and compiles it into architecture-dependent binary typelibs.
> The GIR XML is used directly by some language bindings (typically at
> compile-time for static languages like Rust and C++), and the typelibs are
> used for others (typically at runtime for dynamic languages like Perl and
> Paython).
>
> I had initially hoped that we could wrap the build-architecture
> g-ir-compiler/gi-compile-repository to compile GIR XML into typelibs,
> but that was broken and caused a RC bug (#1066900), so we cannot do that.
Yeah I know. Thanks for writing it down again. Not blaming you.
> In principle we could build one copy of gi-compile-repository per (build
> architecture, host architecture) pair, like cross-compilers, but that
> seems bad from a maintainabiity point of view. Upstream does not support
> this use-case (and in general has little interest in cross g-i) so it
> would be very easy for it to regress, it would require either GLib or
> some other package to know a complete list of all interesting Debian
> architectures in the same way that gcc-13-cross does, and running
> g-ir-scanner fundamentally requires executing code from the host
> architecture *anyway* because that's just how g-i works.
>
> If the dependency was changed to:
>
> something-native | qemu-user | qemu-user-static,
> python3:any,
>
> where "something-native" is any package that is either M-A: no or
> M-A: allowed (but must not be M-A: foreign, and in practice is required
> to be installed from a runnable architecture and not a barbarian
> architecture) but is *not* Python, would that solve the problem?
I guess that it might solve the immediate problem, but I'm not sure this
is a good compromise. We've had the idea of restricting
Multi-Arch:foreign packages from satisfying dependencies with host
architecture instances since a while and crossqa's scheduler (but not
builder) actually enforces this, so I've been looking into also
enforcing it on the builder and that might be a better solution. I
cannot tell yet. That said, you can find it at:
https://salsa.debian.org/debian/sbuild/-/merge_requests/69
I note that my testing of this was not satisfactory. For one thing, it
didn't seem to be reliable (and I note that none of the python core
stack actually is Multi-Arch:foreign) and it also doesn't make any
package cross buildable. When I get past this problem, I run into a new
problem where libc6.postinst fails to understand that it is run in a
chroot (as it looks too much like a container) and thus systemd-sysv's
telinit kills the parent process of apt making sbuild unhappy
(#1071462). While I welcome feedback on this issue, it quite definitely
is a timesink.
> Most of the obvious Essential or Build-Essential packages like base-files
> are Multi-Arch: foreign, therefore not suitable for this purpose, but for
> example a short-term hack version of this might be to use apt, perl or
> perl-base as the something-native - because they are already installed
> (and because apt is Important: yes and perl-base is Essential: yes),
> hopefully apt will prefer to keep the existing version installed and take
> the qemu-user alternative, in preference to attempting to crossgrade them
> to a version that does not, in fact, work.
I appreciate your effort for a quick solution, but I question whether
the cure really is better than the disease. I also note that crossqa.d.n
was completely broken for a week by c-t-b being uninstallable, so maybe
we are better off acknowledging the problem and finding a solution we
can all live with.
Knowing the available options does help though. Thank you.
> I don't know whether apt might become Multi-Arch: foreign in the future,
> though. If it did, that would break our ability to use it like this.
apt also isn't technically build-essential and we are looking into
building inside apt-less chroots.
> This intersects with #1070773, in which users of amd64 + i386 multiarch
> don't want to have to install qemu-user, and would prefer to have some
> way to say "it's OK, my amd64 system can run i386 binaries directly".
Yes, absolutely.
> If someone with more time available for cross-development
> implemented the cross-exe-wrapper design that I sketched in
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030223#37, then
> gobject-introspection and libglib2.0-dev would be able to migrate to
> depending on
>
> cross-exe-wrapper | can-run-arch,
> python3:any,
>
> or for backwards-compat maybe
>
> cross-exe-wrapper | can-run-arch | qemu-user | qemu-user-static,
> python3:any,
>
> and I think this would also solve the problem?
I think this is technically unsolvable on the package relation level.
The "can-run-arch" property that you suggest here cannot technically be
expressed by a dependency relation, because the answer depends on things
apt doesn't know. Consider the following cases:
* We cross build from amd64 to x32, but the kernel does not enable the
x32 abi.
* We cross build from amd64 to m68k and have externally installed
qemu-user-static as we mostly care about speeding up builds rather
than doing "pure" cross builds.
* We cross build for i386 from amd64 and disable the i386 kernel ABI
via seccomp-bpf to simulate a QA cross build.
* Some arm64 CPUs can run arm32, but not all.
Whether the host architecture is runnable is to be defined by the
builder and the builder is supposed to set the cross build profile when
the host architecture is not runnable. So in regular, amd64->i386 cross
builds we'd want to remove the cross build profile unless we're
interested in QA.
Unfortunately, the dependency at hand is a runtime dependency and we
cannot use build profiles in runtime dependencies, so my whole
elaboration on the cross build profile gets us pretty much nothing here.
I don't have a better option.
> If that doesn't work, another option (but IMO a bad one) is that because
> g-i functionality in GLib is a new feature in trixie, it would in
> principle be possible to move gi-compile-repository and related tools to
> a separate binary package, at the cost of breaking backwards compatibility
> with older versions of trixie-as-testing (and also breaking compatibility
> with Ubuntu 24.04, which would be unfortunate). We would probably need
> *two* new binary packages:
>
> - one Multi-Arch: same, analogous to gobject-introspection, to contain
> /usr/bin/$(DEB_HOST_GNU_ARCH)-gi-*, plus their implementations in a
> private directory (at the moment this is part of libglib2.0-dev);
>
> - and one Multi-Arch: no containing the non-architecture-prefixed
> /usr/bin/gi-* which will be expected by simple non-cross-capable
> upstream scripts and build systems (at the moment this is part of
> libglib2.0-dev-bin, which is Multi-Arch: foreign, which is not 100%
> correct but is the best we can do to accommodate these naive build
> systems' expectations)
>
> Hoever, I believe the package containing
> /usr/bin/$(DEB_HOST_GNU_ARCH)-gi-* could cause the same installation
> issue for your cross-build infrastructure that libglib2.0-dev now causes:
> you'll see that gobject-introspection.deb also has this
> "python3 | qemu-user, python3:any" dependency. So I'm not convinced that
> would actually help, it would probably just move the problem around.
I fear that I don't quite understand the implications, but let me note
that it would move the problem around sufficiently for a relevant
package set. We've done without cross building gobject-introspection
stuff for a very long time. Regressing on libglib2.0-dev is relevant
here. So consider that "just move the problem around" provides
noticeable benefit from my point of view (without implying that doing so
is worth the downsides that you depict).
> > On a technical level, python3 | qemu expresses what we want. The crossqa
> > service sees this and has special handling for python3. It considers
> > python3.*-minimal to not be installable for the host architecture (as
> > that pratically fails postinst).
> ...
> > and then apt looks at the dependencies. It sees python3 as the first
> > alternative and is generally happy with installing the host one.
>
> I think this is the real problem here. We say python3:any because that's
> what the multiarch dependency syntax allows us to say, but what we
> *mean* is "well *obviously* I need to be able to run the interpreter".
> A better syntax, if it was allowed, would be python3:runnable or even
> python3:native - but we can't say that in this context.
I concur.
> crossqa's dependency resolver has this special case to treat non-native
> python3 as uninstallable, but apt does not, and it's that disagreement
> that is breaking your test scenario.
That is exactly why I looked into configuring "barbarian" architectures
(see the sbuild MR above).
> I'm sorry that GObject-Introspection is complicated, but that's just
> the way it is. I am doing the best I can do make it work the way you
> would want it to within my limited time and resources, and in some cases
> against upstream hostility. If you would have preferred cross-g-i to
> remain impossible, that would have been the other realistic option here.
I very much appreciate your cooperation and feedback. I'm not meaning to
imply work on you here. To me, this is a technical problem and the most
responsible person is the porter (i.e. me), not you.
> I'm open to suggestions for how to resolve this, but I cannot always
> drop everything to solve cross issues immediately, and I certainly
> didn't intend to be Debian's single point of failure for all things
> gobject-introspection.
"some urgency" is very different from "drop everything" in my book. You
read it differently than it was meant. In the grand scheme of things,
cross building is a niche feature and it is up to cross build porters
(though the plural can be seen as a lie) to make it work. Your
supporting of the effort beyond what the average maintainer does is
appreciated.
Helmut
Reply to: