[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Speedup when using qemu to cross-build packages.


Quoting Christof Warlich (2022-09-08 12:35:38)
> > If half the time people spend on hacks would be spent on a proper solution,
> > so much more could've been done already... I know, everybody can spend
> > their free time on the stuff they find fun but sometimes it's a bit
> > frustrating...
> Yes, fully agreed. But the same applies for so many things: If people (and
> companies) would only be less selfish, everyone would benefit. The only way
> to fix that would probably be to create some incentive to do it right (for
> everyone). How this could be achieved for Debian should imho be a widely
> discussed topic within the community.

yes, absolutely.

> > Cross-building is often the same. Every new architecture that comes along
> > is bootstrapped in a rush and everybody rewrites the same hacks over and
> > over and over again.
> To me, the major problem is that cross-building seems to be not even a
> second-class citizen to package maintainers: When cross-building only the
> most basic packages, i.e.  those being reported by "debootstrap
> --print-debs", less than a third build sucessfully, the rest fail for various
> reasons, many of them (e.g. bash, glibc, apparmor, ...) even fail already to
> install their required build dependencies!
> Thus, if even most of the maintainers don't care, it very much looks to me
> like almost everyone already gave up on this. And considering that Debian has
> some 20.000 source packages, it certainly would be almost impossible for that
> to be fixed by a single individual.

Yes, the cross-building team in Debian is very small and mainly carried by
Helmut. This is a chicken-and-egg problem. Before we can make
cross-buildability a higher priority we have to have enough people that solve
the problems. It would be nice to display cross-build-failures on tracker.d.o
or other packaging overviews but that doesn't make sense if there is not enough
person-power to solve these problems. There are also still some very
fundamental problems to cross-building that make thousands of source packages
FTCBFS, namely the gcc-for-host integration with gcc: #666743

More information here: https://wiki.debian.org/CrossBuildPackagingGuidelines

Some good news are, that our gitlab CI systems now automatically add a
cross-build job to the default packaging pipeline, so there is some visibility
for maintainers that use our gitlab CI to make their package cross-buildable.
Unfortunately, many failures are problems in other packages, like gcc-for-host.

Another good thing that exists now is https://crossqa.debian.net which shows
you which packages in Debian can be successfully cross built today. For example
above you say bash, glibc and apparmor do not cross build, but they do:

https://crossqa.debian.net/src/bash https://crossqa.debian.net/src/glibc

Maybe you are not cross-building them correctly? How are you cross-building
Debian packages? Above you say that less than a third cross-build successfully.
I cannot confirm this observation. Using crossqa.d.n, the situation for the
packages shown by debootstrap --print-debs is not too bad. The only source
packages in that selection that never succeeded to cross-build in the last runs
on crossqa.d.n are db5.3, gcc-12, libselinux, libsemanage and vim. So only five
source packages out of your selection of 109.

> Considering all this, qemu-user really seems to be a rather attractive
> solution: It's slow, but it works! And if it could significantly be sped up,
> maybe even close to a native build, we may  seriously ask whether
> cross-compiling is still needed at all!

Yes, there are situations in which Qemu is not an option. This is why we are
currently working on DPKG_ROOT. When a new architecture pops up it usually
doesn't have qemu support and if it does, it is too slow to do anything useful
with it. In those situations, crossbuilding without qemu is the only way to get
binaries for your initial system. What you propose is for people who do not
have a arm64 or mips machine around and still want to build for those
well-established architectures. That's useful but not the same as what is
required for early bootstrapping.

In cross-building, qemu is currently used by some players in a different way. A
lot of packages fail to cross build because they need to execute some tool of
the foreign architecture and fail to do so without qemu. For those packages,
qemu can enable the package to build successfully. The gobject-introspections
is an example for this because gobject-introspections doesn't support
cross-building at all, making all packages that rely on gobject-introspections
fail from the start. If we are talking about cross-building for established
architectures and not the early bootstrap situation, then qemu can help here.


cheers, josch

Attachment: signature.asc
Description: signature

Reply to: