Re: RFC: Patches for supporting cross-building
Hi Kazuhiro,
On Fri, Jun 15, 2018 at 11:09:58PM +0900, Kazuhiro Hayashi wrote:
> I'd like to share patches for 70 Debian packages with this ML.
> https://github.com/meta-debian/debian-cross-patches
Thanks for your work!
> The patches are intended to make the packages cross-buildable.
> According to the previous summary [1], these packages are still failed
> for cross-building.
Oh, a lot of time has passed since that summary and a lot has changed.
Present numbers are more like: 18% of Debian is cross buildable. That's
more than a fourfold increase.
> Here are several questions about the patches and future contributing
> to Debian community:
>
> * The target packages of above patches are selected based on
> the information in 2016 [2].
> Can I find any other latest cross-building results?
We're seeking a hardware sponsor for running a cross buildd. That's why
we don't have any public CI at the moment. I occasionally run cross
builds for myself now and have a pile of 3400 vaguely current build logs
here, but they are not published anywhere. I've just uploaded a one-shot
summary of these logs at
http://subdivi.de/~helmut/cross_report_20180614.html.
> Also, is there a sharable place for debian-cross developers
> to update such data together? At least, the bug report number
> should be shared here to avoid duplicated efforts.
I have tagged[1] all of my cross build bugs with user helmutg@debian.org
and usertag rebootstrap. More than 500 open bugs affect cross building
of more than 700 source packages. I'm ok with other people using that
usertag for cross build and cross bootstrap issues. I'm also ok with
moving to a different usertag, e.g. user debian-cross@lists.debian.org
usertag ftcbfs would make a lot of sense for build failures. Also
https://bootstrap.debian.net/cross_all.html uses user
debian-cross@lists.debian.org usertag cross-satisfiability to display
dependency issues.
Is there any preference from you or any other contributor here?
X-Debbugs-Ccing such patches to this list could make sense, but it would
vastly increase the mail volume. How do other participants feel about
doing so? Expect around 100 mails/month.
> * I'm planning to send the patches to each package maintainer
> through Debian bug report.
Let me briefly skim through semi-random entries of your patches
repository:
* analog: Breaks building with clang. Better patch filed as #812231.
These days we also have buildtools.mk, which does the right thing.
* blktrace: Already filed as #894836 and fixed in unstable.
* bmon: The approach is wrong. A proper fix can be found in #870864. By
using my approach, we can help share it with other efforts such as
yocto, ptxdist, etc.
* coreutils: Nice patch, because it builds from source. Please send it
to #721358 immediately and Cc me or this list.
* libxt: This one needs discussion. It is about detecting the behaviour
of malloc, which is impossible during cross building. Moving that
check into the packaging is a bad idea, because it means modifying
hundreds of packages. We already have a standard way for autoconf and
that is "cache variables". These can either be set as environment
variables or via a config.site file. For libxt it happens to be
enough to set xorg_cv_malloc0_returns_null to yes (for glibc, this is
wrong for musl). The cross-config package has a number of such
assignments but it is very out of date and even has some wrong ones.
My own cross building efforts remove the broken cross-config from the
equation and I use environment variables. Getting a central piece to
maintain such variables is certainly a good thing. Having individual
packages be that place is going to be a maintenance hell though.
Thus I NACK this patch.
* netkit-telnet: Your patch breaks the test. That snippet will always
compile, but the question is whether it runs. It would break native
builds.
* openhpi: The approach is wrong. A proper fix can be found in #899198
and the patch is applied in unstable already.
* pixz: Already filed as #883888 with a very similar patch.
Given this sample I suggest that you:
* Check which of your patches are still relevant to unstable. Given the
current speed, I don't think it makes sense to target older releases.
* Check which of your patches already have bug reports.
* Defer all cache variable patches (e.g. malloc/realloc non-NULL)
until there is consensus.
* Then repost your patch repository on this list.
> Are there any standard ways for posting patches to support cross-building?
> Ex: Common keywords in subjects, information which should be shown, etc.
With my filing of patches (and I've filed more than 1000 now), I've used
the term FTCBFS (fails to cross build from source, which mimics the
common FTBFS). So the subject tends to be "$sourcepackage FTCBFS:
$summary". Sometimes a failure is caused by another package. Then I've
tried to describe the problem (e.g. "mark $binarypackage Multi-Arch:
foreign" or "move $pkgconfigfile.pc to a multiarch location") and add
"Control: affects -1 + src:$affectedsourcepackage".
Agreeing on a usertag (see above) would help here.
> * My current focus is to create patches for support cross-building,
> but it might be better to concentrate on other packages or topics
> which are more important for debian-cross community.
Indeed, I think much of the low hanging fruit in plain FTCBFS is gone
now. There are a number of harder issues that nobody has tried to solve
yet. Let me try to give some examples that I've poked at, but not
succeeded yet:
* cross building perl. src:perl is now built from source in Debian, so
patching the upstream configure generator to not need try_run checks
for e.g. sizeof is possible.
* gcc-for-host. WIP at #666743, but I'm stuck.
* Reducing the toolchain bootstrap from 3 gcc stages to two. The glibc
developers indicated that this is no longer necessary and
build_many_glibcs.py seems to work with a two-stage bootstrap.
* cross building perl extensions.
* gobject-introspections (yocto and ptxdist gave up and use qemu).
* foreign package installation. I've brought DPKG_ROOT and
--force-script-chrootless into dpkg, but we need update-alternatives
and dpkg-divert to support it.
* non-glibc ports (e.g. musl-linux-any)
* fixing failures from https://jenkins.debian.org/view/rebootstrap/
(is arm64 important to anyone?)
* ...
All of these are hard problems that one isn't going to solve in a week.
Just making random packages cross buildable is substantially easier.
> For example, some kinds of working were already shown by Helmut [3].
Hah. Most of that is still current except for the numbers. Though we
made qt "just work" now and having buildtools.mk makes writing patches
easier.
> It's important for me to cross build a part (not all) of Debian packages
> which are often used in embedded systems, and also interested in
> the difficulty of bootstrapping Debian. I'd like to know if
> there are big issues to be solved related to my interests.
If the list above is insufficient, I can try digging up more. ;)
> Here is a project to achieve such goals by cross-building Debian sources
> with Yocto Project build system [4]. It can cross-build everything
> and provide high customizability, but metadata of packages needs to be
> implemented from scratch, which takes high development & maintenance costs.
I fully agree that duplicating the metadata is a bad idea from a
maintencance pov. That's why I avoided that from the start. You actually
loose the whole qa (e.g. reproducible builds). Trying to compete with
Debian on high quality package maintenance isn't going to end well.
> Another approach is to make Debian packages cross-buildable with existing
> package metadata (i.e. debian/*), which is just now progressing
> in this great community. The first things I'd like to do is to join
> debian-cross then keep better contributions.
It's been progressing a lot for a long while now. You correctly observed
that the approach has downsides as well:
* Customizing packages is not that easy, but with build profiles now
well supported and the extension namespace (pkg.$source.$anything),
customizing packages with build profiles is a reasonable and
maintainable thing now. What kind of customizations do you need?
* Small footprint is another difficult thing. The essential set is
slowly growing all the time and making it smaller is non-trivial. I
learned that the hard way when I made e2fsprogs non-essential and it
still is required. Essential is roughly 110MB. If you nuke
/usr/share/locale and /usr/share/doc and a few more, you can get it
below 70MB.
Still I think that we have reached the turning point for all but the
smallest of embedded systems. Integrating with Debian means that you
pick from a large set of packages with immense QA and long support
windows. If you repackage the Debian's source code, you don't get much
of that beyond license documentation.
Hope this helps
Helmut
[1] https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=rebootstrap;users=helmutg@debian.org
Reply to: