[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#771070: requirements for cross toolchain packages in the distribution

[Most of this was written a few weeks ago when this bug was first filed, but I
didn't post it then as I wanted to get better docs on the compiler
builds in place at https://wiki.debian.org/ToolChain/Cross so that the
substantive argument was easier to follow for those not previously
involved.  That could still do with more work, but it's time to
actually reply, so sending. Some of this has since also been said in 766708 ]

Matthias Klose <doko@debian.org> wrote:

> Recently some crippled cross toolchain packages were uploaded to
> Debian unstable, 

I do wish you'd stop calling these toolchains 'crippled' every time
you refer to them.  They are the only (so far), and thus 'best',
cross-toolchains uploaded to debian itself and meet the requirements
of most users. In fact there are only two people who have expressed
any dissatisfaction with them so far. You and Dmitry. Quite a few
other people have been very pleased to see them.

I do not claim that they are perfect, and expect there to be changes
for Stretch, but they are serviceable and working and maintainable (if
you don't actively block that maintenance).

> without having any consent with the Debian GCC maintainers, nor
> announcing these packages anywhere.

As you know I have been working on Debian-compatible cross-toolchains
since about 2004 on and off. I have discussed them many times with you
personally at Connect, Debconf, given talks about them, and of course
talked to many other people about them too. There was a thread with a
pretty good summary of our basic disagreement here:

Most recently we discussed this again at the Bootstrap/Crossbuild
Sprint in Paris. In all that time we have never agreed about the best
way of doing it, and that disagreement has delayed any upload for the
whole of wheezy.

No-one uploaded packages using your preferred method in the 4 years
they have been used on Ubuntu. I wanted to get some cross-toolchains
in Jessie (https://wiki.debian.org/ReleaseGoals/CrossToolchains
created Sept 2013)

I spent quite some time trying to get the ubuntu cross-toochain-base
'full-bootstrap' packages to build on debian earlier this year but
failed. I tried again at Connect in September - still not
working. After a couple of weeks I decided it was easier to make the
sbuild and wanna-build changes to enable multiarch-built
cross-toochains than to fix that mess. So that's what I did (with much
help from Dima) so now we do at least have some cross-toolchains. 

> Maintained by a single person, not even trying to form a maintainer group,

There is a cross-toolchain maintainer group on Alioth:
https://alioth.debian.org/projects/crosstoolchain/ It has existed for
a long time, but was re-invigorated at Debconf13 where there were
productive cross-toolchain discussions and a clear desire for the
above release goal of toolchains in the archive.

All the code for the recently uploaded toolchains is there:
(and other packages currently in development, like gcc-cross-support)

All those packages are now team-maintained.

A few people have been actively working on cross-toolchains over
the last 18 months. (Me, Daniel Grosser, Dima Kogan, Helmut Grohne on the
linux toolchains, and Keith Packard, Augustin Henze and Thomas
Preudhomme on the bare-metal toolchains).

There is a team of at least 3 people who have committed to maintaining
the existing cross-toolchains (Me, Dima Kogan, Helmut Grohne). I
expect that a few more will help now that it has a higher profile in
the archive, and indeed YunQiang Su has been helping out recently with
mips and multilib changes . You want to replace this team, but who

> and downgrading bug reports without addressing concerns of the GCC
> maintainers.

You filed (#766619, and 5 others for other arches) as 'grave' the fact
that the cross-toolchain (providing cpp, gcc, g++ anf gfortran) did
not also provide gccgo and gobj. No-one has ever provided
crosstoolchains for Debian with those frontends enabled (TTBOMK), so
to claim that their abscence is a 'grave' bug seems, well, wrong. So I
downgraded it to normal (not even 'wishlist' which is arguably more

No-one else has asked for these front-ends, and clearly they are not
needed for the vast majority of packages. Nevertheless they are quite
easy to provide, there is a branch in the cross-gcc git repo that adds
those, and I have uploaded them to the cross-toolchain test repo at:
http://emdebian.org/tools/debian/. I have not uploaded these to the
archive yet because they would have to go through NEW. More
front-ends is a fairly minor issue, and the only limitation is how
many (each with corresponding meta-package from cross-gcc-defaults) do
we want in the archive?

Yes. There could have been better communication. However I know that
any postings are likely to get an agressive, dismissive, response from
you. That is a great way to discourage talking to you. Everytime I
think about posting to debian-devel or cross-toolchain mailing list I
know to expect some abuse about 'crippled' compilers, and make some
time to refute it. It's much easier to just get on with some
productive work instead, so I admit that I have discussed on mailing
lists less than is ideal, to avoid conflict. (There has been plenty of
IRC discussion within the team). But the idea that it is a big
surprise to you make no sense: I've been following through on the
same stuff for a decade.

I hereby commit to continue this debate in public until we reach some
kind of accomodation, or you succeed in having me declared an
incompetent and unsuitable maintainer.

(I'm very happy not to be the gcc cross-toolchain maintainer. It's a
pile of work to keep up with, and you are not making it as much fun as
it ought to be). Is anyone else vounteering?

The wider issue

Clearly we do need to deal with this disagreement about how
cross-compilers are built, and more generally about what the
cross-toolchains should look like in the longer term. And maybe the
technical committee is ultimately the right way to deal with that,
however I think this referral may be premature. The toolchains have
been available in the archive for about 6 weeks now, and from my point
of view they are there so that people can test them, bugs can be found
and we can see how it works in practice. Several have been found and
fixed already. Two have been filed to go upstream into gcc: 760770,

In our (my and doko's) discussions about this I have repeatedly had
responses like 'you are doing it wrong', 'it won't work', but he has
never been able to explain to my satisfaction _why_ it is
'wrong'. Perhaps in this bug report that will become clear.

Doko did once say (Connect, Dublin) 'OK then, show me how it will
work', so that is what I have been working on recently: making the
packages in question, and using them, to see whether in fact there is
some fundamental problem.

Principles of cross-toolchains

Mattias's framing of the basic principles as being 'can they build the
current packaging of gcc and get functionaly the same as the native
package build' seems somewhat arbitrary and circular to me. GCC is an
unusual cross target, and this definition of 'can cross-gcc
cross-build gcc' is circular, because it depends on how gcc is
cross-built, and tangles up questions of multiarch and multilib when
they should actually be separate issues.

It _is_ useful to be able to cross-build gcc. It's a necessary part of
a bootstrap, and we want to make debian bootstrappable. We both want
that working, but it is not necessary to cross-build a multilibbed gcc
in a bootstrap, just one targetting the architecture.

And Bootstraping canbe done with or without wdotap, as Helmut has
already shown.

So, lets be clear that there are actually 2 separate matters of substance here:
1) Should cross-compilers be built against foreign-arch libc/kernel-headers, or not 
2) Should cross-compilers be multilibbed, or not
Adn there is a third question related to 1:
3) How are the libc packages the cross-toolchain is built against, built?

There are quite a few other factors that come into this, but those are
the 3 core questions, IMHO. And those questions are, in principle,
independent. You could build a multilibbed cross-compiler against
foreign-arch libc or a non-multilib cross-compiler using in-arch
-cross libc. However, currently these questions get dragged together
because not every combination is actually supported by current
packaging. You will correct me if there is some fundamental reason why
it would not be possible to make all four combinations?

So, let us look at these three issues to first clarify what we are
talking about, the technical details of them, and the pros/cons.

To limit the length of this mail, I have tried to document the various
build methods in: https://wiki.debian.org/ToolChain/Cross
Please refer to that for background.

MA-built vs in-arch

The cross-compiler build _does_ depend on a target-arch libc (or several
for mulitlib builds). So the big questions are where should this(they) come
from and how is it(are they) built?

In the within-arch build the target-arch lib is installed in the
'classic compiler' location in /usr/triplet/.

That works, but is the 'old' way of doing things. For all other
cross-building we now recommend using multiarch locations for
libraries/headers. Is there a good reason for treating gcc specially in
this regard and keeping this old method around indefinitely, or should
we be moving to the new paths?

This was the thinking which caused me to persue building
cross-toolchains aginst the multiarch paths. This has now been shown
to work, and is comendably simple in packaging terms: cross- gccs are
just built against their dependencies, like any other package. This is
quick, simple, and the packaging is totally consistent across

It seems to me to be a good thing that the default build for a
cross-gcc-$arch in the archive is just a simple one-stage build as
this is the normal case. Doing a whole bootstrap build every time to
get a copy of a libc we already have available seems
pointless. Obviously doing a bootstrap build when you do in fact need
to bootstrap (because you are bootstraping, or the target arch is not
in the archive) is perfectly reasonable. 

To make this (building against foreign-arch dependencies) work in the
archive using existing packages does need tooling changes (sbuild,
wanna-build, britney), which have mostly been done, but not entirely (britney
still pending).

The in-arch/supported build still needs the same target-arch library,
but converted with dpkg-cross to the /usr/triplet/ paths.

The dpkg-cross (and dpkg) maintainers think dpkg-cross should go away
as multiarch should have made its path-conversion functionality
unnecessary, and other useful functionality has already moved in to
dpkg itself. The only useful feature it still has is the autoconf
cache variables (which are planned to move into a cross-support
package or similar).

For gcc itself the final build is actually quite simple, and the only
things that changes is whether the target-libc(s) is(are) in /usr/triplet/ or
in /usr/lib/triplet/.

The real interesting questions are about how those libcs are built and
packaged. Are they natively-built or cross-built? Are they converted
from foreign-arch packages with dpkg-cross or do we have some packages
to make libc-$arch-cross packages directly in those locations? Is that a
native or a cross build?

The ubuntu cross-toolchain-base package is responsible for making
libc-$arch-cross packages. It works by doing the whole cross-toolchain
bootstrap from glibc and gcc sources. stage1 cross-gcc, stage1
cross-libc, stage2 cross-gcc built against stage1 cross-libc, stage2
cross-libc built with stage2 cross-gcc. Then dpkg-crossing the
resulting cross-libc to put it into the 'classic toolchain' location.

This seems like a lot of faff for arches where we already have a libc
and you can replace all of that with Build-depends: libc6:$arch.

Something like that _is_ needed for arches where the target-libc is
not already built in debian.

A halfway house would be to build-depend on libc6:$arch, but
dpkg-cross that into libc-$arch-cross, rather than using it
in-situ. That keeps the packaging simple. 

It would also be possible to get the libc packaging itself to spit out
a set of -cross packages.

The outcome of building a cross-gcc against a libc-$arch-cross in the
'classic toolchain' location is that as soon as you also install
libc:$arch to actually build anything except a kernel against, you now
have two libcs for $arch installed on both the 'old' and 'new'
paths. This does not seem 'better' to me. If the -cross libc was
dpkg-cross converted then it's the exact same files. 

I guess an interesting question is 'what does the cross-compiler
actually _use_ the foreign arch libc for'? Does it need its own
independent copy? What happens when the compiler libc-$arch-cross and
the system libc:$arch get out of sync. Does it matter?

Those packages (cross-toolchain-base) in Ubuntu are different for each
supported arch and need/seem to be maintained separately. That doesn't
seem very satisfactory. I am very keen that any solution we come up with
is properly generalisable across architectures.

multilib vs mulitarch

On the multiarch vs multilib matter, native compilers do not have the
same requirements as cross-compilers.

It seems to me to be sensible for Debian to use multiarch, rather than
multilib, where it has a choice, and clearly cross-toolchains is one
such place.

Because cross-toolchains are co-installable it is simple to install
gcc-arm-linux-gnueabihf if you want to target both of those ABIs, as
opposed to building both of those compliers to target both, then
installing either.

Native compilers are not yet co-installable so have to use multilib to
target more than one ABI.

The main advantage of this for the cross-toolchains is that the builds
are entirely consistent across architectures. As is shown by the
current cross-gcc (source)/ cross-gcc-dev (binary) in expermental. One
set of rules and patches works to build all the cross-gcc-4.9-$arch
packages in the archive. Once you introduce multilib and the
corresponding sets of 'sibling architectures' things get a lot more
complicated, because there may be one, two, or three multilib siblings
to deal with in deps for control, and the build rules. We may be able
to get all this working in one consistent packaging, but I can't yet
see how to do this.

If we can't then you have to maintain each target-arch package
separately, (as the cross-toolchain-base packages are in ubuntu), and
that is a major minus-point. I'm not sure I want to maintain a set of
cross-toolchain packages individually - I want to maintain them as a
consistent, generated set.

It should be the case that we can make MA-built cross-toolchains also
support multilib, and indeed YunQiang Su says it works for him, and he
has been submitting patches, but I have not managed to get it working
myself yet (mostly due to tuits).

Multilibs do make much better sense for arches that are not in the
archive (x32/mipsn32) and are possibly the easier way to support
those, but even there, all the difficulty is in getting the right
libc:$arch or libc-$arch-cross packages. Once you have those you can
build per-arch toolchains or multilib toolchains.

There is some upstream expectation of multilib support, which is the
best argument for enabling it. I don't think debian packages should be
using it - they should use gcc-triplet thoughout (then they don't care
how those triplets are packaged-up). We could use a gcc wrapper to
recignise multilib options call the corresponding triplet-gcc. I
beleive this was how guillems 'fake cross-toolchains' worked?

Using multilib makes things irregular. See
https://wiki.debian.org/ToolChain/Cross#Multiarch_vs_Multilib for
details, but the point is that the options are all differrent
depending which set of sibling-arches you are building. As opposed to
just using <triplet>-gcc everywhere, which is nice and consistent. It
makes packaging irregular too.

But ultimately 'multilib or not' is an independent question from
'MA-built or not'. We should not get them unhelpfully mixed up. 

Current build status

Helmut has been working on getting the 'supported' cross-toolchain
builds working, but the situation to-date is (ironically) that they
only work if multilibs are disabled (on x86 and mips). #773266 #772665

There are also other issues which cause breakage with this
build-method: (dpkg-cross abd glibc issues) #771497 #772045 #773265.

So, as I say, we are just exploring now how these toolchains work in
the archive, and if we have alternative builds working too then we can
compare and contrast. We are getting there, but currently the
MA-built, non-multilib toolchains are the only ones actually building
consistently (because they are simplest). We have some working builds
(varies with arch) of standalone+non-multilib and MA-built+multilib
and standalone+multilib.

The fact that both MAbuilt (wdotap) and non-multilib have been
disabled in gcc-4.9 is not helpful, so everyone building
cross-toolchains has to patch some or all of that back in to get
things to actually built at the moment. The removal was premature, at
best; one could be a lot ruder.

Patched gcc sources

I have always been totally in favour of building the cross-toolchains
directly from the gcc packaging with no extra patches. That was in
fact working nicely until the maintainer uploaded gcc-4.9 4.9.1-19 which
prompted the other tech-ctte cross-toolchains-related bug: 766708
Since then we have had to patch gcc in order to build cross-toolchains.

Later gcc-4.9 4.9.2 (-4 particularly) uploads removed the rest of the
build functionality we are using, so the cross-toolchain team has
spent some time over the last month setting up a mechanism to maintian
this outside gcc-4.9. No-one involved wants this extra work, and we'd
all prefer it if this working functionality had not been removed from
the gcc packaging. Integrating it originally was indeed a non-trivial
piece of work, but leaving it there now is surely very little burden?
This has moved the maintenance burden from the gcc maintainer to the
cross-toolchain team, which is fair enough, but it's more work overall
as this is easier to maintain in-situ than as a patchset.

If the maintainer could revert that change then the current
cross-toolchains would build patchless, and wouldn't have to have
strong version-sync to a particular gcc upload (which is does curently
due to the patches). So we are in full agreement on this point but the
maintainer seems very strongly opposed to keeping it working patchless
with the multiarch-build method.


This gets complicated and this mail is long enough already, but the
existing (in Debian experimental/unstable) cross-gcc packaging is nice
and simple. It is easy to maintain, and would be trivial if we didn't
also have to maintain a large rules patch for the build machinery.

It is not architecture-specific, which I think is important for
long-term maintenance. So long as the libc build-deps are available
it's a trivial packaging change to change whether it uses wdotap or
not.  It's a (much?) harder thing to enable multilib builds as those
lack the consistency across arches that we currently rely on.

And the thing about _not_ using wdotap is that you need to provide all
the 'special' -cross packages rather than the standard packages (via
multiarch) somehow, which is actually the fiddly bit. After thinking
about it a fair amount it seems to me that this is in fact where the
interesting packaging questions lie.

Final remarks

The gist of this bug is for the gcc maintainer to require that only
complicated 'does everything' cross-toolchains should be permitted in
the archive. Why is it bad to have simpler, easy-to-maintain,
arch-consistent cross-toolchains in the archive, especially if the
latter is what someone is volunteering to maintain, and no-one is
volunteering to maintain the former (yet?). The simple ones can evolve
and, in the way of things, are likely to become more capable and
complicated over time. But what is the point of vetoing them,
especially when we _know_ that such simpler cross-toochains (all that
has been available for the last decade or so), satisfy most users

There is more to discuss, but I've gone on long enough already :-)

Principal hats:  Linaro, Debian, Wookware, ARM

Reply to: