[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#766708: supported GCC based cross compilers in Debian

+++ Matthias Klose [2014-11-26 15:29 +0100]:

I'm not sure how best to respond to all this. Obviously I disagree
with some of it, and it's remarkable how much our views of the history
of debian cross-compilers differ :-)

I don't think anyone wants a lot of tit-for-tat debate as that's not
very constructive. I'll respond to some points inline that are
particularly egregious, and include my version of the relevant history.

I think the more useful and substantive technical discussion about
what cross-compilers should look like in general should go in the
other bug (771070), so I'll try not to get into that here.

> Contrary to some claims in this thread there never was any commitment
> by myself to support the ability to rely on dependencies on foreign
> architectures within the Debian archive.

No, it's been clear that you are not enthusiastic, but it is in the
packages, and it is used, and it is working today, and there has been
a long expectation by plenty of other people that it was the way to
go, for example, Marcin (author of the linaro/ubuntu
cross-toolchain-base packages) wrote (on
https://wiki.debian.org/ToolChain/Cross in 2011/2012):

There is ongoing work on having multiarch dpkg working for both Debian
and Ubuntu distributions. When it will get to final state both ways of
building cross compiler will have to be changed because there will be
no need to manually fetch target arch packages because we could just
build-depend on them.

But what we will have to do when we will have final multiarch support?
I think that there will be will be able to abandon
armel-cross-toolchain-base package in favour of binutils-cross one as
there will be no need to cross build eglibc or linux headers (we will
just build-depend on target packages).

Here's my version of the history of this:

Debian cross-toolchains have existed in various forms since around

First (circa 2000) was the 'toolchain-source' package which was a copy
of the gcc sources with the rules modified to build
cross-compilers. This suffered from divergence from the normal gcc
packages, with different versions, patches and bugs. The cross-support
rules in this was merged into the main gcc package, and gcc output a
gcc-source package so that cross-toolchains could be built using that.

I got properly involved somewhen round here in trying to push the idea
to (particularly) embedded devs, but in fact anyone crossbulding, that
they should use a distro cross-toolchain, not build their own damn one
for every project. That was a radical idea back then. It seems a lot
more normal now.

For many years (since 2004) the emdebian project used the gcc cross-
functionality to build cross-toolchain binaries for Debian. These
builds were done by using the libc/linux-libc packages from the target
arch, converted to libc-<target>-cross/linux-libc-dev-<target>-cross
with dpkg-cross, then building the package against those. The
'buildcross' tool was developed to mechanise this process, and build
for multiple host architectures.

The problem with this method was that it could not easily be made into
a standard package that would build in the archive, because there was
no way to express the dependency on a foreign-arch
libc/linux-libc-dev, prior to multiarch, and also because downloading
as part of a package build is not permitted. Thus the packages lived
outside the archive (at emdebian) for a decade or so, and became well
used. Note that these toolchains were hosted at emdebian but they were
_for_ Debian, not emdebian particularly.

Whilst multiarch was being developed around 2009/10 it became clear
that it could solve this problem of specifying foreign dependencies
for cross-toolchains, and explicit-arch dependencies were included in
the spec partly for that reason.

Meanwhile linaro wanted cross-toolchains in Ubuntu before all this was
ready so packages were created (by Marcin) which ran the whole
toolchain bootstrap procedure, build-depping on linux, binutils, libc,
and gcc sources, and building linux-libc-dev-<target>-cross,
binutils-<triplet>, libc-<target>-cross, gcc-<triplet>, via gcc
stage1, libc stage1, gcc stage2, libc stage2, gcc stage3. This was the
only way to build a cross-toolchain inside a standard package at the
time. Those toolchains went into Ubuntu 10.10 and at the emdebian
sprint at ARM in early 2011 it was planned that despite their
over-complicated build they were a lot better than nothing and would
be fixed up (by Marcin) to build on Debian and uploaded there too,
until multiarch-built cross-toolchains were available/practical.

A GSOC 2012 project to make the necessary changes to gcc for multiarch
builds, was suggested (by me), and was done (by Thibaut Girka)
(mentored by hector and Marcin), and merged (thank you Doko) in late
2012. So now it was possible to build a cross-compiler very simply by
just depending on the foreign-arch libraries needed, and building it
like any other package.

Neither Marcin nor anyone else ever got round to uploading the
full-bootstrap packages to Debian (I don't know why - busy? focussed
on Ubuntu?) so Debian still had no in-archive cross-toolchains for
wheezy, and the emdebian toolchains were not really maintained any
more as we expected a move to the new multiarch ones quickly. That
took much longer than expected in the way of things. This left wheezy
users with a lack of handy cross-toolchains (it was possible to
install the emdebain 'unstable' toolchains with a bit of kicking but
it was way less than ideal). A release goal of cross-toolchains in
jessie was proposed, but official goals were abandonned so it had no
actual force.

Multiarch-built cross-toolchains were working (and documented) from
early 2013, but still could not be usefully uploaded until the
infrastructure learned about foreign-arch dependencies. Sbuild,
wanna-build and britney needed changes. Sbuild was fixed in time for
Jessie - it now automatically enables a foreign architecture if a
package build-deps on one so that the dependency can be installed
during the build. Wanna-build was also patched (to ask dose the right
question) and tested in time for Jessie, but by then the admins didn't
want to change such important stuff, which was fair enough. Similarly
Britney needed to be taught to consider foreign arches when migrating

So from my POV there has been a steady progress from the
toolchain-source packages towards cross-compiler builds just being
another simple package build, albeit with the (almost) unique feature
of having foreign-arch build-deps. The Linaro/Ubuntu
cross-toolchain-base packages were an expedient diversion that was
working around the limitations of the time, not something to be used
any longer than necessary.


As you can see, my and Doko's views of the same history are
significantly divergent. These are nominally just facts after all. I'm
not sure I can say much useful about that. I include this because I
think it illustrates that we are both operating in good faith, but
disagree about what the best solution looks like.

> [GSOC]
> The mentor was no
> help, neither understanding the existing binutils and GCC packaging,
> nor willing to understand it, and still claiming that he is able to
> "simplify" it. 

I'm not sure who you are referring to here, but just to clarify: the
mentors for that project were Hector Oron and Marcin Juszkiewicz, not me.

> Earlier this year, and last time at the bootstrap sprint in Paris,
> Wookey committed to work on the cross-toolchain-base package (this
> package isolates the bootstrap process and builds binary cross glibc
> packages).  I hope other participants of the bootstrap sprint can
> confirm that this commitment was made. 

The report from that meeting https://wiki.debian.org/Sprints/2014/BootstrapSprint/Results says:
3.13. Multiarch cross-toolchains vs single-arch cross-toolchains

This contentious issue was discussed, and is partly covered under other
headings. Wookey prefers the multiarch builds, Doko prefers the single-arch
bootstrap builds. We agreed that either provides useful cross-toolchains. It's
not clear whether it's easier to fix the Ubuntu cross-toolchain-base packages
to do a bootstrap build in Debian, or to fix the blockers for multiarch builds
in the archive. Whichever is working first should get uploaded.

Some work on both was done during the sprint, with current multiarch builds
uploaded to the [CROSS-TOOLS] repo for testing, and various fixups of the
cross-toolchain-base-armhf package:
 * remove obsolete versioned build-deps (nearly all of them)
 * update versions for unstable
 * remove the binutils part of the build as that now comes from cross-binutils
 * start looking at why build fails.

> Then nothing happened.  I

That is not true! A great deal happened: 

* Helmut's gcc-cross-support package was updated, tested, improved and shown to work. 
* I spent week or two trying (and failing) to get cross-toolchains-base to
build on debian, before giving up again. 
* Marcins lost git-repo for cross-toolchain-base was recovered.
* Various cross-toolchain-base versions were merged and patches updated.
* Cross-binutils packages were prepared and uploaded.
* The recently-uploaded cross-gcc-* packages were prepared and uploaded. 
* Sbuild was fixed to add foreign-arch dependency support.
* A whole sbuild release was co-ordinated with bdrung as the maintainer was poorly (RSI).
* Wanna-build was fixed to add foreign-arch dependency support. 
* Cross-gcc-defaults packages were created.
* All these cross packages were uploaded to the Alioth cross-toolchain team site.
* Helmut's multiarch pkg-config changes were tested. Discussion with the 
   maintainer took place.
* A Cross-build test setup was created for the base 100 packages.
* The xbuilder package was updated and uploaded to the archive.

That's a great big pile of 'nothing'. I worked bloody hard. Ask my wife.

> can't remember that he said anything about a change of mind during
> DebConf.  The big surprise then came when Wookey was uploading
> initially six cross toolchain packages to unstable without saying
> anything, without having worked on the cross-toolchain-base packages
> at any time. 

As stated above that is not true. I worked on them at Connect and
afterwards. I had a range of build failures due to differences between
ubuntu and debian kernel packaging and ubuntu and debian libc
patches. I asked Adam for help and he agreed that things needed fixing
so I left it with him after Connect. He never actually got back to me
with a 'it should build now'.

I worked on it again at the UK minidebconf with xnox and Ben
Hutchings. We fixed up the kernel-headers breakage, although only by
apt-getting the kernel source, which I'm not sure is really a
solution. Dmitry said he'd go away and finish getting that going for
debian. I'm not sure if he's finished that yet (I've prodded a couple
of times on IRC, and I think again back in this thread somewhere).

I just tried building the powerpc-cross-toolchain-base-1.2d that you
pointed me at a little while ago just now and it fails with 
 CC      kernel/bounds.s
gcc: error: unrecognized argument in option '-mabi=aapcs-linux'
gcc: note: valid arguments to '-mabi=' are: ms sysv
gcc: error: unrecognized command line option '-mlittle-endian'
gcc: error: unrecognized command line option '-mno-thumb-interwork'
gcc: error: unrecognized command line option '-mfpu=vfp'
Kbuild:35: recipe for target 'kernel/bounds.s' failed

This stuff did work on Debian sometime in 2012. I'm not sure it has
ever worked since. Clearly it could be made to work, but I have found
it more productive to work on stuff that was already working _and_ to
me at least seems like a cleaner build method.

Now we should discuss in 771070 the details of this, and hopefully
reach some sort of agreement about the way forward. I just put this
stuff here to counter your accusations of 'no work', 'not operating in
good faith', and 'just use the supported method it works fine'.

> You may call this politics, or tactics, however I call
> this sneaking in these packages, and his communication behaviour as
> plain XXing (somebody mentioned I should just call this "not telling
> the truth").

And I call it 'getting some actual cross compilers into Debian for the
first time ever'. Something which no-one else has done anytime in the
last 4 years since it was practical/possible. You, or anyone else,
could have uploaded packages using your favoured scheme at any
time. But no-one did.

> At this point I decided to remove the unsupported cross
> build support, and then was accused on irc by Wookey "But ultimately I
> don't think a cross-gcc maintiner can operate if the gcc maintainer is
> actively working against him", and Helmut complaining "I would not
> have forwarded this issue to the tc if Matthias had not combined the
> bad timing with an absence of advance notice". But sorry I must have
> missed the absence of advance notice for the cross build uploads and
> the notice of dropping the work on the cross-toolchain-base packages.
> This is a reaction, not an action.

You were at the bootstrap sprint with both of us. The quoted section
above seems pretty clear to me. I'm sorry if the upload came as a
surprise to you, but it really shouldn't have. 

And obviously I am very disapointed by the way that you have reacted
to some actual progress on this topic by removing functionality, so
the cross- packages have to patch gcc back to how it worked for the
previous 2 years.

> It is odd the somebody first tells people that he is working on an
> agreed solution, then silently stops working on that solution and then
> claims that others can't work with him.  The conversation continued
> together with Adam Conrad and Helmut Grohne how to provide the
> cross-toolchain-base package in Debian, with Wookey ending "OK. I'll
> take a look. I'm away this weekend though [...], so nothing much will
> happen till sunday evg at earliest".  This was three weeks ago, and
> nothing happened, except another new cross toolchain or two uploaded
> to unstable.  So again, usual behaviour of this guy, same procedure as
> every year.

Right, and I have said repeatedly, including in this thread, that I am
entirely prepared to improve these packages, and yes I haven't got
round to doing what you asked yet, but of course that hasn't been
helped by all the extra work you created by removing bits of cross-gcc
functionality so I have to patch the toolchains to keep them working: 
see 766924, 771383

And of course once one uploads something one finds bugs, and gets
users. So I spent some time making things work better, and adding i386
builds (ecause someone asked for them), and even a couple of bug
fixes: 770413  I've also added the gccgo and objc frontends you asked for.

And there was a miniconf in there (where as stated above we made some
small progress on this), and the old emdebian server got hacked and I
had to set up a new one. And I do have other work than just
cross-toolchains to do.

So please, less of the accusations of bad faith.

Yes we have a disagreement about how this should best be done, but we
both want working cross-toolchains in Debian, and I'm sorry that the
first iteration of that uses my preferred build scheme, but that was
what I managed to get working. I don't think having something that I
am prepared to maintain, and you are prepared not to block, is
actually an insoluble problem by the time Stretch is done. 

I agree that I need to communicate with you and others better, there
has been too much IRC and not enough email. I just sent mail to the
debian-cross list tonight. But you need to stop being obstructive
too, otherwise this simply isn't going to work.

> In the meantime there is some progress from Adam Conrad, Dimitri
> Ledkov and my side on the cross-toolchain-base package. Both Wookey
> and Helmut Grohne were CCed on these efforts but they choose to stay
> quiet.

Does cross-toolchain-base now build on Debian? Is that still the
7-stage full-bootstrap package, or just one to produce the necessary
-cross libraries? I don't think the former is sensible, or
maintainable, even if it does now build.

> At this point I don't think it will make sense to work together
> with Wookey if he keeps behaving like this.

Yep. Everything is my fault. I suck big-time. I'm surprised people can
bear to be in the same room. :-)
> I don't mind getting help maintaining GCC in Debian, however this
> shouldn't be limited leaving an odour signature in the packaging, but
> involves interaction with upstream, test rebuilds, bug submissions,
> bug triage and many more.  The involvement of Helmut Grohne and Wookey
> with these tasks is very close to zero.

I submitted a bug earlier: 771383, which you instantly closed
with a snarky message. That's not the way to encourage people. 

And now I see you've just upped the ante further with 4.9.2-4 with
"Remove unsupported with_deps_on_target_arch_pkgs configurations.",
closing the associated bugs: #760770, #766924, #770413.

In contrast to your rejection of my contribution, I'm happy to work
with you on this, even though it can be hard work sometimes. Obviously
you do have a fairly effective veto as gcc maintainer, so you can get
me to give up and go away if you try hard enough. Is anyone else
volunteering to maintain the cross-toolchains? Or are you going to do
it yourself?

Principal hats:  Linaro, Debian, Wookware, ARM

Reply to: