cross building go packages?
Hi go people,
I've vaguely looked into cross compiling go packages since a while. The
first steps were making dh-golang supply the correct environment (see
#960255 and #945479). Thank you for applying these patches.
Later on, I've started filing bugs about Multi-Arch metadata (mostly
Multi-Arch: foreign) and intensified that work lately. Tackling these
dependency problems seems to work out. Most of these bugs are also
reported by the multiarch hinter[1] and thus displayed on tracker.d.o. I
wonder whether it would make sense to mass-commit multiarch hints for go
packages, but that is not the important point of this mail.
Once dependency issues are resolved, go packages always fail to cross
build for the same reason: A build dependency on golang-go uses the host
architecture golang-go and we usually cannot run that. What one really
wants for cross compiling is a build architecture go together with the
correct environment variables for cross compilation. As pointed out
earlier, dh-golang already sets up that environment. Therefore what we
need to do here is somehow change this dependency (for every single go
package). It is not entirely obvious how to change it (and this is the
reason for me not having done it any earlier). So what are the options?
1. Leave as is. Go packages won't be cross buildable.
2. Annotate golang-go with :native in Build-Depends. Doing so tells the
resolver that we need a build architecture go and things will work.
However, doing so requires changing each and every go package (that
builds architecture-dependent binary packages). It also may be a
suboptimal solution as it does not help with binary package
relations.
3. Mark golang-go Multi-Arch: foreign. Doing so is kinda wrong, because
go really doesn't have a foreign interface. You can select the
target architecture via environment variables, but in the absence of
such variables, it'll select its own architecture and that means
that a strict reading of Multi-Arch: foreign is violated. If we
happen to get into situations where the architecture of the go
executable must match the architecture of other pieces, there is no
way to express that using dependency relations. However, it would
make a number of packages cross buildable instantly.
4. Mark golang-go Multi-Arch: allowed and annotate golang-go with :any.
The allowed marking is required for using :any. :any kinda is like
the :native proposal earlier except that it also works in binary
package relations. It is also the solution used by perl, python and
ruby.
5. Introduce go-<arch> packages and <arch>-go cross compilation
wrappers. This is what we do for binutils, gcc, pkg-config and
qmake. Doing so requires changing the way we run go. Instead of
calling plain "go", we need to call ${DEB_HOST_GNU_TYPE}-go. What to
do about dependencies is even less clear in this case. We could
mimic the -for-host way implemented for binutils. Doing so would
require most of the golang-go dependencies with golang-go-for-host.
qmake does things a little different where the main qt5-qmake
package has the "*-for-host" functionality. In other words, we'd
move the contents of golang-go to a Multi-Arch: foreign
golang-go-bin package and have golang-go depend on golang-go-bin as
well as provide the <arch>-go wrapper in order to not change
dependencies. Beware that this method requires making all of the
"go" invocations substitutable, something that doesn't seem to have
happened in any distribution packaging go.
The most similar situation to golang-go I can think of is clang as it
also has a default target and allows you to select the target. We don't
have a solution for clang yet.
All options but 1 and 3 require changing almost all go-related source
packages. Are there any tools to mass-commit such changes to many go
packages?
Since some of the dependency issues are already fixed, I was able to
cross build some go packages using option 2 (which means that they'd
also cross build with option 3 and 4). The following packages can cross
built that way today (non-exhaustive):
* canid
* go-qrcode
* golang-bindata
* golang-github-benbjohnson-tmpl
* golang-github-calmh-xdr
* golang-github-eknkc-amber
* golang-github-odeke-em-ripper
* golang-github-rakyll-statik
* golang-gogoprotobuf
The following packages don't quite work yet despite having satisfiable
depends:
* golang-github-mailru-easyjson -> #960613
* golang-github-docker-docker-credential-helpers has a Makefile that
runs go without the cross environment of dh-golang. I think we need
another way to create this environment for such cases.
In general, my tests suggest that it is a good time to look into solving
the golang-go dependency for cross compilation as many other pieces of
the puzzle fit already.
Now, I'm looking for your (go maintainers) input on how to proceed.
* How easy is it to mass-commit changes to every go package?
* Which of the options presented above seem attractive to you?
If some of this is difficult to understand, don't hesitate to ask me
for more detailed explanations.
Helmut
[1] https://wiki.debian.org/MultiArch/Hints
Reply to: