[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

cross building go packages?

Hi go people,

I've vaguely looked into cross compiling go packages since a while. The
first steps were making dh-golang supply the correct environment (see
#960255 and #945479). Thank you for applying these patches.

Later on, I've started filing bugs about Multi-Arch metadata (mostly
Multi-Arch: foreign) and intensified that work lately. Tackling these
dependency problems seems to work out. Most of these bugs are also
reported by the multiarch hinter[1] and thus displayed on tracker.d.o. I
wonder whether it would make sense to mass-commit multiarch hints for go
packages, but that is not the important point of this mail.

Once dependency issues are resolved, go packages always fail to cross
build for the same reason: A build dependency on golang-go uses the host
architecture golang-go and we usually cannot run that. What one really
wants for cross compiling is a build architecture go together with the
correct environment variables for cross compilation. As pointed out
earlier, dh-golang already sets up that environment. Therefore what we
need to do here is somehow change this dependency (for every single go
package). It is not entirely obvious how to change it (and this is the
reason for me not having done it any earlier). So what are the options?

 1. Leave as is. Go packages won't be cross buildable.
 2. Annotate golang-go with :native in Build-Depends. Doing so tells the
    resolver that we need a build architecture go and things will work.
    However, doing so requires changing each and every go package (that
    builds architecture-dependent binary packages). It also may be a
    suboptimal solution as it does not help with binary package
 3. Mark golang-go Multi-Arch: foreign. Doing so is kinda wrong, because
    go really doesn't have a foreign interface. You can select the
    target architecture via environment variables, but in the absence of
    such variables, it'll select its own architecture and that means
    that a strict reading of Multi-Arch: foreign is violated. If we
    happen to get into situations where the architecture of the go
    executable must match the architecture of other pieces, there is no
    way to express that using dependency relations. However, it would
    make a number of packages cross buildable instantly.
 4. Mark golang-go Multi-Arch: allowed and annotate golang-go with :any.
    The allowed marking is required for using :any. :any kinda is like
    the :native proposal earlier except that it also works in binary
    package relations. It is also the solution used by perl, python and
 5. Introduce go-<arch> packages and <arch>-go cross compilation
    wrappers. This is what we do for binutils, gcc, pkg-config and
    qmake. Doing so requires changing the way we run go. Instead of
    calling plain "go", we need to call ${DEB_HOST_GNU_TYPE}-go. What to
    do about dependencies is even less clear in this case. We could
    mimic the -for-host way implemented for binutils. Doing so would
    require most of the golang-go dependencies with golang-go-for-host.
    qmake does things a little different where the main qt5-qmake
    package has the "*-for-host" functionality. In other words, we'd
    move the contents of golang-go to a Multi-Arch: foreign
    golang-go-bin package and have golang-go depend on golang-go-bin as
    well as provide the <arch>-go wrapper in order to not change
    dependencies. Beware that this method requires making all of the
    "go" invocations substitutable, something that doesn't seem to have
    happened in any distribution packaging go.

The most similar situation to golang-go I can think of is clang as it
also has a default target and allows you to select the target. We don't
have a solution for clang yet.

All options but 1 and 3 require changing almost all go-related source
packages.  Are there any tools to mass-commit such changes to many go

Since some of the dependency issues are already fixed, I was able to
cross build some go packages using option 2 (which means that they'd
also cross build with option 3 and 4). The following packages can cross
built that way today (non-exhaustive):

 * canid
 * go-qrcode
 * golang-bindata
 * golang-github-benbjohnson-tmpl
 * golang-github-calmh-xdr
 * golang-github-eknkc-amber
 * golang-github-odeke-em-ripper
 * golang-github-rakyll-statik
 * golang-gogoprotobuf

The following packages don't quite work yet despite having satisfiable
 * golang-github-mailru-easyjson -> #960613
 * golang-github-docker-docker-credential-helpers has a Makefile that
   runs go without the cross environment of dh-golang. I think we need
   another way to create this environment for such cases.

In general, my tests suggest that it is a good time to look into solving
the golang-go dependency for cross compilation as many other pieces of
the puzzle fit already.

Now, I'm looking for your (go maintainers) input on how to proceed.
 * How easy is it to mass-commit changes to every go package?
 * Which of the options presented above seem attractive to you?

If some of this is difficult to understand, don't hesitate to ask me
for more detailed explanations.


[1] https://wiki.debian.org/MultiArch/Hints

Reply to: