[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: translating build-deps, crossbuildable base and marking packages as crossbuildable



Hi,

Quoting Dimitri John Ledkov (2014-04-25 01:37:52)
> Kind of wish this thread would be on debian-devel.

probably there will be an email to devel soon but I though there would be a
lesser likelihood of highly emotional responses if I run these ideas by this
list first and filter out the ideas that turn out to be too whacky. :)

> But here are some comments =)

Thanks!

> On 24 April 2014 16:43, Johannes Schauer <j.schauer@email.de> wrote:
> > == 1. New Build-Depends syntax extension ==
> >
> > Extend the Build-Depends syntax with a templating format that allows to replace
> > certain values by a dependency resolver. It would look like this:
> >
> >     Build-Depends: gcc-${host} (>= 4.8)
> >
> > When parsing the above, the "${host}" bit would be replaced by the host
> > architecture name.
> >
> > Advantages:
> >  * subjectively most elegant and intuitive
> >
> > Disadvantages:
> >  * requires changes to all tools that parse build dependencies (see
> >    BuildProfileSpec for a list)
> >
> 
> I like this. It's simplish and elegant. Not sure about the dash, e.g.
> gcc${host} might be nicer -> for cross that would be replaced with
> "-powerpc" but for native it would be replaced with e.g. ":powerpc".
> This also has the caveat as option 3:
> 
> """
> >  * the information to what package name to translate the build dependency to
> >    has to be stored somehow (additional field?)
> """
> since cross-compilers use different notations: debian arch, gnu
> tripplet, custom/upstream-specific notation, etc.

A nice way to solve this naming problem would be
https://wiki.debian.org/CoinstallableToolchains

That way, for example installing gcc-armhf would install the cross compiler
binary gcc-arm-linux-gnueabihf. With such naming schemes using debian
architecture names, it would not be necessary to store additional information
about how to translate them.


> > == 2. Use build profile syntax ==
> >
> > Use the existing build profile syntax extension to express the translation when
> > the "cross" profile is activated during cross compilation.
> >
> >     Build-Depends:
> >       gcc-4.8 <!profile.cross>,
> >       gcc-alpha-4.8 <profile.cross> [alpha],
> >       gcc-amd64-4.8 <profile.cross> [amd64],
> >       gcc-arm64-4.8 <profile.cross> [arm64],
> >       [...]
> >
> 
> Why is the build-profile needed? As far as I understand [arch] qualifier is
> of DEB_HOST_ARCH not DEB_BUILD_ARCH,

Correct.

> thus even without a build profile I can do:
> 
> Build-Depends: gfortran-4.8-powerpc-linux-gnu [ powerpc ] |
> gfortran-4.8 [ !powerpc ]
> 
> To for example support native compilation on all arches & cross-compilation
> to powerpc.

But then you would depend on a crosscompiler for powerpc even when compiling
natively on powerpc for powerpc.

The profile.cross is needed so that the cross compiler for a certain
architecture only gets picked if cross compiling for that architecture. If
native compilation is done, the package should build depend on the native
compiler.

> We can write a helper that helps developer to construct/add such a dependency
> list.

That brings us to yet another possibility. Use the gcc-${host} syntax but only
in the Build-Depends line of debian/control and not outside of it. During
package compilation, dpkg will translate a gcc-${host} dependency to the
variant (2) format. That way, only dpkg would have to be adapted and when a new
architecture is added, only a recompilation of affected source packages would
be needed to update the build dependency list. The ugly, long and
unmaintainable form of variant (2) would be hidden from the package maintainer
and only exist in the resulting dsc.

> > Advantages:
> >  * no changes to any tool needed - works automatically through build profiles
> >
> > Disadvantages:
> >  * very long dependency lists
> >  * has to be updated for every package for every new port
> 
> Well that's the worst case, common-case however is that only packages that:
> * depend on a versioned compiler
> * a compiler that is not part of build-essentials/cross-build-essentials
> * and the cross-compiler is packaged in the archive
> 
> are affected. gcc in this case is not the best example, as default gcc is
> used by most packages and it is part of [cross]build-essential.

I dont understand why you list the second point "a compiler that is not part of
(cross-)build-essentials" because I dont see how this issue is connected to
whether or not the compiler is drawn in by (cross-)build-essential.

> The biggest example i can think off here is gfortran (~ 200 packages), but
> i'm not quite sure it's a desirable cross-building target.
> gobject-introspection is a good example, but as far as i know a
> cross-compiler does not exist for that beast yet.

Yes, the number of affected packages doesnt seem to be big.

I ran a quick test and searched for source packages that directly build depend
on version 4.4, 4.6, 4.7, and 4.8 of gcc, cpp and g++ as well as binutils. I
found 166 source packages.

> Hm, building upon option 1 - can we overload Multi-Arch spec a bit here?

I dont have any strong opinions about that option. If you like, please add it
to https://wiki.debian.org/CrossTranslatableBuildDeps

> > What do you think about adding information to source packages whether or
> > not they can be cross compiled?  Having this information attached to source
> > packages would allow botch (the tool behind bootstrap.debian.net) to
> > further automate generating a good build order for the cross as well as the
> > native bootstrap. It can also be used to continuously check
> > crossbuildability of source packages that claim to be crossbuildable. In
> > light of https://wiki.debian.org/ReleaseGoals/CrossBuildableBase at least
> > the base packages should be cross buildable. The information could either
> > sit in its own field:
> >
> >      Cross-Builds: Yes
> >
> > Or piggy-back on a new field that lists supported build profiles of a source
> > package:
> >
> >      Supports-Build-Profiles: cross
> >
> > A problem that I see with adding such information to source packages is: for
> > which architecture combination has a source package to be cross compilable to
> > earn this field? Is the information which architecture combinations work
> > important? In the simplest case, a source package cross compiling for any
> > architecture combination could be eligible for that field because in most cases
> > that means that making more combinations work is relatively simple.
> >
> 
> What is the meaning of this field? the fact that source package hello
> is cross-buildable depends a lot on the factors outside of package's
> authority, and can regress arbitrary when build-depends collectively
> become not cross-installable. If some sort of tracking/knowledge base
> is required, I'd prefer for it to be declared elsewhere, e.g. with aid
> of debtags, archive publishing time generated overrides, etc. Not a
> field that a package maintainer must manually update and do a
> sourceful upload.
> 
> Or are we gonna somehow enforce that certain packages must be
> cross-buildable? E.g. if dpkg declares such a field, then packages
> that cause dpkg's build-depends to be un-cross-installable will not be
> allowed to migrate to testing?

Yes, that would be one of its use cases. Certainly it is not possible to
crossbuild the whole archive (for any architecture) simply because cross
compilers are missing for certain things (you identified that there is no cross
compiler for gobject-introspection but there are many more examples).
Nevertheless it is important to have at least as many base packages as possible
cross compile. The very minimum would be essential:yes and build-essential plus
all packages necessary to install them together. In practice you probably also
want to have debhelper and apt and all packages needed to install them as well.
The more source packages can be cross compiled, the more binary packages for
the target architecture are available during the native bootstrapping and the
less dependency cycles have to be broken. But an automated dependency resolver
cannot know whether or not a source package is supposed to be crosscompilable
and thus cannot make the decision whether it's better to build more source
packages natively (and break more cycles) or cross compile some source packages
(it can't know which ones should be compilable). Therefore human intervention
is needed but we dont want that and would ideally like to automate this task.
Therefore, an algorithm has to know from somewhere whether a source package
should be cross compilable.  Surely this information can also be collected
somewhere else but I think if the information is maintained as part of the
source packages themselves, then it's easier to:

 - run an algorithm (botch) and continuously check whether the bare minimum of
   source packages (those needed to compile essential:yes and build-essential)
   have that field. If they dont, make it a bug.
 - continuously test all packages that declare that field whether they still
   cross compile for a selection of architectures (amd64->armhf for example).
   If they dont, make it a bug.

And given the right bug-severity they would not migrate to testing, yes. In
light of the https://wiki.debian.org/ReleaseGoals/CrossBuildableBase release
goal this would be one way to approach it.

Thanks for your input!

cheers, josch


Reply to: