Re: debian-xcontrol updated
Hi,
On Tue, May 12, 2009 at 01:46:40PM +0100, Neil Williams wrote:
> I plan to parse debian/xcontrol and construct a changelog entry that
> can be put into the .changes file, summarising what was dropped, what
> was added etc. :
Hm, so you plan on generating a modified control file at build time?
> > > This allows e.g. toolchain packages to list the cross variants in the
> > > xcontrol file so an xcontrol aware autobuilder can see them and
> > > specifically request them to be built. This is per-binary, defaults
> > > to "no".
> "specifically requesting packages to be built" presumably means relying
> on 'fakeroot debian/rules $target' which in turn means not calling
> dpkg-buildpackage at all.
That, or teaching dpkg-buildpackage about invoking different targets.
> Yes, 'dpkg-architecture -a$arch -c' can do
> the job but with discussions around environment variables and lack of
> "official" support for calling debian/rules directly, this could be
> problematic.
Well, "debian/rules foo" is the official interface, according to policy.
Any package that fails to build because it requires dpkg-buildpackage is
therefore broken.
If there is consensus that dpkg-buildpackage should be the official way to
build packages, then policy should be amended to include the environment
variables defined by dpkg-buildpackage. I do not like the idea of having a
policy that treats an implementation of something as a normative reference.
> It probably needs to be built via DEB_VENDOR support in
> dpkg-buildpackage which then triggers the relevant sub-builds within
> the debian/rules file using normal dpkg-buildpackage procedures.
Right -- an xcontrol file should not define policy.
> All xcontrol achieves in this situation is that the final .changes file
> doesn't list the unbuilt packages as all the real work has to be done
> in debian/rules anyway. However, this is still important because CDBS
> and debhelper 7 will always build every package listed in
> debian/control.
That is a real problem, and I think CDBS and debhelper need to be fixed.
The package list in the control files is what is taken over into the .dsc,
from there into Sources and from there into dak, where it is used to
associate source and binary packages. As a goal, vendors should be able to
take over the source packages unchanged (because their policy can be
applied via DEB_VENDOR), however if the binary package list needs to be
changed in the source package first in order not to confuse (the vendor's)
dak, then this is no longer possible.
> There are two reasons for Crush to use Optional:
> 1. To change functionality in an existing source package by replacing a
> standard Debian binary package with a renamed Emdebian binary package.
> The standard Debian binary package (which would normally cross-build but
> is too bloated for Crush) is deemed "Optional: yes" and the replacement
> binary package is deemed "Optional: no && Cross-Compiling: yes".
That would mean that the xcontrol file is changed per-vendor, because every
vendor has different ideas about what should be Optional, and what should
be default builds, which would again require every vendor to maintain their
own source package patching infrastructure.
I was trying to keep vendor policy entirely external, but I'm not sure this
is going to work given the CDBS/debhelper situation. It might make sense
for these to match DEB_VENDOR against a Vendor field in control/xcontrol to
filter the package list.
emdebian could work without such a thing in the beginning though: our
policy would be to build all packages, including Optional ones, and
including the bloated ones (because someone might find them useful, and for
everyone else there are smaller alternatives), and to expect system vendors
using our binary packages to filter the package list.
> (Optional: no && Cross-Compiling: no makes no sense for Crush and the
> autobuilder would need to throw an error.
"Optional: no" maps to "a policy that asks for this package to be built".
That is an error for the autobuilder, and needs to be resolved manually,
either by fixing the package, or by changing the policy so the autobuilder
knows that this package can be safely ignored.
> 2. To omit a language binding or other binary package from an existing
> source package that would otherwise fail to cross-build. Optional: yes
> && Cross-Compiling: no
Interesting point: these packages would not be Optional from Debian's POV,
but cannot be cross-built, so we want to ignore them. Since they are not
Optional, the "build"/"binary" targets would include them, and thus fail.
We could either define that "Cross-Compiling: no" packages should not be
attempted in the rules file if cross building (less work for the
autobuilder), or that this makes the regular rules unusable for Crush and
the autobuilder should use the explicit targets (less work for the package
maintainers).
> This leads to potential misunderstandings about "Optional". The
> replacement package is Optional: no. However, under the original
> description, the renamed package is not built by the normal Debian
> build so could be understood as Optional: yes.
Basically, vendor policy overrides Optional. Optional means two things:
- with DEB_VENDOR == "debian", this package is not built by the targets
defined in Debian policy.
This is all that matters to Debian.
- there are separate targets for this package
This allows anyone to build the binary package on its own if they want to,
without defining a new vendor or changing the source package. How they
decide that they want to build the package is not part of the package.
Pretty much orthogonal to that is DEB_VENDOR, which is currently not
handled by xcontrol at all. If you change DEB_VENDOR, you can also change
which packages are built by default. How this is achieved in the rules file
is not xcontrol's problem, but we can add a Vendor field that can be
interpreted at build time.
> What we actually need is:
> Package: libgconf-noldap-2-4
> Optional: no
> Cross-Compiling: yes
"Optional: no" would mean that this package is built from the regular
targets, which it isn't unless either the rules file is patched or
DEB_VENDOR != "debian" (but all bets are off in that case anyway), so it
would have to be "yes" for emdebian as well.
> Package: libgconf-2-4
> Optional: yes
> Cross-Compiling: yes
Same reasoning here: Optional needs to be "no" unless we have patched the
source.
What is missing is the bit that makes the Crush autobuilder decide to build
the package. Our options are
- assume that we want every crossbuildable package, i.e. declare that
emdebian is a binary distribution and we generally do not expect people
to rebuild packages unless they have a very good reason (Sounds good to
me).
- define DEB_VENDOR="emdebian" with our own policy that ignores Optional.
> Package: python-xml
> Optional: yes
> Cross-Compiling: no
Assuming this package is "Optional: no" from Debian's POV, this is the
not-yet-well-defined case I was talking about above. With a strict
interpretation, we could immediately see that any attempt to cross-build
this package must fail before even attempting it, because a single binary
package that is part of the default set will make the default rule fail.
IMO, that is nonsensical -- if a package's default rules do not support
cross compilation, it should declare "Cross-Compilation: no" in the source
stanza and maybe add exceptions for individual Optional binary packages.
Again, two options:
- Make it an error to declare cross building support in the source stanza,
then retracting that in a non-Optional binary stanza
- Define that case to mean that the standard rules will omit that package
when cross-compiling.
Technically, omitting the package would also be allowed without declaring
it, but it is good to know what to expect before starting the build.
> The "optional" keyword has hence been overloaded too many times and has
> become unclear.
> I think Optional should be restricted to *only those packages that can
> be dropped* from the normal Debian build and that these packages are
> dropped from the output of 'xcontrol build'.
"xcontrol build" is not meant to be run at build time (due to Debian
policy) -- that's why there is a separate "check" command that indicates
that "build" would return something different than what is there currently.
The reason these packages are emitted is precisely that I'm not allowed to
re-add them later on.
> If a package is Optional: yes and Cross-Compiling: yes, then Optional
> should override Cross-Compiling and the package be omitted. This allows
> debian/xcontrol to correctly indicate which packages do cross-build and
> which do not, independent of decisions to drop dependency chains or
> similar.
Indeed, these are orthogonal.
> (i.e. it is perfectly acceptable to have gconf with ldap
> support - IIRC it does build - but it doesn't make a lot of sense when
> the only point of cross-building in Emdebian Crush it is to make the
> dependency chain smaller.)
IMO, Crush should have both -- and so far, there is no mechanism to stop
the full version from being built, simply because Debian does not have such
a mechanism. Our options are:
- extend the package building interface to also include a way to omit
packages at build time (the method defined for Optional packages only
allows adding them). The most obvious approach would be passing
DEB_VENDOR.
- build the packages, filter them later (either by not uploading them, by
filtering them in the archive by migration rules, or by filtering on the
target system)
I prefer the latter, since that requires less changes in individual
packages and also makes Crush useful for a greater group of people as there
are simply more prebuilt options available and I don't need to rebuild if I
need a small system that still supports LDAP.
> It might be better to therefore rename "Optional" as "Omit" to preclude
> the possibility that an additional package (like libgconf-noldap-2-4)
> can be tagged "Optional" because it is an option that we are adding to
> the package when what we actually need is to deem the standard Debian
> package as suitable for omission and our own package to replace it.
"Omit" would better be named "Vendor: !emdebian" -- then the xcontrol file
stays policy free.
> Where a replacement binary package is not a library, should it Provide:
> the replaced binary package?
Debian policy says to Provide only if all users can expect the same
interface (command line options etc.), and I think we should follow that.
If we cannot Provide a package by this rule, then everyone who accepts
either needs to loosen their dependency spec with an alternative.
> Package: busybox-crush
> Provides: busybox
> Omit: no
> Cross-Compiling: yes
> Package: busybox
> Omit: yes
> Cross-Compiling: yes
> busybox-crush is then built instead of the standard Debian busybox
> target, using our own busybox-crush config setup and our own
> busybox-crush.install file in the busybox sources. It is also up to us
> whether to retain busybox-static and busybox-udeb by toggling Omit:
> without giving the misleading impression that these are omitted because
> they don't cross-build when, of course, they do.
Source: busybox
Cross-Compiling: yes
Package: busybox-crush
Provides: busybox
Optional: yes
Package: busybox
Optional: no
If we really do not want a full-featured busybox, then we can omit it when
DEB_VENDOR=="emdebian". Optional is more about allowing additional packages
to be built without cluttering up the rules with vendor handling when it's
not needed.
We can also add "Vendor: emdebian" resp. "Vendor: !emdebian" fields to
allow CDBS et al. to build a list of to-be-built packages at runtime. If
they don't want to parse debian/xcontrol, generating X-Vendor/X-Optional
would be an idea.
> Importantly, the actual instruction to build the busybox-crush target
> must still be 'fakeroot debian/rules binary-arch' so that
> dpkg-buildpackage can still operate.
As above, dpkg-buildpackage disagreeing with policy means that either needs
to be fixed -- dpkg-buildpackage is just the reference implementation.
> debian/rules can include a
> busybox-crush: target but it should point at binary-arch.
Then the target would be pretty useless.
> The actual
> decision about which binary-arch instruction is used is then down to
> conditionals within debian/rules which can be influenced by DEB_VENDOR
> or other mechanisms within debian/rules itself. It doesn't matter how
> that support is delivered, as long as it is in place prior to calling
> dpkg-buildpackage - so we can continue to test the mechanisms via
> 'emsource' patches and then push those changes into the Debian package.
"Optional" basically adds an intermediate option between unmodified
packages and full DEB_VENDOR handling -- when simply building additional
variants on demand is enough.
> A major potential problem is that debian-xcontrol is positioning itself
> as being used as part of the build process - but modifying
> debian/control after executing debian/rules is meant to be a Policy
> violation.
That's why I use it only to check whether the control file is still up to
date, and fail the build if it isn't. For the current feature set, this
should never happen on an autobuilder, since it means someone forgot to
rebuild the control file prior to uploading source.
For the "for every linux-image-* package that exists, build a
"foo-modules-*" package feature, that will change -- xcontrol check failing
indicates the need for a source update (list of binary packages has
changed) and "xcontrol build" provides an easy standardized way of updating
such a package.
Simon
Reply to: