On Tue, 12 May 2009 19:47:36 +0200 Simon Richter <Simon.Richter@hogyros.de> wrote: > Hi, > > On Tue, May 12, 2009 at 01:46:40PM +0100, Neil Williams wrote: > > > I plan to parse debian/xcontrol and construct a changelog entry that > > can be put into the .changes file, summarising what was dropped, > > what was added etc. : > > Hm, so you plan on generating a modified control file at build time? At emdebuild time but that prevents builds with 'fakeroot debian/rules foo' or dpkg-buildpackage -a. In effect, Crush would prohibit changes to debian/control that are not mediated by xcontrol (although that may not be possible for all packages - it may turn out that Crush prohibits changes to debian/control.in or control.m4 and lets debian/control be written during the build as some packages already do). > > "specifically requesting packages to be built" presumably means > > relying on 'fakeroot debian/rules $target' which in turn means not > > calling dpkg-buildpackage at all. > > That, or teaching dpkg-buildpackage about invoking different targets. I need something that will work now, not something that will take months of painful negotiation and to-and-fro with debian-devel. dpkg-buildpackage isn't going to be taught about multiple targets without a lot of arguing. > > Yes, 'dpkg-architecture -a$arch -c' can do > > the job but with discussions around environment variables and lack > > of "official" support for calling debian/rules directly, this could > > be problematic. > > Well, "debian/rules foo" is the official interface, according to > policy. Any package that fails to build because it requires > dpkg-buildpackage is therefore broken. http://lists.debian.org/debian-devel/2009/05/msg00351.html debian/rules is an interface for testing, it is not the "official" method of building packages for *upload* to the archive which is what we are considering here. No package in Emdebian Crush 1.0 will build with just debian/rules - cross-building simply cannot work that way and won't be able to migrate to a method that supports this until after Squeeze and possibly not until Squeeze+1. Some packages in Crush 2.0 will cross-build with debian/rules only but those are the ones that do not need any changes so the whole problem goes away. > If there is consensus that dpkg-buildpackage should be the official > way to build packages, then policy should be amended to include the > environment variables defined by dpkg-buildpackage. I do not like the > idea of having a policy that treats an implementation of something as > a normative reference. Packages for upload need to use dpkg-buildpackage. Packages for upload to Crush *must* use dpkg-buildpackage -a although this can be more easily wrapped in emdebuild (some packages will still require emdebuild in Crush 2.0). > > All xcontrol achieves in this situation is that the final .changes > > file doesn't list the unbuilt packages as all the real work has to > > be done in debian/rules anyway. However, this is still important > > because CDBS and debhelper 7 will always build every package listed > > in debian/control. > > That is a real problem, and I think CDBS and debhelper need to be > fixed. Again, I need a fix that works now. I don't see either of those potential bugs being fixed any time soon. I also disagree that these are actual problems. To me, it makes sense and I want other build systems to follow the same principle. > The package list in the control files is what is taken over into > the .dsc, from there into Sources and from there into dak, where it > is used to associate source and binary packages. As a goal, vendors > should be able to take over the source packages unchanged (because > their policy can be applied via DEB_VENDOR), however if the binary > package list needs to be changed in the source package first in order > not to confuse (the vendor's) dak, then this is no longer possible. There should be no libgconf-2-4 in Emdebian Crush Packages files or Sources files, only libgconf-noldap-2-4 can exist. If individuals want it, maybe it can be done with machine:variant support but it won't be easy. (Nothing about Crush is particularly easy.) The correlation between binary and source package is via the source package name - each binary refers back to Source: - there is no need to correlate binaries other than referring back to the Source. Comparison with Debian (or others) goes via the Source package and binaries are merely a means to obtain the Source: data. We don't care about dak - it is reprepro that matters for Crush and reprepro maps everything exclusively back to the source package name. > > There are two reasons for Crush to use Optional: > > > 1. To change functionality in an existing source package by > > replacing a standard Debian binary package with a renamed Emdebian > > binary package. The standard Debian binary package (which would > > normally cross-build but is too bloated for Crush) is deemed > > "Optional: yes" and the replacement binary package is deemed > > "Optional: no && Cross-Compiling: yes". > > That would mean that the xcontrol file is changed per-vendor, because > every vendor has different ideas about what should be Optional, and > what should be default builds, which would again require every vendor > to maintain their own source package patching infrastructure. and? > I was trying to keep vendor policy entirely external, but I'm not > sure this is going to work given the CDBS/debhelper situation. It > might make sense for these to match DEB_VENDOR against a Vendor field > in control/xcontrol to filter the package list. That would bloat the debian/rules file beyond any reasonable limits. If people want different implementations, they need to patch the xcontrol file (and quite probably debian/rules). I cannot see that debian/xcontrol can be vendor-neutral - unless nobody needs to change anything in the package. As soon as a package needs to be amended, xcontrol becomes vendor-specific. I was actually thinking of a wrapper in emdebian-tools that handles the xcontrol file by reference to a set of overrides. We specify in /etc/emdebian/foo or something that certain packages are disallowed (libgconf-2-4) and that certain packages must replace them (libgconf-noldap-2-4). The wrapper then modifies the existing xcontrol file to match that Policy. xcontrol needs to exist beforehand so that things like Build-Depends and Build-Depends-Tools can be set by hand. Similarly for busybox (removed) and busybox-crush (added). > emdebian could work without such a thing in the beginning though: our > policy would be to build all packages, including Optional ones, and > including the bloated ones (because someone might find them useful, > and for everyone else there are smaller alternatives), and to expect > system vendors using our binary packages to filter the package list. No, as discussed later, I can't build certain packages - that is why I need to change the package names in some cases. There is no way I can build busybox using the Debian config, it has to be a functionally different configuration that actually allows the device to boot. Emdebian Crush *cannot* include the standard Debian busybox package, it is completely useless for our purposes. That's without considering those packages that simply do not cross-build at all. These simply have to be omitted. Emdebian Crush never did have those packages - principally because they could not be built. Crush 1.0 used patches to drop them from debian/control - I need xcontrol to do that instead. > > 2. To omit a language binding or other binary package from an > > existing source package that would otherwise fail to cross-build. > > Optional: yes && Cross-Compiling: no > > Interesting point: these packages would not be Optional from Debian's > POV, but cannot be cross-built, so we want to ignore them. Since they > are not Optional, the "build"/"binary" targets would include them, These are Optional, according to the xcontrol file (note, I specifically described these as Optional: yes). There are ways of dropping these packages using debian/rules conditionals. Debian needs to ignore everything about xcontrol. > > This leads to potential misunderstandings about "Optional". The > > replacement package is Optional: no. However, under the original > > description, the renamed package is not built by the normal Debian > > build so could be understood as Optional: yes. > > Basically, vendor policy overrides Optional. Optional means two > things: It would be better if it meant one. > - with DEB_VENDOR == "debian", this package is not built by the > targets defined in Debian policy. > > This is all that matters to Debian. > > - there are separate targets for this package > > This allows anyone to build the binary package on its own if they > want to, without defining a new vendor or changing the source > package. How they decide that they want to build the package is not > part of the package. It has to be part of debian/rules - the package must support that target and it should support it via dpkg-buildpackage because otherwise the package cannot be uploaded normally. > > What we actually need is: > > > Package: libgconf-noldap-2-4 > > Optional: no > > Cross-Compiling: yes > > "Optional: no" would mean that this package is built from the regular > targets, which it isn't unless either the rules file is patched or > DEB_VENDOR != "debian" (but all bets are off in that case anyway), so > it would have to be "yes" for emdebian as well. It is not optional from xcontrol, it has to be built if xcontrol is active. If this package is Optional: yes, how is the autobuilder to know which target to select? The autobuilder selects any package that is Optional: no and fails if one of those is also set Cross-Compiling: no (which means a bug in the package and/or the Emdebian patches). That is why I need xcontrol to omit any package that is Optional: yes from the output that I can redirect to debian/control prior to calling debian/rules. > > Package: libgconf-2-4 > > Optional: yes > > Cross-Compiling: yes > > Same reasoning here: Optional needs to be "no" unless we have patched > the source. That's the point, the debian/rules file is patched to support noldap. > What is missing is the bit that makes the Crush autobuilder decide to > build the package. Our options are > > - assume that we want every crossbuildable package, i.e. declare that > emdebian is a binary distribution and we generally do not expect > people to rebuild packages unless they have a very good reason > (Sounds good to me). There is no support for building on Crush. Never has been. Crush is, has been and will continue to be binary-only. There is zero expectation of -dev packages being usable in Crush or even of gcc finding the busybox implementation sane enough to build anything. It's surprising how many configure checks would fail under busybox alone. > > Package: python-xml > > Optional: yes > > Cross-Compiling: no > > Assuming this package is "Optional: no" from Debian's POV, this is the > not-yet-well-defined case I was talking about above. With a strict > interpretation, we could immediately see that any attempt to > cross-build this package must fail before even attempting it, because > a single binary package that is part of the default set will make the > default rule fail. > > IMO, that is nonsensical -- if a package's default rules do not > support cross compilation, it should declare "Cross-Compilation: no" > in the source stanza and maybe add exceptions for individual Optional > binary packages. > > Again, two options: > > - Make it an error to declare cross building support in the source > stanza, then retracting that in a non-Optional binary stanza The problem there is that I'd like the source stanza to be authoritative - if 'Cross-Compiling: no' exists in the Source stanza, I want that to mean that *nothing* in that source package is cross-buildable - e.g. perl and python interpreters. So I'd like the reverse assertion - make it an error to disable cross-building support in the source stanza and then enable it later. (The autobuilder will ignore the binary packages if the source package says no, so it would be ignored anyway.) > - Define that case to mean that the standard rules will omit that > package when cross-compiling. Yes, please. > Technically, omitting the package would also be allowed without > declaring it, but it is good to know what to expect before starting > the build. Omitting only works for all packages if the entire stanza is dropped from the output of xcontrol. > > The "optional" keyword has hence been overloaded too many times and > > has become unclear. > > > I think Optional should be restricted to *only those packages that > > can be dropped* from the normal Debian build and that these > > packages are dropped from the output of 'xcontrol build'. > > "xcontrol build" is not meant to be run at build time (due to Debian > policy) -- that's why there is a separate "check" command that > indicates that "build" would return something different than what is > there currently. The reason these packages are emitted is precisely > that I'm not allowed to re-add them later on. It would be run immediately prior to the build, explicitly to override the Debian config and build ours instead. Optionally, the results of running xcontrol could be stored as patches and applied using emsource, as now, inside the autobuilder. Or, the autobuilder could migrate to the same build method as external test builds and ignore ../emdebian-control.patch. > > (i.e. it is perfectly acceptable to have gconf with ldap > > support - IIRC it does build - but it doesn't make a lot of sense > > when the only point of cross-building in Emdebian Crush it is to > > make the dependency chain smaller.) > > IMO, Crush should have both -- and so far, there is no mechanism to > stop the full version from being built, simply because Debian does > not have such a mechanism. Our options are: Modifying debian/rules via the mass bug filing for cross-build support is the mechanism. > - extend the package building interface to also include a way to omit > packages at build time (the method defined for Optional packages > only allows adding them). The most obvious approach would be passing > DEB_VENDOR. > - build the packages, filter them later (either by not uploading > them, by filtering them in the archive by migration rules, or by > filtering on the target system) Dependencies are the only realistic method. No packages in Crush would be allowed to depend on libgconf-2-4 but would have to be patched to depend on libgconf-noldap-2-4 instead. The emrecent upload rules could be amended to disallow such dependencies. One way of doing this would be to migrate apt-cross to getting the -dev packages from the Crush repository. That way, dpkg-shlibdeps would automatically pick up that /usr/lib/libgconf.so.2.4.1 comes from libgconf-noldap-2-dev-armel-cross, although whether it would correctly then identify the dependency as libgconf-noldap-2-4 or as libgconf-noldap-2-4-armel-cross is debatable and needs testing. It also complicates the build cycle by mandating that the reverse dependencies are built prior to starting this build. That brings Crush closer into line with normal Debian practice but still makes things a little harder to manage. > I prefer the latter, since that requires less changes in individual > packages and also makes Crush useful for a greater group of people as > there are simply more prebuilt options available and I don't need to > rebuild if I need a small system that still supports LDAP. It requires a lot of changes in packages, that is the main problem, because you keep having to mangle dependencies output by dpkg-shlibdeps. That's why Provides: might have to be deployed. > > It might be better to therefore rename "Optional" as "Omit" to > > preclude the possibility that an additional package (like > > libgconf-noldap-2-4) can be tagged "Optional" because it is an > > option that we are adding to the package when what we actually need > > is to deem the standard Debian package as suitable for omission and > > our own package to replace it. > > "Omit" would better be named "Vendor: !emdebian" -- then the xcontrol > file stays policy free. ??? I don't think xcontrol can remain policy free. I think debian/xcontrol should be generated as a result of Policy - specifically Emdebian Crush Policy. > > Where a replacement binary package is not a library, should it > > Provide: the replaced binary package? > > Debian policy says to Provide only if all users can expect the same > interface (command line options etc.), and I think we should follow > that. We are free to modify Debian Policy for Emdebian Policy. If using Provides means changing one package instead of 150, I'll be using Provides. The point is that this provision of Debian Policy is meant to protect the use of Provides *within* Debian, so that multiple packages can work together. There is little or no chance that modified Crush packages are going to work with standard Debian (or even Emdebian Grip) packages and there is no requirement for us to mandate that Provides: needs to support such restrictions. (Modified packages in Crush 1.0 already lack support for being mixed with Debian, except that there is no Provides:, the package itself pretends to be something it is not. At least if the package name has changed, it gives us one way to identify such problems.) Crush is all about functional changes in packages - functional changes mean that certain Debian practices will simply break. We accept that as the price of reducing the size of Debian. If that is too much pain, users have to use Grip. That is why Crush mandates that certain maintainer script constructs are invalid - because they rely on functionality that is in coreutils but not in busybox. Crush has already broken Debian Policy in multiple areas, modifying Provides: is a minor step. > If we cannot Provide a package by this rule, then everyone who > accepts either needs to loosen their dependency spec with an > alternative. THAT IS A VAST AMOUNT OF WORK, Simon! It isn't going to happen! As far as Crush is concerned, busybox-crush == busybox. It is the only version of busybox that exists, therefore it can Provide: busybox with impunity. busybox in Crush 1.0 already provides coreutils! > > Source: busybox > Cross-Compiling: yes > > Package: busybox-crush > Provides: busybox > Optional: yes > > Package: busybox > Optional: no > > If we really do not want a full-featured busybox, then we can omit it > when DEB_VENDOR=="emdebian". Optional is more about allowing > additional packages to be built without cluttering up the rules with > vendor handling when it's not needed. That's my point, Optional should not just be about adding packages, it must support removing packages - indeed in most cases, adding is only secondary to removing because a package is only added if it replaces one that already exists. Without coreutils, the standard Debian busybox simply will not support booting the device. That is not a reasonable choice and Crush really cannot support coreutils. There is no other realistic option - if you want coreutils, use Grip. If you use Crush, busybox has to be bootable and that means modifying the standard Debian busybox config very substantially. > We can also add "Vendor: emdebian" resp. "Vendor: !emdebian" fields to > allow CDBS et al. to build a list of to-be-built packages at runtime. > If they don't want to parse debian/xcontrol, generating > X-Vendor/X-Optional would be an idea. When? by Squeeze+12 ? > > Importantly, the actual instruction to build the busybox-crush > > target must still be 'fakeroot debian/rules binary-arch' so that > > dpkg-buildpackage can still operate. > > As above, dpkg-buildpackage disagreeing with policy means that either > needs to be fixed -- dpkg-buildpackage is just the reference > implementation. No, it is the reality - packages for upload use dpkg-buildpackage - especially when considering cross-building. > > debian/rules can include a > > busybox-crush: target but it should point at binary-arch. > > Then the target would be pretty useless. It would be useful for debugging the packaging via fakeroot debian/rules busybox-crush > > The actual > > decision about which binary-arch instruction is used is then down to > > conditionals within debian/rules which can be influenced by > > DEB_VENDOR or other mechanisms within debian/rules itself. It > > doesn't matter how that support is delivered, as long as it is in > > place prior to calling dpkg-buildpackage - so we can continue to > > test the mechanisms via 'emsource' patches and then push those > > changes into the Debian package. > > "Optional" basically adds an intermediate option between unmodified > packages and full DEB_VENDOR handling -- when simply building > additional variants on demand is enough. That isn't going to work, it needs to be about a complete replacement package. > > A major potential problem is that debian-xcontrol is positioning > > itself as being used as part of the build process - but modifying > > debian/control after executing debian/rules is meant to be a Policy > > violation. > > That's why I use it only to check whether the control file is still > up to date, and fail the build if it isn't. For the current feature > set, this should never happen on an autobuilder, since it means > someone forgot to rebuild the control file prior to uploading source. OK, so xcontrol is used when updating the patches and the debian/control file is modified by the patches. However, I'd much rather get rid of the patches entirely, eventually. > For the "for every linux-image-* package that exists, build a > "foo-modules-*" package feature, that will change -- xcontrol check > failing indicates the need for a source update (list of binary > packages has changed) and "xcontrol build" provides an easy > standardized way of updating such a package. Crush isn't about kernels and modules, so I really don't care about that kind of support, sorry. -- Neil Williams ============= http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/
Attachment:
pgp7jP6yauv0O.pgp
Description: PGP signature