Hi Neil, On Sun, Oct 23, 2011 at 08:11:25AM +0100, Neil Williams wrote: > On Sat, 22 Oct 2011 14:58:01 -0700 > Steve Langasek <vorlon@debian.org> wrote: > > I've been having lots of fun playing with the 'apt-get build-dep -a $arch' > > support added to apt in version 0.8.15.3 (fixing bug #632221). As I do so, > > however, I think we've made a wrong trade-off in terms of the default > > behavior for packages. I've actually thought this for a while, but it's > > become enough of a hassle now that I think I need to raise it as a bug and > > agitate for getting the behavior changed. :) > IIRC the reasoning for the original default was to provide > cross-building behaviour post-Multiarch which was the most similar to > cross-building behaviour under dpkg-cross, with the fundamental change > that the cross-dependencies in Multiarch world would be understood by > apt and dpkg. > Logically, I still think this should be the long term goal. What I > think we're covering here is the interim stages whilst there is this > lack of clear direction on the behaviour of -dev packages in Multiarch > and a lack of -dev packages with appropriate changes. > If we can get packages to work with the long term goal it would make it > easier to achieve the goal without having to make the changes twice. I think there are three long-term goals here: - that it be clear from any build-dep line which version of a package is needed / will be installed when cross building - that it be possible to express *any* combination of needed native/foreign build-dependencies in debian/control, and that these be actually installable for any package in the archive - that all the work done around this be integrated into the official Debian archive, and not carried as a delta. Now, I think it's worth pointing out that the M-A: foreign metadata I'm proposing to add in support of flipping the apt-get build-dep default is *not* incorrect: it is entirely accurate to say that these packages satisfy foreign-architecture dependencies and build-dependencies, it's just that we wouldn't have bothered to annotate many of these if they were only used as build-dependencies. Conversely, it's not incorrect to convert all the -dev packages so that they're coinstallable and mark them Multi-Arch: same. It's just that one of these things can be achieved much more quickly and with less developer effort, allowing us to begin reaping the benefits of multiarch for cross-building immediately provided that we flip one of the defaults in the spec. >> - Unlike marking a package Multi-Arch: foreign, which is just adding >> metadata, converting a -dev package to be Multi-Arch: same is not >> always easy. There are no best practice guidelines yet for converting >> a -dev package to be Multi-Arch: same when it contains a mix of >> architecture-dependent and architecture-independent headers, so many >> of these -dev packages are not getting converted right now. Of 696 >> packages that have been tagged Multi-Arch: same in unstable[1], only >> 74 of them are -dev packages. Since some of the affected packages are >> pretty low in the dependency tree, this effectively stops us from >> reaching critical mass of cross-build-friendly multiarch -dev >> packages. > Exactly - so in the interim period where there are no clear guidelines > yet and no critical mass, do we actually need to change the default or > just accept the *result* of these complications? So we had a couple of sessions at UDS/Linaro Connect this past week where this issue was discussed with the apt developers and with folks interested in ARM cross-compiling. I really ought to have invited you to follow the sessions remotely, but it slipped my mind in all the pre-conference planning. I apologize! You can read the session agendas and "minutes" here: https://blueprints.launchpad.net/ubuntu/+spec/ubuntu-arm-p-cross-compilable-packages http://summit.ubuntu.com/uds-p/meeting/19538/ubuntu-arm-p-cross-compilable-packages/ https://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-lc4.11-multiarch-crossbuilding http://summit.ubuntu.com/uds-p/meeting/19823/linaro-platforms-lc4.11-multiarch-crossbuilding/ There was a very strong consensus among those present that we would target multiarch-based auto-cross-building of the relevant packages for the Ubuntu Precise cycle (April 2012 release). So one way or the other, there will be a core set of cross-buildable packages available by then. But I don't know if it will be a /critical mass/ if we have to stop to convert every -dev package along the way, vs. letting this conversion continue to happen in parallel. There was also a consensus that defaulting to DEB_HOST_ARCH for M-A: no build-dependencies is the better route, because it lets us get farther up the stack much quicker. This doesn't preclude getting -dev packages made M-A: same (indeed, I just filed bug #647855 for libattr1-dev), it just means a different short-term focus. >> - Whereas cross-installability of -dev packages is a must for >> cross-building, for the vast majority of libraries, >> *co*installability is merely a nice to have. There are very few >> libraries that you need to build against both the host and build >> versions of as part of a single package build; so while it's >> inconvenient to have to swap -dev packages out in a build environment >> when switching target architectures, it's tolerable... especially >> since it would enable uses that aren't possible today. Only the apt >> implementation makes M-A: same a requirement. > The result being, of course, that swapping -dev packages in and out of > the build environment is simply not going to happen (because people > will generally need to be able to make a bug fix natively, cross-build > it quickly and get back to development). Instead "swapping -dev > packages in and out of the build environment" simply translates into > "requiring cross-builds to happen in chroots" like pbuilder, schroot > etc. > I think we can specify that without needing to change the default > behaviour. Are there reasons why this wouldn't work? Effectively we > don't hot-swap build-deps, we just have two build environments. That's an interesting take on it that doesn't really resemble my own development process these days. I do all of my development and test-builds in a chroot; and because chroots are cheap (especially with schroot+LVM snapshotting), if I need to fire up a separate native chroot next to the one I'm using for cross-building, I can easily do so. And I have a local mirror for the things I'm working on. I guess you're doing something else that makes swapping -dev packages more expensive in your development environment? I think that's one reason among many why we want to make -dev packages co-installable. My only concern is with decoupling this problem from the problem of being able to auto-cross-build in a pristine environment. > The packages which caused trouble with this in Lenny were GTK and pango > which both build native tools which are executed during the build to > generate package metadata which is used by the cross-built binaries. > There was always a problem here about potentially getting the wrong > data because the work was done on the wrong arch and potentially in the > wrong directories (due to dpkg-cross). Those problems are fixable but > *will* require the build-dependencies of the native tools to be present > in the same build environment and at precisely the same time. > libglib2.0-dev is one of the most likely here. No disagreement! There are a number of -dev packages whose conversion to M-A: same is critical path. I just want to separate these out from the ones that don't need to be. > Upgradability of cross -dev packages is the single biggest gain I need > from Multiarch. Proper cross-architecture installation dependency > resolution and management. I don't really care if I can't have -dev > packages for armel and amd64 in the same build environment initially - > as long as that can happen eventually. What I need is for armel build > dependency packages to be automatically and easily upgradable in-situ, > using apt itself, without the need for secondary tools like apt-cross, > xapt etc. Yep, and you definitely get that whether or not a given -dev package is coinstallable. > i.e. apt-get build-dep -a $arch isn't actually that much of a gain - we > can do that without MultiArch being complete. I think it's a *huge* gain to have the standard tools do the right thing for cross builds. Among other things, it facilitates setting up cross-autobuilders. If you consider apt-get build-dep -a $arch not a big gain, does that mean you don't mind changes to its behavior as long as it doesn't get in the way of the other things you're working on? BTW, there's another benefit to changing the interpretation of M-A: no build-deps: it brings it into line with the interpretation of M-A: no in Depends, which means that tools like mk-build-deps, which converts a Build-Depends: line into a Depends: line, can also work in a cross-building context. (cf. bug #647745.) Currently, apt-get build-dep gives an opposite interpretation, making it impossible to sensibly map these. > If we accept that switching -dev packages in and out of a build > environment just makes cross-building unnecessarily painful for actual > cross architecture development (rather than cross-arch distro > preparation) then we have to mandate that until the guidelines for -dev > packages are more achievable, that cross-building in MultiArch world is > restricted to chroots. If that is accepted, then it doesn't need a > change in apt - the chroot can use DEB_BUILD_ARCH in the chroot as > easily as it can use DEB_HOST_ARCH, just without needing a change to > the default. I'm not sure what you're arguing here. Are you advocating a fully emulated chroot? That would be significantly less useful than a proper cross-chroot. Again, my goal is to help get a quicker payoff from these changes, because that will encourage more people to work on this in parallel. If you're not using apt-get build-dep -a $arch anyway, then changing its behavior doesn't seem to cost you anything... but it does immediately benefit anyone trying to do automatic or (semi-automatic) cross-builds of packages. *That* is what will enable us to reach critical mass most quickly. > > Cc:ed to multiarch-devel and debian-embedded for comments. Since this > > proposal implies pushing more patches to packages to make them Multi-Arch: > > foreign, perhaps this should be discussed on debian-devel to get a consensus > > there? > If we mandate chroots, until critical mass is achieved, would we still > need changes? If you mean fully-emulated chroots, that's not something that can be mandated. Various people working in this space have need of cross-build environments (chrooted or otherwise). If you mean cross-build chroots, then yes, this change is absolutely still relevant. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer http://www.debian.org/ slangasek@ubuntu.com vorlon@debian.org
Attachment:
signature.asc
Description: Digital signature