On Sat, 26 Mar 2011 20:06:21 +0000 Roger Leigh <rleigh@codelibre.net> wrote: > On Fri, Mar 25, 2011 at 05:06:02PM +0000, Hector Oron wrote: > > 2011/3/10 Roger Leigh <rleigh@codelibre.net>: > > > For the cross-packages, you can add these to the core build > > > dependencies list, so they will be installed along with build-essential > > > etc. Or (better) you can add a separate XAPT_DEPENDS variable which can > > > be appended to CORE_DEPENDS in Build.pm prior to installing the core > > > dependencies. Does xapt not bring in these packages directly, or do > > > they always need installing by hand? > > > > xapt downloads the foreign architecture packages using APT with -o > > APT::Architecture=$HOST_ARCH into a directory controlled by xapt > > (/var/lib/xapt/, irrc), then throws dpkg-cross into that directory to > > convert those packages and installs them. > > This is the part that I think can be significantly improved. Not until we have Multi-Arch-Cross. Indeed, there's no point making special changes to sbuild ahead of Multi-Arch as the entire cross-dependency resolution mechanism will be completely thrown away at that stage. What you're trying to create in this bug report is a Multi-Arch-Cross resolver for sbuild and then hacking it to handle dpkg-cross. That is the wrong way around. Use the hacks which already work with the dpkg-cross hacks and then implement a *clean* Multi-Arch-Cross resolver when that becomes possible. dpkg-cross is fundamentally broken and is completely alien to Multi-Arch. Anything based on dpkg-cross will need to be completely re-written for Multi-Arch-Cross, so use the tools which exist (and which will disappear alongside dpkg-cross) to retain as much isolation as possible between the sbuild code and the broken dpkg-cross methods. > sbuild creates a local APT archive to install dependency packages from. > We can use the same mechanism to allow the dpkg-cross packages to be > installed. It could even be used to cache generated packages between > builds. We can quite easily refactor the code to allow multiple > archives (if required). The problems in this area are: 0: The setup_apt_archive code uses the Dpkg bindings to resolve the dependencies but these are inherently native (despite taking an arch, it is difficult to get this right when there are alternatives and architecture-specific dependencies). These have been re-written in embuilddeps but all that code will be completely pointless once Multi-Arch-Cross arrives. There is no point implementing yet another broken version of it in an XaptResolver for sbuild. 1: embuilddeps has tried to solve these problems but has a fundamentally different approach to actually installing the packages. Sbuild::AptResolver and Sbuild::AptitudeResolver use the dummy-package method to actually do the work of resolving the dependencies within the specified constraints. embuilddeps resolves the constraints directly. e.g. constraints of foo (>= 1.2.3) [armel] | bar [!linux-any] | baz [i386| amd64] are particularly hard to resolve correctly when running on i386 to cross-build for armel. 2: Cross-dependency resolution requires re-doing the dpkg deps_parse calculation with an empty dpkg status file and then chasing the dependencies all the way down because -cross packages do not exist in the archives. This is why I'm looking at Multi-Arch being the preferred solution here. 3: dummy-package methods won't work for cross because we must resolve all the constraints ahead of passing a list of packages to apt so that these can be downloaded *ahead* of being installed, then converted using dpkg-cross and installed as packagename-$arch-cross packages. libfoo-dev_1.2.3_armel.deb becomes libfoo-dev-armel-cross_1.2.3_all.deb 4: The converted packages do not (indeed cannot) exist in any network-accessible archive - these are all generated on-the-fly until Multi-Arch makes the whole conversion avoidable. 5: apt and aptitude consistently treat -$arch-cross packages as "local or obsolete" and consistently get removal parsing wrong when upgrading existing packages. This means that the list of -$arch-cross packages to be removed needs to be carefully managed, especially if the cross toolchain is not to be removed upon every build. Much easier is for xapt|embuilddeps to put the downloaded and the converted packages into a directory which sbuild can manage, or for sbuild to become aware of the current defaults which are /var/lib/xapt/output for -cross packages and /var/lib/xapt/archives for the foreign architecture originals. This will also solve problems of package removal after the build, albeit that some extra processing is required to skip certain -cross packages which would otherwise cause the removal of the cross-building toolchain. (libc6-dev-$arch-cross is a frequent problem here.) This list of packages does need to be kept updated to cope with changes in gcc and in eglibc - e.g. the Squeeze chroots will have a libc-bin-$ARCH-cross package which doesn't exist in Lenny chroots. Whilst it is certainly a good idea to try and re-use the existing sbuild support for keeping the downloaded packages available, there are significant problems with doing this the way that sbuild currently does it. It would need to be a post-process update, reading in the list from the xapt directory. (xapt can be told to keep the archives around, it usually deletes them once all the -cross packages are installed and this option has also been added to embuilddeps, so the new -k option passes that down to xapt which keeps the /var/lib/xapt/ tree in place for subsequent handling.) I did think about putting that list into a file but the list is simply the list of .debs which were downloaded into a clean directory - one directory for the (typically) armel packages and one for the -armel-cross packages. A simple opendir(); readdir and pattern match on /^(.*)_.*_($arch|all)\.deb$/ or similar would get the list of package names and if /var/lib/xapt/* is removed by whatever does the subsequent package removal handling, then this list will be guaranteed to be the complete and exact list of all dependencies downloaded as $host architecture and installed as -$host-cross packages for that specific build. Overall, sbuild cross-build support ahead of Multi-Arch-Cross is best done by leaving as much of the work as possible to tools outside sbuild - tools which themselves will disappear once Multi-Arch-Cross is in place. Only at that stage is it worth looking at how the internals of sbuild can enfold the needs of the Multi-Arch-Cross world which, currently, is largely opaque. > > > Well, when we install the build dependencies the package isn't yet > > > unpacked. The unpacking is done in the build() function. But there's > > > no reason we can't reorder it if we need to. The unpacking could > > > be split out of build() into a separate unpack_source() function, > > > which could be run earlier if xapt/embuilddeps is in use. > > > > > > What information does embuilddeps require from the unpacked source > > > package? Is this information also available in the .dsc? (We do > > > have the .dsc information directly to hand). > > > > It is possible to use .dsc information, but embuilddeps does not know > > how (yet). :-) It will - in the next release. It's as easy to parse debian/control as the .dsc. I've got a change in SVN which adds a --dsc option to embuilddeps. So sbuild would actually use: embuilddeps -a $host_arch -k --dsc $dsc_file No use of -n, no parsing of the output - let embuilddeps call xapt to download the packages and dpkg-cross to convert and install them, then parse the list of downloaded and converted packages directly from the directory listing, handle the removal of the $host-cross converted packages when appropriate and then tidy up /var/lib/xapt/* Once we have Multi-Arch-Cross, the entire process folds back into a standard sbuild/apt|aptitude problem because M-A-C will *have to* allow dpkg to fix the problems of foreign architecture dependency resolution. (And it will do a better job of it than either sbuild or embuilddeps could do themselves.) > OK. For the purpose of actually implementing this, I would like to > have a description of exactly which fields we need for cross building > from which files in the source or DSC, and what they do and why. Is > this all documented somewhere? There was an idea to separate the "necessary" build depends from the "ancillary" build depends - the tools like debhelper and automake which need to be installed but also need to be executable, so need to be the build arch, not host. This was the xcontrol debacle and it's dead. Instead, embuilddeps parses the entirety of Build-Depends and Build-Depends-Indep as well as Build-Depends-Conflicts and resolves them all for the host architecture. This results in tools like debhelper being converted by dpkg-cross into empty debhelper-armel-cross packages but that can't be helped. (We tried to do that with apt-cross and it resulted in an unfixable RC bug and the removal of apt-cross.) It's another reason why this needs to be done properly via Multi-Arch-Cross, eventually. So the fields are exactly as per Policy - albeit that the parsing of those fields for a $host architecture is not currently supported properly by dpkg or apt. > > > xapt is doing two jobs it seems: > > > 1) from the package list, make a list containing all the cross packages > > > needed > > > > That is given with simulation option (-n). > > This is with embuilddeps, not xapt? embuilddeps parses the debian/control file or the .dsc to get a list of packages. That list is then passed to xapt which downloads the host architecture binaries, passes those to dpkg-cross which converts and installs the -cross packages. > This option appears to give a package list, prefixed with > "/usr/bin/apt-get -y". This just isn't suitable. > - it's omitted packages which were already installed; this is not > necessary and involves poking around under /var/lib/dpkg. > - we need to strip off the "/usr/bin/apt-get -y" which might be subject > to change and then cause breakage, at some point in the future. > All we want is the /complete/ package list, without any removal of > packages or addition of other bits. The list output by the -n|--simulate option does NOT include any of the dependencies of the packages listed. The complete package list needs to be parsed from the listing of /var/lib/xapt/archives (for the downloaded host architecture packages) and /var/lib/xapt/output/ for the -$arch-cross packages converted by dpkg-cross. > > > 2) install them (but it doesn't list them in the list of packages to > > > install, which makes removal harder potentially) See the directory listing. > > > From the resolver POV, what would be really good if it could just do > > > (1). This way, we can just give the expanded list back to the > > > resolver for it to install. The resolver can't handle the conversion of the package name from foo to foo-armel-cross and the resulting changes in the dependencies from Depends: bar (>= 1.2.3) to Depends: bar-armel-cross (>= 1.2.3) No resolver *can* handle that (current, past or future) - this is why we need Multi-Arch-Cross. xapt doesn't try to handle it. xapt simply gets the *unconverted* build-dependencies and downloads them, dpkg-cross converts them and installs. The entire process is based on simply converting and installing everything that appears to be a .deb in /var/lib/xapt/archives/ and putting the converted stuff into /var/lib/xapt/output prior to using dpkg -i *. > Because we need to clean up after > > > ourselves, the apt and aptitude resolvers build a dummy > > > "dependency package" from the dependency list, and then get apt/ > > > aptitude to install it. > > > > This is how old pbuilder used to do it, but it does not work with > > cross packages as adding dependencies on > > $foreign_package-$HOST_ARCH-cross won't work, as those packages are > > output from dpkg-cross and not available from repositories. > > Right. But with sbuild, as I mentioned above, we can install the > dpkg-cross-generated packages into a local archive, which *is* > accessible to apt-get/aptitude, and hence you will then be able to > use cross packages in the dependencies /directly/. This will permit > proper cleanup after a build using the normal mechanisms sbuild uses > for all builds. You can copy the converted packages in /var/lib/xapt/output into an archive, yes. Or you could just use the packages which are already installed and parse the directory listing to get the list to remove - always remembering to skip $host-cross packages which would otherwise cause the removal of the toolchain. This hack too will disappear with M-A-C. > This is to say the resolver can do the following: > - get the cross dependencies from the dsc or unpacked source > - download the debs > - run dpkg-cross > - install the dpkg-cross-generated debs into the local archive > - add the cross-dependencies to the package build dependencies > ... > - run dpkg-buildpackage with the necessary options to cross build > ... > > We already allow various things to add to the package build deps, > and so adding cross-deps will be trivial. All we really need is the > necessary information for what to parse from the dsc or source package, > and then how to use dpkg-cross. > > Just to be clear: none of the above uses xapt/embuilddeps, because > sbuild can, I think, do a better job with its local archive and which > will integrate much better with our existing cleanup code when it comes > to removing build deps. It will have to use embuilddeps because Dpkg::Deps.pm is *not able* to resolve the build-dependencies for a different $host architecture. It will need to use either xapt or something almost identical to xapt (which is not much more than a wrapper to apt-get -o Apt::get::Architecture=armel -d install $packages which sbuild would have to do anyway). All that's being changed is the final path. It's embuilddeps which is the attempt at being clever and resolving dependencies for a foreign architecture using modifications to Dpkg::Deb.pm. Doing this without xapt and embuilddeps is just going to cause more work because dpkg-cross IS going to be removed and the entire process will have to be re-written. > > > We then reverse the process to clean up > > > after the build. So ideally, we don't really want xapt to do any > > > installation: we just want it to give us the package list. If > > > this is possible, it might not even be require to have a separate > > > XaptResolver--we just expand the cross-dependencies and give them > > > to the regular resolver. No. There is no point re-doing all the work encoded within xapt and embuilddeps and then doing it all again when Multi-Arch-Cross arrives. Do it once - when M-A-C is usable. > > > If xapt can't do this directly, how complex is the code that > > > implements (1)? Would it be possible to split it out as a separate > > > tool? By the time you've got the list of converted packages, there is nothing for the resolver to do except dpkg -i * > > embuilddeps -a armel -d /path/to/source -n will give you that > > information, and /path/to/dsc might be in the way. That's meant to be human-readable, not machine-parseable. > > We really need to think more about how we are going to do it to be > > able to fit multiarch in the cooking. Exactly - use xapt/embuilddeps and then handle M-A-C later. > As I mentioned above, I really don't like this. It's not giving us the > unfiltered information we need. The unfiltered information you need is simply the directory listing of /var/lib/xapt/archives and /var/lib/xapt/output > And it's telling us nothing about what > xapt is actually going to install (in terms of letting us reverse the > procedure). Those directory listing specify *exactly* what xapt has downloaded and what dpkg-cross has converted and installed. > embuilddeps is giving us command lines to do the job. What we need is > the information to do that job ourselves. No. The information needs to come from the results of xapt and embuilddeps. -- Neil Williams ============= http://www.linux.codehelp.co.uk/
Attachment:
pgpGF6s9_Gypd.pgp
Description: PGP signature