On Sun, 26 Apr 2009 18:57:37 +0200 Martin Fuzzey <mfuzzey@gmail.com> wrote: > >> There doesn't seem to be any process to > >> allow contributions of "emdebianisations" of _lenny_ packages, only > >> _sid_ ones. But I'm not at all sure people developping a linux > >> embedded device want to be running sid any more than a system > >> administrator would want to run it on their production servers. > >> > > > > A few issues here. > > > > 1. Yes, Emdebian is Debian, not Gentoo or OE. One of the core > > features of Debian is that the *default* build environment for all > > those binary packages is *Sid* and that migrations into testing and > > finally into stable primarily happen involving pre-built packages > > that were originally built against Sid. There are carefully managed > > mechanisms in Debian to allow packages to be uploaded directly to > > testing and stable but these are intensely labour intensive, > > relatively complicated compared to standard Sid builds and are only > > used when essential - i.e. to fix release critical bugs. Out of > > 20,000 packages that went into > > Hum, not being a DD there are are a few things I don't understand in > detail. I think I understand the basic sid->testing->stable migration > and that in Debian (unlike some other .deb based distros) the actual > binary .deb uploaded by the developper is used for one architecture > and the others are built by the buildds [though I believe this is / > has been a subject of discussion on -devel] Yes, there are ongoing discussions about whether the binaries uploaded by maintainers should be used directly - there are arguments for and against but that's off-topic. > What I'm not sure about (sorry a bit OT for this list) is how Debian > handles something like: > > 1 Jan: Upload of package X built with current sid toolchain (say gcc > 4.3.2) 1 Feb: Sid toolchain updated to gcc 4.4 > > In this scenario what ensures that X is buildable with gcc 4.4? A binary-NMU - a system whereby the package is "given back" to the buildd machines, including the buildd for the architecture that the maintainer originally uploaded. You'll see these packages in your system with a +b1 version suffix. The Release Team manage such operations and there is currently no real method of doing this for Emdebian Crush. (Even in Debian, it is not entirely automated.) By using edos-debcheck in the Emdebian build / upload scripts, we are able to identify uploads that would break other dependencies and this helps us to rebuild those packages as well. That is OK to do in unstable, it becomes much more problematic for stable. The version of gcc will not change in Debian stable - neither should the SONAME or API of any other package - for precisely this reason. As long as you take your Debian package from Lenny and then cross-build it on Lenny for Crush 1.0.x, things will work. > [until the next version of X is uploaded]. I use gcc as an example > but it could apply to any Build-Depends. Do all package maintainers > have to generate new versions of their packages for this or do the > buildds rebuild everything when a build dependency changes? It can be either - if the package is due for an upload for other reasons, then it is easiest to do the rebuild at that stage but that, again, is targeting unstable. The +b1 upload also goes into unstable. > I know > that during the release process the toolchain is frozen first but > that doesn't explain what happens to packages already in testing that > don't have any later updates. The +b1 then migrates as normal - or in a release freeze, it gets put into testing-proposed-updates etc. Such changes are simply not made in a stable release - unless absolutely unavoidable for a release-critical bug. > > 3. Cross-building for testing or stable is immensely more difficult > > than cross-building for sid because all the dependencies change. > > Until Lenny was released with (an old version of) emdebian-tools > > available, it was all but impossible. Lenny has made that > > achievable. > > I understand the second part of this (obviously until working tools > are available it's hard to do anything) but not the first part. Given > that emdebian-tools is available in lenny why is it "immensely more > difficult" to cross build lenny packages on lenny than sid packages on > sid? OK, maybe I over-stated that initially. Building for testing or stable is more difficult for a few reasons: 1. Limitations of the method itself (i.e. bugs) 2. Patches that have not made their way into the Debian source package are in step with unstable, not necessarily testing or stable and so the build method needs to be adapted to get the source package with patches included - getting the source package from Emdebian instead of Debian and/or using a branch in SVN. 3. Dependency changes in the packages themselves. The current patches implement functionality and ABI changes and don't do this particularly cleanly because the tools were still evolving and the patches ended up doing too many different things. (Another component of the Code Audit). The state of those patches also complicates migrating those changes into the versions in unstable or using those patches as examples of how to fix other packages. These are not massive problems in and of themselves, but collectively, these issues will take quite a bit of time to fix. > I'm quite willing to believe that the the emdebian-tools now in > sid are better than those in lenny (I've only used the lenny version > on just a few packages but I didn't find it particularly difficult - > maybe I was just lucky). I also understand that over time it will > become easier due to the code audit causing changes to the Debian > packages themselves to facilitate cross building and that this is > something that can only be done in sid. I just don't see why if > _today_ I choose some package from Debian and emdebianise it twice > (in lenny and in sid) I should expect an immense difference in the > difficulty. Now that lenny has been released, it is easier than if you were using oldstable but problems within the packages themselves will make things harder than they should be. > > 4. Since Lenny was released, there hasn't been time to fix the build > > system and the build mechanisms used for Emdebian Crush 1.0 are only > > usable for the builds you want - cross-building packages for Lenny > > on Lenny. There are known bugs in emdebian-tools 1.4.3 - > > backporting 1.6.0 might be manageable but not later versions. > > However, the patches themselves have changed (as a result of the > > Code Audit) to work with emdebian-tools 1.9.0 and later. The > > existing packages might be upgradable using the existing sources in > > the repository but patches for new packages are going to be > > out-of-tree and that has different problems. A branch in SVN is > > probably the simplest method but care will still be needed. > > > Yes I am imagining one svn branch containing patches against the lenny > version of the Debian packages to be used with the lenny version > emdebian-tools [though if a backported version of emdebain-tools would > be useful and possible why not] and another containing patches against > the sid version of the Debian packages to be used with the sid version > of emdebian tools. Yes, this can be done, it just hasn't been done yet. > But this raises another question (not related to my main point of > stable support) - how does emdebian track changes in sid which is > itself constantly evolving for reasons unrelated to cross building? Via the autobuilder, originally. Things have gone stale since Lenny because the ARM packages providing the dependencies have disappeared. The autobuilder works by getting the newest Debian package and trying to apply the patches from the previous version. If the patches apply, it tries the cross-build and if it all checks out, the patches are updated to apply to that version. Quite often, one of those stages fails and a human (usually me) has to go in and work out why, fix the patches and upload. i.e. I build the first version and then update those subsequent versions that fail to rebuild. Some packages never fail, some packages always fail. Murphy's Law dictates that the ones that always fail are the ones that have the most complicated and longest builds, like gcc. > Though in practice I would hope once a package succesfully cross > builds most changes to it shoudn't break the cross building > [especially if the needed changes can be rolled into the Debian > package which is the purpose of the code audit as I understand it.] Yes. > > Debian doesn't add lots and lots of packages with updates to stable > > releases but I think Emdebian can get away with doing that in the > > 1.0 release as it's such early days. I don't think we should expect > > that to be possible in future stable releases. > > > True but I think Debian and Emdebian are different in that the role of > Debian is to take lots of upstream sources and massage them into a > coherent and easily usable whole. This coherent whole is already the > starting point for Emdebian so this work doesn't have to be done > again. Emdebian's job is to enable cross build support and shrink it > to something usable on an embedded device. When I talk about adding > packages to Emdebian stable I'm _only_ considering packages _already_ > in Debian stable, not packaging some random tarball from sourceforge > (which would be the equivalent of Debian adding packages to a stable > release). Yes, however, 99% of packages in Debian do not cross-build at the first attempt. Even taking a new package that already exists in Lenny but does not yet exist in Crush means a potentially significant amount of work to get it to cross-build. It's not impossible, just hasn't been done routinely at this point. Debian does take lots of upstream sources and create a cohesive whole. Unfortunately for us, the model that drives that process does not consider cross-building to be a requirement - it is seen as "nice to have but not a blocker if it breaks". > Indeed, if a tool is available that takes a list of Debian packages > and generates a repository of gripped versions of those packages the > need for a central official grip seems to be diminished (I may be > missing a few details here though) That is part of the point of Grip, yes - that is why I keep the code updated in the emdebian-grip-server package so that anyone can install that package and create another repository of gripped packages. > What I meant was that if more packages are added to Crush 1.0 (using > the lenny build system) that will mean more work migrating them to the > squeeze build system for Crush 2.0 than if no packages had been added. > On the other hand it means a more usable Crush for the (2?) years > until Squeeze. Yes, it is true that creating patches in a branch for Lenny does mean that the patches need to be modified to apply against the versions of those packages in unstable so that the Debian package can actually be fixed. Whatever changes we send back to Debian, the patches have to apply against the version of the package in unstable. Indeed in many cases, the patches have to apply against the version of the package in an upstream VCS. > > Migrations are difficult to do well, yes. There are tools that can > > help but the risk of completely breaking a stable release would be > > high. Hence, updates need to go into stable-proposed-updates and be > > available for testing without replacing the package released in > > Crush 1.0 until Crush 1.0.2 or later is ready. > > > > > Yes but you are talking about _changing_ a package already included in > Crush. What about adding a package (from lenny) to Crush? Cross-building for Crush can involve turning off certain options to ./configure or disabling certain functionality or removing / replacing certain scripts in the package. All of these changes can have unpredicted effects on packages that depend on the modified package - the reverse dependencies. These kind of changes need testing to ensure that the package added from Lenny (which itself is likely to require at least some changes to be able to cross-build at all) is compatible with the potentially modified packages already in Crush. That's the core problem with Crush - Wookey spotted it early on but it is a hard problem to fix properly. One example is that gconf in Crush is built without LDAP support. Any package building against gconf which expects to find LDAP symbols in the gconf shared library will then crash - but it will only crash when tested on the device running Emdebian Crush. Another more subtle example is that gnupg is built with --enable-minimal which turns off lots of options internally and could have unpredicted effects on packages not already tested in Crush. The use of busybox also has a huge impact on testing packages from Lenny. Debian expects coreutils, bsdmainutils and perl but Crush replaces all of those with busybox. This is a very low-level change and packages that were built and designed for coreutils can generate new and surprising bugs when asked to work with busybox. Adding to the complexity is the fact that we are still using such an old version of busybox, for reasons described previously. This area affects all kinds of packages - the dependency changes principally affect packages with lots of dependencies in Debian, so those are the GUI libraries. The use of busybox outside the initramfs causes new bugs in lots of low level tools and the lack of perl requires rewriting some fairly basic perl scripts in POSIX shell. Lots and lots of new bugs is inevitable. This is where Grip is so much easier, we don't take the ground from underneath the package in the same way as happens in Crush. Therefore, you cannot actually just take a package from Lenny and add it to Crush, you have to take a package from Lenny, change it to work with Crush and then test those changes in a real Crush environment before uploading (to avoid breaking stuff for others). > > Thanks - some very clear summaries. Hopefully, the problems outlined > > won't put you off actually getting some builds done using Debian 5.0 > > "lenny". > > > As I said I've been doing mostly kernel stuff and using emdebian as a > very convenient way of getting a working userspace. The time is > approaching however when I will have to choose how to do the real > userspace. I already know that the current set of crush packages is > not sufficient for my needs (but I haven't evaluated the exact > package set yet). I prefer the "emdebian vision" I outlined to the > alternatives and don't mind if it entails a bit of extra work > provided: > > a) I can realistically do that work in a reasonable time scale [for > that I'll need to determine the exact packages I need and look at > them] As soon as you have that list, let me know because I'll need to push the existing packages in that set through the Code Audit to permit the later stages of your work. > b) It's not wasted work that will have to be redone over and > over again [the reason for my post] Absolutely agree. > c) There aren't any show stopper technical problems [I need to look > more closely at the space required for the package cache] OK, some figures for you. You'll need a fairly complete build tree and space for a few branch builds. From my own systems, that can easily get to 9Gb. As for show-stopping problems, I don't think that will be an issue - more likely a slow drip-drip process of smaller package-based problems. Deciding in advance just how many packages you want to handle will be critical in the amount of time necessary. One practical problem is that you will principally be working with fixing Debian packaging problems in Debian packages in build methods that are based on Debian. i.e. you need to know quite a lot about Debian packaging and how to use it to fix your problems. You will need to be familiar with Debian packaging and when you come across a build error, do whatever you can to test the changes in a native Debian build to isolate the problem. 'embug --prepare' has some support for assisting such builds. -- Neil Williams ============= http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/
Attachment:
pgpwolo2TteCw.pgp
Description: PGP signature