On Sun, 7 Nov 2010 20:01:20 +0000 Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote: > > I want to be able to (cross-)compile my firmware for Cortex M3 or > > MSP430. (bare-metal) Then that's just the Emdebian toolchains - no need for anything else. (We do that already.) Firmware doesn't need a build system or packaging tools. It's just built using the bare toolchain - any toolchain. Emdebian supports that because our toolchains are to be installed on Debian to be able to do this on Debian. > > Debian (binary based) != OpenEmbedded (source based) > > that's the traditional view. actually, i'd say that debian is a > hybrid binary+source based. and emdebian dpkg-cross-compilation is > _definitely_ source-based. Not true. dpkg-cross is BINARY based. Emdebian is binary based - solely binary based. dpkg-cross cannot operate with source, it can only operate on the binary package - in nearly all cases that binary was built natively by Debian. Show dpkg-cross a tarball and it will fail. Building any package is source-based but only so far as the source of that one package. Everything else needed for that one build is provided solely as binaries. Emdebian Crush was binary based too - we didn't build against our own cross-built packages, instead we built against the native binaries converted by dpkg-cross. (Right or wrong, that's what actually happened - there were reasons, lack of resources was one.) Like Debian, Emdebian builds one package at a time and uses binary packages to provide the dependencies. Emdebian and Debian never download source packages from random upstream websites - except when maintainers are preparing new uploads. Such builds are asynchronous and do not prevent other maintainers preparing their builds independently, unless specific reverse dependencies are involved. The only time anyone tries to build an entire series of packages is when testing Debian for bugs. *None* of the packages built in these tests are *ever* uploaded anywhere public. The entire process is done merely to identify the packages which fail to build natively. The process is designed to produce binaries and then move on. Individual failures do not affect any other package - in this particular case, even reverse dependencies are not affected because these are *rebuilds* of binaries that already exist at this precise version number and if binaryH fails, packageK which build-depends on binaryH-dev will still be able to get exactly the same version of binaryH-dev from the Debian binary repositories. The entire purpose of this process is *NOT* to allow changes to the packages being built - to mimic the process of the original build as closely as possible. We did try to create a process which allowed modification - this was what emdebian-tools tried to do and would have taken an insane amount of maintenance to do that for more than one architecture, let alone variants. > so it's not quiiite as clear-cut as it looks. remember that just > because _you_ install binary packages doesn't mean that the packages > come out of thin air. All free software comes from sources but that doesn't make Debian or Fedora source-based in the way that Gentoo and OE are source based. > (and yes, btw, i'm discovering that > openembedded supports the same sort of concept of "-dev" packages) The OE build is still a monolithic process. Debian builds one source package at a time, uploads that and then builds the next ones. At any point, a collection of binaries have been uploaded so that if one fails, the rest of the build does not have to fail. The uploads are then built natively for other architectures and maintainers use the binaries to support their own builds of other packages. I have toyed with the idea of breaking up the OE process and populating a repository one package at a time. If one package fails, so what, move on to the next one until you get to a log jam where everything left depends on packages that failed to build. That is how Debian ports are created. Once the problematic package builds, the rest follow and if the next release breaks again, well at least you've got the old version and you only have to care about those reverse dependencies which need the updated version. At each increment, the number of failed builds falls but the number of installable packages rises. Nobody should be expected to build-the-entire-world unless that world encompasses only kernel+busybox+private_app or just kernel+firmware. Building a monolith is *always* a BadIdea (TM) because one breakage at step 245 breaks all 12,000 steps and there is no process anywhere where all 12,000 steps will always succeed. The more packages you add and the more architectures/variants you add, the lower the odds. With something the size of Debian, the odds are worse than playing the lottery - and the electricity costs alone make it more expensive. OE builds fail regularly. Debian packages fail to build from source regularly. The difference is that when a Debian package fails, it is only that one package that is affected and everyone else continues as normal. Generally, the failed package isn't even uploaded so only that one maintainer is affected - not the entire image. If Emdebian was to use a Yocto/OE model, it would have to be morphed into an asynchronous, incremental model which relies on an independent repository and where builds simply skip anything which fails to build, just moving on to the next. Equally, the build must be stateful and pick up where it left off, not try to rebuild stuff it has already built at the same version. That is the Debian binary model and tying those two together is a huge amount of work. That is why expecting a Debian build sequence to give you a bootstrap image direct from the Debian or upstream source packages is a nonsense. A Debian build sequence gives you a repository of binaries and you or someone else then prepares the image from that repository, IF, the dependencies necessary exist. If not, the repository still exists for someone else to make an image from the same binaries but with a different list of packages to install. That is where multistrap comes in. Debian Live, debootstrap, multistrap - these are all solely binary based. They are completely independent of the build process and can use any repository of binaries, built however you desire, as long as the complete set of packages exist. It is entirely possible for Debian to give you an image built from binary packages where some of the original source packages no longer build successfully (natively, let alone cross or with modifications). Eventually, someone will notice and file a bug but those people creating the image from the binaries have absolutely no way of knowing whether the relevant source code would actually build anymore. This weakness in the Debian binary model is handled by various Debian people running intermittent builds to check for failing packages, as described above and by maintainers who are working on the next release and notice that their builds start failing for reasons unrelated to their changes. > so you're missing the point by saying debian = binary-based, > openembedded = source-based: i envisage that openembedded would be > utilised to _build_ binary-based .deb packages, not "openembedded > would force the installation of source-based packages" which it > doesn't do anyway. The point is that OE is utilised to build in a single synchronous process using largely unchanged source packages directly downloaded from upstream and if one build fails, bitbake fails the entire OS build. Debian is utilised to build in a multiple layer asynchronous process using source packages and Debian modifications prepared by individual maintainers with detailed knowledge of that package. Binary packages are uploaded with sources and the build of any one package can only affect builds of it's own reverse dependencies which dramatically reduces the impact of single package build failures. "Installation of source-based packages" makes no sense. OE doesn't do it, Debian doesn't do it - even Gentoo doesn't actually do that. Source doesn't get installed, it gets compiled - unless it's for an interpreter. The difference comes in how build failures are handled. OE is monolithic / synchronous, Debian is asynchronous and utterly binary-reliant. > i envisage - quite literally - openembedded calling dpkg-buildpackage, > and that's pretty much it, besides setting up the cross-compiler > arguments, and taking care of situations where it all falls in a heap > :) I tried that model with Emdebian Crush - it fails. > ... which openembedded have already done the work on, thus making the > maintainability "burden" equal to zero.... That is simply untrue. The OE changes do not transfer to Debian without an immense amount of work. I've been subscribed to the cross-dev mailing list which was set up for this purpose since it started - the success rate is far from impressive and is diminishing as time goes by. The OE and Debian build systems are orthogonal, the package versions do not match up, the purposes of the patches are incompatible and patches inter-depend on each other and other dependencies. If patches come from the wrong version of the source, the patches are not going to apply cleanly, often they won't apply at all. If you think you can simply copy OE/patches/* to debian/patches/ you are living in a dream world. The simple reality of independent release cycles means that version numbers of upstream, OE and Debian never actually coincide for more than a handful of packages at any one time. Even then, the dependencies of those packages do not match up, so the patches differ again. Differing patches means maintenance work. > why exactly. well for a start, i'd want, just like that guy who did > a partial rebuild of the entire debian mirror with Cortex A8 > optimisations, to do instead of a partial rebuild of the entire debian > mirror to do a TOTAL rebuild of the entire debian mirror with Cortex > A8 optimisations. Just build the ones that would benefit, not everything. What is the point of cross-compiling debhelper or random perl modules? In most cases, they don't cross-compile anyway. Cross-building architecture independent packages which already exist as natively built binaries is nonsensical and an immense waste of effort. Most would not benefit from any optimisations - certainly have a go at the interpreter but optimising python, perl, lisp and ruby is a specialised job, requiring detailed testing and QA of several thousand packages. i.e. a job for Debian. Guess what, Debian has already done that. If you fancy second guessing the Debian maintainers, go ahead. It will take you a while and it's every bit as hard as optimising gcc without breaking any other gcc support. Emdebian Crush got around this by simply dropping all interpreters but that makes everything else VERY difficult, which is one reason why Crush is now dead. > for my special super-duper lovely production screamingly-fast > ship-it-noww product i don't _want_ a debootstrap in less than a > minute, because that debootstrap will have been precompiled with utter > shit options for the N out of 600 ARM licencee variants, and i want > the absolute fastest, or the absolute most efficient space-saving. You want Gentoo or OE, not Debian nor Emdebian and it might build or it might not. What you do does not necessarily affect which result you get. There is no point in Emdebian wasting time duplicating that functionality when Debian does not support ultra small sets of packages which do not rely on eglibc, coreutils and perl. We proved that with Crush - the Debian additions will not operate with the busybox utilities without a huge amount of Debian-specific patching. Without the Debian additions, it's not Debian anymore. > remember that you make these choices for people, whether to enable > thumb instructions which give a better compression ratio but worse > performance, you are _never_ going to have absolutely everyone happy. ditto, Gentoo/OE. Changing those choices requires an immense amount of effort. Packages simply do not build if you change the basic assumptions of the rest of the system. > but, again, this misses the point: the purpose of suggesting that an > openembedded recipe be created which utilises dpkg-buildpackage (or > cross-dpkg-buildpackage to be more precise) is to save those poor sods > who have to create those QA'd debootstrap images one f*** of a lot of > time. It doesn't save time, it simply transfers the time burden to us and we do not have the resources to carry that burden. We have looked at this. We have spent hundreds of hours collectively studying these problems and I have to tell you that you are wasting your time and ours by trying to reinvent this particular wheel which is, TBH, so rusty and fragile that it will ne'er turn again. Been there, done that, got the T-shirt and even got some binary packages and patches to prove it. It did not work then, it does not work now and so much of cross-building in Debian is currently broken that there is little chance of it working again for at least a couple of years. > emdebian / crush is just a different mirror with a different series > of packages. No, Emdebian Crush is/was an entire cross-built set of packages with a different set of patches and priorities. Emdebian Crush was a set of recipes (patches and build scripts) to move Debian away from the typical server model and closer to an embedded model. It failed because it is unmaintainable. > i envisage that openembedded-dpkg-cross-buildpackage > would simply be set up to have a different set of stuff in > sources.list, and that a build would... just... work! Dream world. Debian is not source based. What goes into your sources.list is a URL where apt can find binaries and we tried to create a magic set of scripts to patch the sources that created those binaries and change the functionality - it is unmaintainable. > > Do we need to have 100 developers to maintain the > > thing? > > i'd say no. if it turns into 100 developers, then you're seriously > doing something wrong. It's more likely to need 200. i.e. most of Debian. i.e. you've proved that this is unworkable because you are right - cross-building everything does need huge amounts of effort and therefore it is seriously the wrong thing to do. It did turn into that amount of work and therefore, yes, it was seriously the wrong thing to do. Please believe us, it is simply a very BadIdea (TM). > the idea is to simply end up with the exact same .udebs and .debs > that you'd normally have to cross-compile by hand. Then there is no point building them again - use the ones you have, native ones. Only cross-build where you absolutely must change functionality or the package cannot go into Debian at all. Everywhere else, native build. It's the only sane way to change the functionality of Debian binary packages. -- Neil Williams ============= http://www.linux.codehelp.co.uk/
Attachment:
pgpgph321FO2s.pgp
Description: PGP signature