[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: emdebian development model



On Sat, 25 Apr 2009 17:34:11 +0200
Martin Fuzzey <mfuzzey@gmail.com> wrote:

> My understanding of the emdebian vision is that someone wanting to
> build a linux device would:
> 
> 1) Port the linux kernel to their hardware (including writing any
> specific device drivers)
> 2) Select the _prebuilt_ emdebian packages needed to support their
> application
> 3) Package their application as debian package(s)
> 4) Build a root filesystem using emdebian tools from 1,2,3

Agreed.

(Feel free to put that on the Wiki somewhere - indeed, a lot of your
points could go on a Wiki page to help others.)
 
> For me the essential difference between emdebian and other contenders
> (buildroot, open embedded) is that emdebian is based on binary
> packages whereas the others are source based.

Definitely. My original problems with OpenEmbedded always tracked back
to difficulties actually building stuff when all I wanted was to test
some prebuilt stuff.

Debian is binary-package based and Emdebian follows that model. (We
don't have the manpower to alter it.)

> So emdebian is like debian rather than gentoo...

Yes.
 
> However the two things I currently believe get in the way of this
> vision are :
> a) The restricted set of architectures (crush is only available for
> ARM) b) The limited set of available packages.
> 
> a) is being actively worked on so I won't say any more about it.
> 
> However I think there is a problem with b) because I don't see how the
> currently small emdebian team can possibly be expected to provide all
> the packages that may be required.

Actually, a) and b) are interlinked. The solution for a) will also
help solve b) because allowing more than one architecture to be
supported means providing sufficiently simple build mechanisms that
multiple architectures can be built for the same packages. It turns out
that such mechanisms also make it easier to build new packages.

(The fix for problem A is the Emdebian Code Audit.)

> Unless I am mistaken crush 1.0 is currently a (small) _fixed_ subset
> of lenny cross compiled with modifications where required to reduce
> footprint (remove perl, ...).

The set isn't fixed, other than that those are the only packages we've
managed to get cross-building in the current build system. The fix for
problem a) - fixing the build system will make it easier to fix b).

The set was determined by my own needs and the dependencies of those
packages.

One problem that will recur is that when you have a set of packages and
want to get to a state where you can install another set of packages,
it is very difficult to work out which packages you are going to build,
which ones you will modify to drop dependencies and which ones will
turn out to be buildable but not actually used. i.e. there is a lot of
wasted effort involved.

'empath' was one attempt to solve this problem but that codebase is
moribund due to recursion problems.

> There doesn't seem to be any process to
> allow contributions of "emdebianisations" of _lenny_ packages, only
> _sid_ ones. But I'm not at all sure people developping a linux
> embedded device want to be running sid any more than a system
> administrator would want to run it on their production servers.

A few issues here.

1. Yes, Emdebian is Debian, not Gentoo or OE. One of the core features
of Debian is that the *default* build environment for all those binary
packages is *Sid* and that migrations into testing and finally into
stable primarily happen involving pre-built packages that were
originally built against Sid. There are carefully managed mechanisms in
Debian to allow packages to be uploaded directly to testing and stable
but these are intensely labour intensive, relatively complicated
compared to standard Sid builds and are only used when essential - i.e.
to fix release critical bugs. Out of 20,000 packages that went into
Lenny for Debian, I'd guess that it was about a hundred or so which went
via testing-proposed-updates. As a proportion, the number of packages
that would have done the same in Emdebian Crush is close to one out of
the 700 which migrated automatically from sid builds - and one package
did go into Crush 1.0 directly, busybox. It was a very complicated and
difficult thing to do and was only done because it was a critical bug
fix in Crush (which has still not been fixed in Debian).

2. Emdebian has none of the infrastructure or manpower of the Debian
Release Team or Stable Release Managers. (Frankly, we don't have the
experience of either of those teams either.) These migrations are
fraught with difficulty and potential bugs.

3. Cross-building for testing or stable is immensely more difficult
than cross-building for sid because all the dependencies change. Until
Lenny was released with (an old version of) emdebian-tools available,
it was all but impossible. Lenny has made that achievable.

4. Since Lenny was released, there hasn't been time to fix the build
system and the build mechanisms used for Emdebian Crush 1.0 are only
usable for the builds you want - cross-building packages for Lenny on
Lenny. There are known bugs in emdebian-tools 1.4.3 - backporting 1.6.0
might be manageable but not later versions. However, the patches
themselves have changed (as a result of the Code Audit) to work with
emdebian-tools 1.9.0 and later. The existing packages might be
upgradable using the existing sources in the repository but patches for
new packages are going to be out-of-tree and that has different
problems. A branch in SVN is probably the simplest method but care
will still be needed.

5. All the builds for testing-proposed-updates in Debian happened when
Debian unstable and Debian testing were all but the same - during the
Lenny release freeze. *Since* that time, unstable has changed beyond
recognition. Updates for Emdebian Crush 1.0 *must* be built exclusively
on Debian Lenny.

> Sure nothing prevents me or anyone else emdebianising lenny packages
> in a private repository but if everyone does this it will lead to
> lots of duplicated effort and an overall lower adoption of emdebian
> since the entry barrier is higher.
> 
> So I am envisionning emdebian-crush-stable,  emdebian-grip-stable
> which both consist a subset of lenny (same package versions) but
> where that subset expands over time as people contribute to "scratch
> their itch".

Grip already has stable-proposed-updates and support for that in the
scripts. Grip is so easy to manage that anyone only has to ask for a
package to be added to stable-proposed-updates. (Although I probably
haven't stated as much before.)

Debian doesn't add lots and lots of packages with updates to stable
releases but I think Emdebian can get away with doing that in the 1.0
release as it's such early days. I don't think we should expect that to
be possible in future stable releases.

> So in this scheme:
>     * debian - defines set of packages that constitutes a stable
> release and hence the maximum theoretical set of emdebian

Yes.

>    * emdebian developpers - work on tools, infrastructure and core
> packages and maintain repositories

Shortage of time for this currently.

>    * contributers - submit emdebianisations of stable packages.

For Crush, possibly. 
 
> I guess the problem with this scheme is that it will mean more
> packages to migrate when the tools change...

I don't think we should be considering changing anything in Emdebian
Crush 1.0 other than via migrations from stable-proposed-updates. That
does not include changing the entire build system beneath a stable
release.

Migrations are difficult to do well, yes. There are tools that can help
but the risk of completely breaking a stable release would be high.
Hence, updates need to go into stable-proposed-updates and be available
for testing without replacing the package released in Crush 1.0 until
Crush 1.0.2 or later is ready.

> Just my 2c
> 
> Martin

Thanks - some very clear summaries. Hopefully, the problems outlined
won't put you off actually getting some builds done using Debian 5.0
"lenny".

Please be careful and consider the points above to minimise risks of
breaking a stable release.

-- 


Neil Williams
=============
http://www.data-freedom.org/
http://www.linux.codehelp.co.uk/
http://e-mail.is-not-s.ms/

Attachment: pgpS9hy8VIC7I.pgp
Description: PGP signature


Reply to: