Re: emdebian development model
Hi Neil,
thank you for the detailed reply.
Neil Williams wrote:
> (Feel free to put that on the Wiki somewhere - indeed, a lot of your
> points could go on a Wiki page to help others.)
>
Done for that part. For the rest I'll wait to see if anything else falls
out of the discussion and then try to write a summary for the wiki.
> Definitely. My original problems with OpenEmbedded always tracked back
> to difficulties actually building stuff when all I wanted was to test
> some prebuilt stuff.
>
Yes, for the moment I've been mostly doing kernel stuff so I really like
the idea of just grabbing the userspace from emdebian. And I actually
find crush pretty easy to use - sure not as easy as Debian itself but I
wasn't expecting that either.
> Debian is binary-package based and Emdebian follows that model. (We
> don't have the manpower to alter it.)
>
Nor do I think it should change - if you want a source based
distribution there are others.
>
>> There doesn't seem to be any process to
>> allow contributions of "emdebianisations" of _lenny_ packages, only
>> _sid_ ones. But I'm not at all sure people developping a linux
>> embedded device want to be running sid any more than a system
>> administrator would want to run it on their production servers.
>>
>
> A few issues here.
>
> 1. Yes, Emdebian is Debian, not Gentoo or OE. One of the core features
> of Debian is that the *default* build environment for all those binary
> packages is *Sid* and that migrations into testing and finally into
> stable primarily happen involving pre-built packages that were
> originally built against Sid. There are carefully managed mechanisms in
> Debian to allow packages to be uploaded directly to testing and stable
> but these are intensely labour intensive, relatively complicated
> compared to standard Sid builds and are only used when essential - i.e.
> to fix release critical bugs. Out of 20,000 packages that went into
>
Hum, not being a DD there are are a few things I don't understand in detail.
I think I understand the basic sid->testing->stable migration and that
in Debian (unlike some other .deb based distros) the actual binary .deb
uploaded by the developper is used for one architecture and the others
are built by the buildds [though I believe this is / has been a subject
of discussion on -devel]
What I'm not sure about (sorry a bit OT for this list) is how Debian
handles something like:
1 Jan: Upload of package X built with current sid toolchain (say gcc 4.3.2)
1 Feb: Sid toolchain updated to gcc 4.4
In this scenario what ensures that X is buildable with gcc 4.4? [until
the next version of X is uploaded]. I use gcc as an example but it could
apply to any Build-Depends. Do all package maintainers have to generate
new versions of their packages for this or do the buildds rebuild
everything when a build dependency changes? I know that during the
release process the toolchain is frozen first but that doesn't explain
what happens to packages already in testing that don't have any later
updates.
> 3. Cross-building for testing or stable is immensely more difficult
> than cross-building for sid because all the dependencies change. Until
> Lenny was released with (an old version of) emdebian-tools available,
> it was all but impossible. Lenny has made that achievable.
>
I understand the second part of this (obviously until working tools are
available it's hard to do anything) but not the first part. Given that
emdebian-tools is available in lenny why is it "immensely more
difficult" to cross build lenny packages on lenny than sid packages on
sid? I'm quite willing to believe that the the emdebian-tools now in
sid are better than those in lenny (I've only used the lenny version on
just a few packages but I didn't find it particularly difficult - maybe
I was just lucky). I also understand that over time it will become
easier due to the code audit causing changes to the Debian packages
themselves to facilitate cross building and that this is something that
can only be done in sid. I just don't see why if _today_ I choose some
package from Debian and emdebianise it twice (in lenny and in sid) I
should expect an immense difference in the difficulty.
> 4. Since Lenny was released, there hasn't been time to fix the build
> system and the build mechanisms used for Emdebian Crush 1.0 are only
> usable for the builds you want - cross-building packages for Lenny on
> Lenny. There are known bugs in emdebian-tools 1.4.3 - backporting 1.6.0
> might be manageable but not later versions. However, the patches
> themselves have changed (as a result of the Code Audit) to work with
> emdebian-tools 1.9.0 and later. The existing packages might be
> upgradable using the existing sources in the repository but patches for
> new packages are going to be out-of-tree and that has different
> problems. A branch in SVN is probably the simplest method but care
> will still be needed.
>
Yes I am imagining one svn branch containing patches against the lenny
version of the Debian packages to be used with the lenny version
emdebian-tools [though if a backported version of emdebain-tools would
be useful and possible why not] and another containing patches against
the sid version of the Debian packages to be used with the sid version
of emdebian tools.
But this raises another question (not related to my main point of stable
support) - how does emdebian track changes in sid which is itself
constantly evolving for reasons unrelated to cross building? Though in
practice I would hope once a package succesfully cross builds most
changes to it shoudn't break the cross building [especially if the
needed changes can be rolled into the Debian package which is the
purpose of the code audit as I understand it.]
> 5. All the builds for testing-proposed-updates in Debian happened when
> Debian unstable and Debian testing were all but the same - during the
> Lenny release freeze. *Since* that time, unstable has changed beyond
> recognition. Updates for Emdebian Crush 1.0 *must* be built exclusively
> on Debian Lenny.
>
Sure
> Grip already has stable-proposed-updates and support for that in the
> scripts. Grip is so easy to manage that anyone only has to ask for a
> package to be added to stable-proposed-updates. (Although I probably
> haven't stated as much before.)
>
Great - now you have :)
> Debian doesn't add lots and lots of packages with updates to stable
> releases but I think Emdebian can get away with doing that in the 1.0
> release as it's such early days. I don't think we should expect that to
> be possible in future stable releases.
>
True but I think Debian and Emdebian are different in that the role of
Debian is to take lots of upstream sources and massage them into a
coherent and easily usable whole. This coherent whole is already the
starting point for Emdebian so this work doesn't have to be done again.
Emdebian's job is to enable cross build support and shrink it to
something usable on an embedded device. When I talk about adding
packages to Emdebian stable I'm _only_ considering packages _already_ in
Debian stable, not packaging some random tarball from sourceforge (which
would be the equivalent of Debian adding packages to a stable release).
>> * contributers - submit emdebianisations of stable packages.
>>
>
> For Crush, possibly.
>
Yes I'm talking about crush here - as you have pointed out adding
packages for grip is much easier.
I haven't tried grip myself but my understanding is that it is just an
automatically filtered version of Debian (no recompilation - just
excluding unneeded things like documentation).
Indeed, if a tool is available that takes a list of Debian packages and
generates a repository of gripped versions of those packages the need
for a central official grip seems to be diminished (I may be missing a
few details here though)
>> I guess the problem with this scheme is that it will mean more
>> packages to migrate when the tools change...
>>
>
> I don't think we should be considering changing anything in Emdebian
> Crush 1.0 other than via migrations from stable-proposed-updates. That
> does not include changing the entire build system beneath a stable
> release.
>
Certainly. I wasn't proposing this - my intention is to use the emdebian
build system in lenny.
What I meant was that if more packages are added to Crush 1.0 (using the
lenny build system) that will mean more work migrating them to the
squeeze build system for Crush 2.0 than if no packages had been added.
On the other hand it means a more usable Crush for the (2?) years until
Squeeze.
> Migrations are difficult to do well, yes. There are tools that can help
> but the risk of completely breaking a stable release would be high.
> Hence, updates need to go into stable-proposed-updates and be available
> for testing without replacing the package released in Crush 1.0 until
> Crush 1.0.2 or later is ready.
>
>
Yes but you are talking about _changing_ a package already included in
Crush. What about adding a package (from lenny) to Crush?
> Thanks - some very clear summaries. Hopefully, the problems outlined
> won't put you off actually getting some builds done using Debian 5.0
> "lenny".
>
As I said I've been doing mostly kernel stuff and using emdebian as a
very convenient way of getting a working userspace. The time is
approaching however when I will have to choose how to do the real
userspace. I already know that the current set of crush packages is not
sufficient for my needs (but I haven't evaluated the exact package set
yet). I prefer the "emdebian vision" I outlined to the alternatives
and don't mind if it entails a bit of extra work provided:
a) I can realistically do that work in a reasonable time scale [for that
I'll need to determine the exact packages I need and look at them]
b) It's not wasted work that will have to be redone over and over again
[the reason for my post]
c) There aren't any show stopper technical problems [I need to look more
closely at the space required for the package cache]
> Please be careful and consider the points above to minimise risks of
> breaking a stable release.
>
>
Ok.
Cheers,
Martin
Reply to: