On 08/11/06 21:06:28, Jim Heck - Sun Microsystems wrote:
I will admit up front that I amnot an expert in Debian policy/procedures, so some of what I discuss may miss the mark completely for what is or isn't important to Emdebian, but I'd like to see what the group thinks of my points below. These are really from the viewpoint of a developer working on an embedded system.
Some of your terminology is also confusing - from a Debian/Emdebian perspective.
1. emdebian source is identical to Debian source. Emdebian just modifies the patch set (.diff.gz) that Debian uses to create a Debian package out of the upstream source. We strip out bits we don't want and rebuild - that's it.
2. I'm confused by "local source control" - is this a private VCS (CVS/SVN/arch etc.) (if so, why?) or is this some form of control in terms of restrictions of freedom, isolation of the source from the main Debian archives and redistribution with non-free software?
3. A fixed distribution with restricted updates sounds similar to Debian's definition of the 'stable' distribution. There will be no direct security team support for Emdebian - at least until emdebian becomes an accepted port within Debian itself. emdebian is likely to continue tracking Debian unstable or possibly testing, rather than stable, until such time.
Specifically, I think embedded Linux systems are unlike classic Linux systems in a few important ways. They are used as the foundation for many embedded products, where one of the prerequisites in the development process is the ability to tightly monitor and control changes to the OS,
I would disagree, personally, because I want emdebian to work the Debian way: easy and transparent updates to always have the most recent packages available - across the board. If users choose not to upgrade, fine, but when they do upgrade, the entire distribution is available for updates and is upgraded in a single, transparent and largely non-interactive manner.
I run GPE/OE on my iPAQ and the idea that the packages are *SO* far behind Debian really sticks in my throat. I built some updated packages and the installed ones are so old I had to rebuild almost the entire OS.
This only happens because GPE/OE do not use the Debian package handling tools and have to rebuild everything all over again, including implementing new dependency chains to fill the gaps.
emdebian does it differently - emdebian, like debian, is a constantly rolling update. Updates are always available, updates are implemented across the entire package list according to the results of reproducible builds within Debian.
usually involving some form of source control forboth the sources and the toolchain that are used to build the specific embedded product. This is necessary, so that a product based on the OS will have high stability and have been rigorously tested before being deployed.
Until emdebian has a release team that can manage such a process, I'm not sure that is practical. That release team, IMHO, is likely to be the main Debian release team when/if emdebian is accepted into the fold. Duplicating that enormous effort is, IMHO, a complete waste of time.
Often following initial development, changes to these systems follow a very conservative approach where it is desirable to be selective in pulling in updates and changing fundamental system tools such as toolchains. As such, there is a great desire to be flexible and current in source and tools during initial development (like a classic Linux model leveraging package management tools), but once an embedded product is "baked", there is real value add to gaining finer control over how one brings new source and new tools (toochains) into the mix.
This sounds much more like the attitude of Fedora and RHEL. Personally, I moved to Debian to get away from "baked" distributions because once baked, they are incredibly difficult to update and upgrade.
Baking interrupts the flow of updates that Debian provides. I'd compare it to: 1. striving to push a boat out of a fast river and towards the shore. 2. Lifting the boat out of the water and putting it on a trolley. 3. Leaving it on the trolley for a while.4. At some future date, moving the trolley downstream and trying to relaunch the boat AND then catch up with where the boat would have been if you hadn't taken it out.
That kind of hiatus breaks things - badly.Not once, prior to moving to Debian, was I able to successfully upgrade a GNU/Linux installation. apt-get dist-upgrade solves *all* those problems and apt-get upgrade keeps things in sync.
1. Be able to easily put a set of Emdebian source packages specific to a given release of their embedded system under local source control.
Emdebian is primarily concerned with binaries of reduced size - managed by a set of tools and patches that strip out unwanted data, documentation and components. The source remains upstream and emdebian will keep up to date with the upstream changes.
2. Be able to leverage package management tools to intelligently bring in a minimal set of sources and libraries to move forward to get new features or bugfixes (this is in the stage following the fluid initial development stage, when the embedded system is in it's conservative update mode).
apt-get update && apt-get upgrade are not restricted - without a release team to limit the flow of packages from testing into stable, all available updates are always installed, each time apt-get upgrade is run.
3. Have those pieces of the host environment that pertain to managing and building a set of packages be self contained and well known enough to be themselves placed under local source control.
That phrase again. In your opinion, is 'apt' under 'local source control' and what precisely do you mean by local source control?
4. Be able to construct a toolchain that is largely independent of the package sources, since changing toolchains following initial development is far more risky than bringing in new sources.
Debian manages that transition and emdebian will benefit from the updated compiler and toolchain.
New Debian packages will necessitate the use of the current Debian toolchain - the new packages won't build otherwise. That debian toolchain is upgraded as time moves by. It's just the way Debian (and emdebian) work.
If you want the latest package, you cannot expect it to compile under an old toolchain - it might, it might not, Debian won't care either way. The Debian build instructions were built, tested on the other ports then reproducibly retested again and again using the *new* toolchain. Debian packages are always required to build on whatever is the current Debian 'Default Compiler'. Currently, that is gcc-4.1. 4.2 is in the wings and once uploaded to Debian unstable, all packages in Debian unstable will be required to build reproducibly using 4.2 on *all* supported architectures. emdebian seeks to become on of those supported ports. To do so, emdebian *must* keep in step with the parent: Debian.
The buildd logs for any source package are plain to see in Debian - as is the reproducible nature of each build, over and over again.
5. Ideally have source packages that are easy to "lift out" of a Debian environment wholly self contained so they can be compiledelsewhere.
emdebian is embedded Debian - I'm not sure it's worth the effort trying to support those who want emdebian without Debian. Personally, I'd recommend OpenEmbedded for such requirements.
Often embedded development of complex systems end up using centralized building environments where many pieces beyond the base OS are brought together. These centralized build environments arise so that builds can be automated and serve many developers in a homogeneous manner (one centralized toolset).
The current Debian default compiler and the toolchain that is built around it *is* the centralised toolset of both Debian and Emdebian. It doesn't change that often but when it does, emdebian must keep up - otherwise we will not be able to build the latest packages of the rest of the system.
Many times these systems may not be a Debian environment (bureacracy...). The ability to easily integrate the building of source packages into such an environment would be nice.
I'm not sure it is something emdebian will support. emdebian *wants* ever closer ties to Debian - that is why we're here. We want embedded Debian GNU/Linux, not embedded GNU/Linux independent of Debian. That is what OE and GPE already do.
There may be an easy way to achieve point 1 already. I am looking at how I might setup my own source repository (under source control) where I can store Emdebian packages that I pull from the main Debian mirrors. Perhaps the tools being discussed can help with this (or at least not make it harder).
Private repositories are the easy part. Managing a restricted set of updates so that the whole continues to build is tricky. Debian does this job for us - in the most part - why dump that method?
Point 2 is one that I personally find to be of most interest. I really want to be able to leverage the Debian packaging tools and apt's dependency management to upgrade pieces of my target root filesystem,
emdebian will do that. emdebian also seeks to modify the Debian apt type tools to foster easier cross-building - as a necessary precursor to even closer ties to emdebian's parent group: Debian.
but do so by bringing in the necessary sources forlibraries and dependent packages in a minimally inclusive source set that can then be built and the sources put under source control.
This, at least to me, seems like an enormous amount of wasted effort.
I wonder how the tools might help me in this regard.
It may help to consider what is widely regarded as "Debian's biggest problem" - the delay between releases. i.e the work involved in "baking" the Debian packages for a fixed release set. Debian works on a rolling update set - fixing that set at a specific point in time is difficult. Will emdebian ever have the developer time to do the same?
What confusesme is that there are really two kinds of dependencies. Build dependencies and installation dependencies, and I'm not sure how the two can be used to the effect I am describing. Ultimately both need to be managed. I haven't heard anyone propose the cross equivalent functionality of 'apt-get build-dep <package>'.
The likelihood is that by keeping up with the rolling Debian update set, emdebian will be able to keep up with the Debian build-time dependencies and therefore the runtime dependencies. There will be differences in the 'essential' package list in emdebian compared to Debian but other than that, package names will remain the same in emdebian and Debian.
Runtime dependencies are what matter to users but they mainly arise from build-time dependencies. Build-time dependencies matter to the maintainers and are usually only installed by those wanting to build packages.
emdebian hopes to simplify the dependency situation within debian itself by a process of fully qualifying all dependencies and stripping out unnecessary dependencies - as well as making default components into separate, optional, packages with their own dependencies. This includes build-time and runtime dependencies.
Point 3 is a tricky one, but central and also one I find to be of interest. In a "normal" linux system, the system is its own source control if you are building Debian packages for local installtion. For cross development, this is inherently not the case, as we all realize, and an overriding concern is being able to reproduce build results in the future.
How is that different to Debian? Debian buildd's and scripts like pbuilder are used for exactly this purpose - to ensure that Debian builds are reproducible at all times and on all supported architectures. That is what is holding back Etch - each time there is a failure to match a build to the results from a comparable (successful) build, a FTBFS bug is generated. The package Fails To Build From Source - a release critical bug that delays the movement of that package into the next "baked" release. In doing so, it may also block the move of any package that depends on the one failing to build in a reproducible manner.
emdebian follows this principle - a package won't move into emdebian until it builds cleanly in Debian. i.e. emdebian relies on reproducible builds even during the constant flow of updates.
To this end, the pieces integral toreproducibly building target packages need to be as seperable as possible from the host Debian installation (i.e. no weird or unknown dependencies on the host Debian environment).
Emdebian has a host of intricate and complex dependencies on Debian - emdebian is debian stripped down. The dependencies will not be unknown (this is free software, everything is public) - I can't say if these dependencies could be construed as 'weird'. It depends who is making that judgement.
emdebian is not about being removed from Debian - quite the opposite, emdebian seeks to become an integral part of the Debian release cycle alongside the existing ports like amd64, arm, powerpc and m68k etc.. emdebian seeks to join it's (eventual) Policy with Debian Policy. To merge the buildd work into the overall Debian buildd network.
Why this is important,is so that if Emdebian is part of a larger build environment that serves the purpose of building the OS, it is at least a reliably reproducible part, and those pieces critical to reproducibility can be properly source controlled and managed.
I think you may be thinking of Emdebian in terms of OpenEmbedded. To me, at least, your approach seems to suit their methods rather than the methods within emdebian.
OpenEmbedded is an OS builder. Emdebian is just embedded Debian - a prebuilt OS. Emdebian isn't about building a new OS, it's about stripping down an *EXISTING* OS into the resource limits of an embedded device.
Point 4 is a fact of life. It is never ever easy to change a toolchain once the product ships.
Debian does this for us. It's happening now with gcc-4.2 in experimental.
New toolchains need to berequalified against all of the software, and thoroughly regression tested.
This is done by debian maintainers building their packages and fixing FTBFS bugs caused by the transition. It's not limited to gcc, there has been an enormous effort to get the python 2.3 -> 2.4 transition completed, before that there were C++ transitions related to gcc, there was the transition from XFree86 to XOrg. Debian has a long history of "requalifiying" the entire unstable distribution against each and every transition in the normal flow of the Debian update stream from experimental to unstable to testing and (eventually) to stable.
Emdebian will benefit from all these fixes and will only take packages that have completed whatever transition is relevant at that time.
In due course, emdebian hopes to implement a few transitions of our own - the separation of translation support into dedicated packages and tools to manage them, the resolution of a host of untidy and unruly dependencies that are unclear, unwanted and over complicated, a solution to the differing requirements for the 'essential' package list the list goes on.
The emchain tool, which is designed to stay right up to date with the compiler makes the most sense from an overall rolling Emdebian point of view, where the easiest to make sure new packages can be built without much worry. From the point of view of an embedded developer working on a shipping product, it would not be an option. Luckily, I don't think this is a problem, since it doesn't sound tied into the package management side of things, and is an easy way to get an instantly usable toolchain. I'm just bringing this up, since in practice, the embedded developer will most likely need a fixed toolchain that gets updated once in a blue moon.
emchain is *only* for toolchain building by emdebian developers and automated build machines. We need to keep the toolchain in step with Debian or we will suffer a host of build failures related to fixes implemented in the upstream Debian packages that are not implemented in the previous version of the toolchain. emchain seeks to ensure that the emdebian binaries are *always* built on the same toolchain as is available in upstream Debian - whatever that is at that time.
emdebian cannot hope to build an embedded Debian without using the toolchain used by Debian. That toolchain is a moving target, so emdebian needs to keep in step. That is what emchain is all about. It isn't perfect but it's a start.
-- Neil Williams ============= http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/
Attachment:
pgpzSU7KA2t07.pgp
Description: PGP signature