[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Fwd: Crush 2.0 abandoned -- Review and Feedback from Neil's Post of August 7th

Hello Everyone,

Again, my thanks to Neil once again for his insight and expert advice on the list of issues he put together for Crush 2.0. I also thank the other members of the list that have joined this discussion with their own comments, insights, and recommendations.

So far, two major themes continue to echo in the feedback on this issue:
  1. Level of expertise needed to address the Crush 2.0 project as a whole
  2. Dependencies elements have on other issues either within main Debian or related projects.
  3. Organization of sufficient collective effort/resources to gain traction and momentum to solve the Crush 2.0 challenges both now and in the future as the project moves ahead.
Losing Neil's insight and expertise, even temporarily, is a serious challenge to progress. However, we all know he's not in a position to do anything further on any this for the foreseeable future.


In his post, Neil wisely points out this crisis results primarily because the Crush 2.0 project has relied mainly if not solely on him. The remedy of this central and critical weakness  is greater collective effort, albeit in small steps at first. The challenges he highlights are daunting at first glance, but not insurmountable providing we all agree to have a greater role and stake in the project.

None of us alone has all the expertise or skills  to bring this off. So more of us have to weight in and help to share the responsibility Neil shouldered. Crush 2.0 can get done, and as we commit to getting it done, along the way the work can get easier.

I'm not simply cheer leading here. Crush 2.0 's 'learning curve' issue is not unique it's common to open source projects and its been solved before. Every serious OSS project "evolves" from one driven by a sole developer to a larger common effort. That evolution is driven by shared learning. 

So the solution is we first recognize it, and then do what other OSS projects has done; we dig in, break down the big problems until their small enough to solve, and work together to solve them. And when we get stuck, we look around and do what 'lazy programmers' always do:  borrow, adapt and use whatever we can. Naively, I say the solution to each of these problems is "out there", so lets go find it.


If someone hasn't already done so, perhaps one of us will break out those items on Neil's list of issues that have the greatest dependencies and present let's discuss ways to address them. , as someone famous once said, "no problem is opaque if you have enough people putting their eyes on it".

Package issues, library issues, tool issues ... these are common OSS project challenges and again there's a way of cutting this Gordian Knot since other projects have done so.  If we need advice from outside this project, then let's post the problem on the right mail list and get better eyes on it.


This is where the proverbial rubber meets the road: its sound off and sign up time. It's obvious to us all the organization of this project has to change in several ways. All of us have a stake in making those changes. So anyone who can probably should offer their advice on how to bring this change forward. Recommendations are welcome; commitments are encouraged.

Again, thanks to everyone for listening and contributing so far.


---------- Forwarded message ----------
From: Brendan Simon <Brendan@brendansimon.com>
Date: Wed, Aug 12, 2009 at 7:32 AM
Subject: Re: Crush 2.0 abandoned -- Review and Feedback from Neil's Post of August 7th
To: debian-embedded@lists.debian.org

There seems to be a lot of work, and it's a very steep learning curve.
One has to be very Debian savvy before the extra knowledge and time
required to be an emDebian developer/maintainer.

I there anyway to partition this so that a smaller knowledge/experience
base is required to contribute ??

I notice uClibc is a candidate, but I remember a post about embedded
glibc which was/is a smaller lighter libc based on glibc sources.  Would
this be a better candidate than uClibc ??  Should glibc, embedded-glibc
and uclibc all be "supported" ??

Cheers, Brendan.

Prince Riley wrote:
> I'm going to ask the members of this list who haven't as yet spoken up
> about the items on Neil's post to please do so by Friday.  Once we
> have those idea, suggestions, and recommendations we can put those
> ideas into a draft summary.
> On Wed, Aug 5, 2009 at 3:53 PM, Neil Williams <codehelp@debian.org
> <mailto:codehelp@debian.org>> wrote:
>     On Wed, 5 Aug 2009 14:52:22 +0000
>     Prince Riley <wmarketing3@gmail.com
>     <mailto:wmarketing3@gmail.com>> wrote:
>     > Neil,
>     >
>     > We are all very glad and appreciative of the great work you've done
>     > so far. We all understand and respect your decision and wish you the
>     > best as you try to recover your health.
>     >
>     > My question, and one I'm sure has also occurred to others interested
>     > in this project, is how do we move ahead during your absence.
>     Perhaps
>     > you can offer some advice or recommendations on how members of this
>     > list can help you transition your efforts so we can maintain the
>     > project's momentum and achieve its goals.
>     It isn't about helping me transition anything, someone else needs to
>     take over maintenance and development. This is the "bus" scenario -
>     whereby a project has had only a single active developer and once that
>     developer is lost, it's about going back to first principles and
>     picking up the threads. I can help in small areas but my time is too
>     limited to go through the details, just the overview.
>     Seeing as you asked, maybe this little list will give you some idea of
>     why I had to abandon this particular task - I'm not convinced that the
>     problems can be fixed before Squeeze, even if someone else takes
>     on the
>     role. Several stages rely on other tasks within Debian that may or may
>     not make it into Squeeze. The list is by no means complete either!
>     Even
>     before my illness, I had *serious* doubts about whether Crush 2.0 was
>     possible.
>     0. You'll find plenty of hacks, errors and mistakes that have remained
>     in the current scritps from the earlier development phases. It is
>     worth
>     revisiting things like that and seeing what breaks without them.
>     Ensure
>     you test with more than a few packages - there are various gotchas
>     that
>     only apply to a handful of packages and some of those packages have
>     since had new upstream releases that may well have fixed the issues
>     that led to the hacks in the emdebian-tools scripts. Other
>     improvements
>     elsewhere could also lead to some procedures within the current
>     scripts
>     becoming redundant or new ones needing to be devised.
>     1. Read the EmdebianCodeAudit wiki page and supporting pages, go
>     through and cross-build each package to work out what each audit point
>     means. Results are for armel, using emsource and emdebuild. Replicate
>     results using chroots builds (with empdebuild) wherever possible.
>     Various other parts of the wiki also need to be cleaned up, as
>     does the
>     website itself which still has a fair bit of outdated information.
>     2. Continue discussion on how xcontrol support can actually be
>     implemented. Check how it is used in the CodeAudit and tie that into
>     the developments at DebConf9.
>     3. Learn the emdebian-tools scripts and manpages, read the source code
>     and get a handle on how things currently work - taking over
>     maintenance of a native package means taking over the source code too,
>     in effect the "upstream".
>     4. Struggle with the inherent weaknesses in apt-cross which lie in the
>     horrible apt perl bindings - the most recent version of apt has (once
>     again) broken these bindings and apt-cross needs a patch to continue
>     operating. (Something to do with /etc/apt/preferences.d/). Hope that
>     someone gets apt to understand multiarch so that this whole issue can
>     be removed - you need a C++ expert for that role.
>     5. Get the toolchains back into installable state - I can't upgrade my
>     own armel boxes today but I don't have time to investigate why. Build
>     scripts that ensure that this persistent problem is fixed for the long
>     term and across all supported architectures.
>     6. Create a local mirror and use the scripts in emdebian-qa and
>     emdebian-tools to upload cross-built packages into the repo and start
>     building a small set of packages for armel. Don't proceed to other
>     architectures or make the mirror publicly accessible until the rest of
>     the issues in the Code Audit have been fixed.
>     7. Pick up development of emdebian-tools from existing SVN - there are
>     some changes in SVN that still need testing.
>     8. Start developing ways to test and fix an existing problem in that
>     the ./configure scripts in some packages insist on looking for stuff
>     in /usr/lib/ and then linking against the shared libraries
>     in /usr/arm-linux-gnueabi/lib/ etc. This is a disaster waiting to
>     happen and must be fixed before Crush can progress to more than one
>     architecture. In the same manner, check builds for /usr/include/
>     pollution and drop the apt-get build-deps stage in a clean chroot. If
>     no xcontrol file, allow (nay require) native deps.
>     9. Make notes in CodeAudit of packages that fail to rebuild twice in a
>     row and detail why.
>     10. test and document the proper support for CC_BUILD in autotools.
>     11. Get Build-Depends split into debian/control and normal Debian?
>     Possibly as well as Depends-Install (installed on build) and
>     Depends-Runtime ? Building a *static* chroot for another machine, only
>     needs runtime stuff. Allows native cross-versions of packages, smaller
>     installed images. No way to put a prefix in front of maintainer
>     scripts
>     for config files. Need a mangling frontend in dpkg before calling the
>     script. Install time scripts vs runtime scripts. (A topic briefly
>     discussed with Wookey, hopefully my notes aren't too brief for the
>     idea
>     to be reconstructed. I knew what I meant at the time.)
>     12. Create a new wiki page to detail things like this and whatever
>     else
>     comes up as you go. Link it from the EmdebianTracker page?
>     13. Please don't come to me continually for input, if Crush is to
>     continue, someone needs to take over maintenance and that means filing
>     and fixing the bug reports without looking at me to do it. (You'd
>     alter
>     the Maintainer: field in debian/control to remove my name and enter
>     your own so that the bug messages for the emdebian-tools source
>     package go to you or to this list.)
>     14. Work out how to fix "X-Build-Cross-Libtool: yes" which is probably
>     tied into other libtool issues identified in the CodeAudit to do with
>     refreshing the autoconf and libtool metadata prior to attempting the
>     cross-build.
>     15. Get CMake build systems cross-building properly.
>     16. Get a mini-perl and some kind of python support.
>     17. Solve what ever other issues turn up during these processes.
>     18. Fix all the packages that used to cross-build for Crush 1.0
>     but not
>     longer cross-build in unstable. (The CodeAudit identifies these
>     packages, one is gcc-4.4). Work out and gain consensus on whether such
>     packages should *not* be cross-built from source and just "borrowed"
>     directly from Grip. This is a Policy decision about whether Crush
>     should be 100% buildable from source or whether compromises are
>     acceptable. Mixing Crush and Grip will *only* work if the xcontrol
>     support is fixed such that changed functionality is always encoded
>     within the binary package name like libgconf-noldap-2-4 etc.
>     19. Identify *new* packages that have become dependencies of necessary
>     packages in Crush since the release of 1.0 and then work out how
>     to get
>     each of these to cross-build.
>     20. Test these packages with real systems and see if the rootfs
>     scripts
>     can still operate and what new bugs arise at installation/deployment
>     time.
>     21. Manage the repository to cope with britney (the script that
>     migrates packages from unstable into testing in Debian).
>     22. Manage the repository to cope with testing-proposed-updates once
>     the Squeeze release freeze starts to happen.
>     23. Test and report bugs with the locale support via Emdebian
>     TDebs and
>     langupdate.
>     24. Fix issues like the bulky gconv files in libc6 and test with the
>     tzdata support.
>     25. uClibc support
>     26. Finalise the cache value support in dpkg-cross and add to it if
>     necessary. Consider deploying the cache values into the Debian package
>     such that the files can be created at build time on a
>     per-package-per-architecture basis.
>     27. Keep an eye on multiarch developments and feed into the process to
>     ensure Debian gets a usable cross-build environment based on
>     multiarch.
>     Get dpkg-cross merged into dpkg as soon as multiarch is supportable.
>     28. Whatever else I've forgotten.
>     Is that enough for now?
>     --
>     Neil Williams
>     =============
>     http://www.data-freedom.org/
>     http://www.linux.codehelp.co.uk/
>     http://e-mail.is-not-s.ms/

To UNSUBSCRIBE, email to debian-embedded-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Reply to: