Re: Ideas for additional large scale tests
On 27/02/07 at 10:53 +0100, Loïc Minier wrote:
> Hi there,
>
> (We had a small exchange via private email with Lucas and he suggested
> we continue the discussion on debian-qa@, hence full text included
> below.)
>
> On Mon, Feb 26, 2007, Lucas Nussbaum wrote:
> > On 24/02/07 at 16:34 +0100, Loïc Minier wrote:
> > > Hi,
> > >
> > > I've been rebuilding the GNOME stack with a couple of interesting QA
> > > tests:
> > > - -Os in CFLAGS
> > > - -Wl,-O1 in LDFLAGS
> > > - -Wl,-z,defs in LDFLAGS
> > > - -j2 in MAKEFLAGS
> > > - use differents directories for builddir and srcdir
> > >
> > > Not all these tests are easily achieved with Debian packages. It was
> > > easier for me because I didn't dpkg-buildpackage these, I used a
> > > GNOME tool named jhbuild instead.
> > >
> > > These tests would be however interesting for Debian packages:
> > > - -Os is useful for embedded platforms
> > > - -Wl,-O1 gives better performances and might be promoted for all
> > > packages at some point
> > > - -Wl,-z,defs checks for missing link flags; this is a very useful test
> > > but it breaks with Python extensions
> > > - -j2 is interesting to catch errors in Makefiles and might be used in
> > > the future to speed up builds
> > >
> > > One way to easily test these flags might be to use wrappers for cc,
> > > gcc, c++, g++..., adding the flags.
> >
> > The main problem with testing -Os, -Wl,-O1 and -Wl,-z,defs is that they
> > could cause the compiler/linker to generate incorrect code instead of
> > just failing, and this is much harder to detect.
>
> I think -Os and -Wl,-O1 are more about testing our toolchain, but I
> think it's interesting to check whether this works on a large scale or
> in e.g. the GNOME stack because this has some direct benefit for people
> interested in the use of these flags and because it will improve our
> toolchain. I agree it might also produce broken binaries and that it
> would be hard to test, but it's a good first step into ensuring we
> could provide a "thin client optimized" archive for example.
>
> -Wl,-z,defs shouldn't cause any incorrect binaries; it should simply
> prevent the build in interesting cases, that is when some link flags
> are missing. The most problematic packages with -z defs are Python
> bindings which explicitely avoid linking to -lpython since the python
> interpreter is not linked to libpython, but provides the same symbols.
>
> > make -j2 looks really interesting.
>
> Yes; I expect it will expose some rare Makefile races (rare is in the
> context of GNOME since all modules are mostly automakized; other
> upstreams might have broken Makefiles :-) but it will certainly help us
> work toward building with parallel=$i, and I'm sure you would benefit
> of that!
All those tests are interesting, and I'll try to dedicate some time to
work on that during the lenny release cycle.
> > Are you interested in working with me on that ?
>
> Well, I've got the tools to do the work over GNOME (jhbuild), and I'm
> busy enough that I'd prefer avoiding putting my finger in more general
> QA stuff. O:-) These were mostly suggestions as a followup to your
> request during your talk at FOSDEM of what tests you could run; it
> seemed you were requesting more ideas to keep more nodes busy. :)
Well, what I'm looking for is:
[A] People with good ideas of tests that would improve Debian's
quality, or Free Software's quality in general
[B] People willing to help with providing code for the tests, and
providing manpower to analyze the results, file the bugs, etc. Just
running the tests and trashing the results is useless.
People matching both criteria are of course much more interesting for
me, since it means less work for me :-)
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.net http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr GPG: 1024D/023B3F4F |
Reply to: