[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: gcc-3.4 emits large amounts of test failures

On Mon, Jan 03, 2005 at 05:12:57PM -0500, Daniel Jacobowitz wrote:
> On Tue, Jan 04, 2005 at 08:47:45AM +1100, Matthew Palmer wrote:
> > On Mon, Jan 03, 2005 at 02:07:59PM +0000, James Troup wrote:
> > > Matthew Palmer <mpalmer@debian.org> writes:
> > > >> would pretty much ensure that the package never, ever builds. And
> > > >
> > > > Well, if it's always broken, we don't really want it, do we?
> > > 
> > > If 'failing tests == broken' then we wouldn't have a working compiler
> > > for any architecture and/or for any release.  I think there's a small
> > > flaw in your logic.
> > 
> > So what are the tests useful for, then?  They're obviously useless as a
> > gauge of quality, because failing tests apparently don't indicate a flaw in
> > the software.
> A little common sense, please?  The test results have to be interpreted
> by a human being.  There are about twenty thousand tests and most
> architectures fail maybe a few dozen.

Common sense would suggest that tests that have to be analysed by a human
being after every test run aren't particularly useful.

* The person needs to be sufficiently clued to work out which tests are
actually important and which ones are fluff.  I couldn't make that call, if
buildd admins aren't compiler experts I wouldn't expect them to be able to
make that call, and I wouldn't expect them to need to.

* If your usually clued test-analyst is having a bad day, they might dismiss
a critical failed test as being harmless, resulting in problems passing
through QA that *were* picked up by the test suite but were ignored.

* Are all of the possible exceptions to tests passing well documented?  If
not, then if your clued test-analyst takes a bus between the eyes, you're
down to guessing at which test failures are real.

* If your test failures are documented, then you can turn that list into a
machine-readable list of tests to not run on particular architectures.  This
way, if new tests fail, they stand out like dogs' bollocks, and your
programmers will know that something's gone wrong.  It may eventually come
to pass that the problem is put on the "ignore" list, but until then it's a
big red flag which is waving in the air and saying "there's a problem here,
look at me".  If you've got one real red flag in a field of ignorable ones,
it gets real hard to pick out of the crowd.

* Tests are supposed to be a "here be monsters" type of thing.  If your
programmers get into the habit of ignoring test failures because "they're
harmless", they *will* ignore real problems as well.

- Matt

Attachment: signature.asc
Description: Digital signature

Reply to: