[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Test suite quality

On 07/01/2012 17:52, Joachim Breitner wrote:

Dear GHC-Team,

to improve the quality of Haskell on Debian started to package the GHC
testsuite in a package of its own, and will upload it to Debian so that
it runs on all architectures and catches possible architecture-dependent
bugs there.

Using the released testsuite for 7.0.4 with that version of the
compiler, and running it with
	 SKIPWAY="optc profc optllvm" fast=YES
I get this result:

OVERALL SUMMARY for test run started at Sa 7. Jan 17:45:54 CET 2012
     2612 total tests, which gave rise to
    10986 test cases, of which
        0 caused framework failures
     8642 were skipped

     2267 expected passes
       59 expected failures
        0 unexpected passes
       18 unexpected failures

Unexpected failures:
    3586(normal)
    Chan001(normal)
    MVar001(normal)
    QSem001(normal)
    QSemN001(normal)
    SampleVar001(normal)
    T1969(normal)
    T3064(normal)
    T3087(normal)
    T3294(normal)
    T4316(ghci)
    T4801(normal)
    cgrun025(normal)
    driver027(normal)
    driver028(normal)
    ghci014(ghci)
    ghcpkg05(normal)
    regex003(normal)

Is that normal to find unexpected failures of the release testsuite on a
“normal” system?
Unfortunately yes, but we are getting better in this area. Here's where we stand on the 7.4 branch for our major platforms currently: 

 x86_64/Linux:  1 unexpected failure
 x86/Linux:     33 unexpected failures
 x86/Windows:   65 unexpected failures
 x86_64/Mac:    unknown (no nightly build?)
In fact the majority of these failures are just bugs in the tests or the test suite framework, and keeping the testsuite clean takes quite a lot of effort, which is why it's hard to get motivated to do it. But I agree it's important, especially for a release, to have zero failures. 

So I've made a ticket to do this for 7.4.1:

  http://hackage.haskell.org/trac/ghc/ticket/5757


Should I file bug reports about unexpected failures found in the latest
testsuite, once I get around to package and run that?


Sure, please do.


And a minor request, in case someone feels like touching the test
driver: It would be nice if the summary lists all library packages that
are not installed but required by some test, i.e. the libraries that
ought to be added to the Build-Dependency of the test suite to get a
more complete coverage.
Yes, good idea. I recently changed the driver so that it now prints out the number of failures due to missing dependencies, but it doesn't print out which packages are required, that would be a good improvement. 

Cheers,
	Simon




I could not attach the build log, as it is too large, but you can find
it here:
http://people.debian.org/~nomeata/ghc-testsuite_7.0.4-1_amd64.build.gz

Thanks,
Joachim




_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply to: