* Sébastien Villemot <sebastien@debian.org> [2017-08-29 10:28]:
On Sun, Aug 27, 2017 at 05:04:34PM +0200, Rafael Laboissière wrote:Yes, I confirm that it does not catch test failures. In order to do that, we must change the check-pkg script, such that it would be able to collect the failures and optionally exit with an error code if any of the tests fails.Ok, sounds good. Note that we will need to whitelist some test failures in some packages, which can probably be achieved by patching the .m file and use the expected failure functionality of Octave.
Yes, I think that patching the file (and eventually forwarding fixes upstream) is the right way to go. The test function of Octave is quite powerful. I looked at it and I think that it should be possible to collect the results of tests and allow check-pkg to make use of the information. I have though to have a sound design for the implementation.
I am also wondering if we should not make test failures fatal at build time. This would probably cause a few FTBFS in the short term, but in the longer run this would improve the quality of the packages.
Yes, absolutely. If a unit test fails, then either the functionality or the test are wrong and must be fixed.
Rafael