[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)



 ❦ 20 février 2017 11:03 GMT, Holger Levsen <holger@layer-acht.org> :

>> Time is a limited resource and we need to set our priorities. Having
>> test suites that work 100% of the time with constrained resources is not
>> a goal I find worthy of the time I can spend on Debian.
>
> While I agree with Niels that it would be very worthwhile to be able to define
> ressource requirements for a package to build (and thus know I have to life
> with some packages having trouble sometimes) I find it *very* strange to be
> content with test suites which randomly fail.
>
> How do you know an error in a testsuite is a non-critical one which can be
> ignored? *Especialy* if you have flaky tests, how can you be sure (or even
> guesstimate) a test failure is harmless one to ignore and not a critical one
> which needs acting upon???

As a distribution, our primary goal is to deliver software from upstream
to users. It is great if we deliver better software, but we don't have
to fix all bugs. Test suite runs fine in my pbuilder environment. Test
suite runs fine on Travis-CI (or whatever CI system upstream is
using).

Moreover, it's not about ignoring bugs without looking at them, it is
about making them non-RC or discard them if they are bad luck (happened
once) or specific to the build environment.

> I *really* don't get why people advocate keeping unreliable tests enabled in
> releases.

As a rule of thumb, upstream usually knows better than me which tests
are important. Tests are quite important for the packager to know if
they didn't make an obvious mistake when updating a package (e.g new
dependency missing, something easy to miss by testing manually). Test
are quite important in a team as you may have to update the package
while knowing little about it (e.g adding a security patch).
-- 
Document your data layouts.
            - The Elements of Programming Style (Kernighan & Plauger)

Attachment: signature.asc
Description: PGP signature


Reply to: