Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)
On Mon, Feb 20, 2017 at 06:10:23PM +0100, Vincent Bernat wrote:
> ❦ 20 février 2017 13:44 GMT, Holger Levsen <holger@layer-acht.org> :
>
> >> As a rule of thumb, upstream usually knows better than me which tests
> >> are important. Tests are quite important for the packager to know if
> >> they didn't make an obvious mistake when updating a package (e.g new
> >> dependency missing, something easy to miss by testing manually). Test
> >> are quite important in a team as you may have to update the package
> >> while knowing little about it (e.g adding a security patch).
> >
> > I get why tests are useful. I just think unreliable tests do more harm then
> > good. (Mostly because this leads to ignoring those tests completely. BTDT.)
>
> We are speaking of tests that succeed reliably on the packager's own builder.
We are also speaking of tests which are well known to be unreliable
under certain circumstances but the maintainer refuses to acknowledge
it or does not care at all.
The rygel bug comes to mind (#841098). Again, because "it does not
happen in buildd.debian.org", except that it was not even true in this
case and it actually happened in buildd.debian.org.
Thanks.
Reply to: