Re: Help requested: Packages which FTBFS randomly

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Christoph Biedl writes ("Re: Help requested: Packages which FTBFS randomly"):
> Ian Jackson wrote...
> > IMO all of these bugs should be RC.  A randomly-reproducible build
> > failure with more than negligible probabilty is likely to show up for
> > some of Debian's users and downstreams and cause them mysterious
> > trouble.  It also causes trouble for stalwarts like Santiago, doing
> > much needed and largely-unloved QA work.
> 
> Tracking down race conditions in the build that occur every now and then
> only isn't joy either.

Well, yes, but if there is some difficult work to be done it should be
done by those who care about the package, not by those who are doing
us all a favour by finding the problems.

> At least some of the packages have the random failures in the test
> suite. If this justifies an RC, the maintainer will rather disable the
> particular test, or even the entire test suite. Something I consider
> the much worse scenario than having to trigger a rebuild.

Most downstream folks who trip over something like this are not going
to think "oh it must be a heisenbug I will just try it again".  They
will wonder what they did wrong.  If the failure probability is
nontrivial they will start flailing, producing and discarding false
theories about the cause.

Far better to disable the problematic test.

> Random build failures do happen, for reasons inside or outside the
> packages.

This is simply not true.  "Random failures" do not "just happen".
They have causes.

With the correct infrastructure (which is not that hard) the causes
can be completely eliminated.  I don't ever experience random build
failures of any of my own packages and if I did I would hunt them down
with a vengeance.[1]

> So downstream (also: security and point release builds) must
> be prepared for a failing build anyway. Therefore I'd rather accept
> random build failures, grudgingly, with the threshold set to somewhere
> between 5% and 10%. Rationale: Build may fail one time, the second one
> should very likely succeed then.

I don't think that security and point release builds should be
prepared to retry failing builds.

IMO any failing build in a security, point release, or QA activity,
should immediately produce an RC bug.

Ian.

[1] Don't get me started on gnupg vs the dgit test suite, but of
that's not an FTBFS.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.