On Thu, May 03, 2018 at 11:52:40AM +0200, Arturo Borrero Gonzalez wrote: > On 3 May 2018 at 11:21, Colin Watson <cjwatson@debian.org> wrote: > > (I echo Simon's thanks for doing this, though!) > Yeah, thanks for this! > I would say, yeah, please wait a couple of stable releases before > going full blocker. > I (and others) may not have the time to polish our autopkgtest tests. > If we end with less autopkgtests tests because of this, then the > result of this push would be the opposite of what we intended :-P I think the status quo is that we have a lot of autopkgtests that are useless as a CI gate. In Ubuntu, we have 245 overrides in place to ignore "regressions" in tests that once passed but no longer do. Many of these are tests that are simply flaky and only pass sometimes. Some are tests that once succeeded but now fail because something else changed in the overall distribution, and wasn't / couldn't be caught by autopkgtest to prevent the regression. Some are tests that are treated as "regressions" now because previously they couldn't be run on a particular architecture, but Ubuntu's infrastructure has improved to where they now can be (and are seen to fail). And there has been no release of Ubuntu in which more than 93% of tests passed, on any architecture. (http://autopkgtest.ubuntu.com/statistics) I have also been submitting a lot of bug reports about autopkgtests that the maintainers have allowed to regress in new versions of the Debian package. https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=ubuntu-devel@lists.ubuntu.com;tag=autopkgtest It's fine if the raw number of tests goes down, if the overall quality of the tests - and therefore the quality of the release - goes up (and the time wasted hunting buggy tests goes down). -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer http://www.debian.org/ slangasek@ubuntu.com vorlon@debian.org
Attachment:
signature.asc
Description: PGP signature