Bug#901804: autopkgtest: consider using exit status 8 ("no tests found") if every test was ignored

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Simon McVittie writes ("Bug#901804: autopkgtest: consider using exit status 8 ("no tests found") if every test was ignored"):
> On the Gitlab MR, Paul Gevers wrote:
> > And maybe a regression from 0 to 2 (without changes to the autopkgtest
> > that runs) should also be a regression for britney (if it is not caused
> > by a flaky test :( ).
> 
> I'm not so sure about this. If debci was previously giving tests
> a particular capability bounding set in their LXC containers, and a
> security fix to lxc removes a capability that turns out to be exploitable,
> then tests that were exercising that capability will have to be skipped
> (ideally they'd already be 'skippable', and exit 77 if they don't have
> it), so they'd appear to "regress". That seems undesirable.

Despite what I wrote in my other mail in the pseudo-restriction
thread, I agree with your analysis.

The *point* of SKIP is that it isn't a regression or a failure.  If a
feature is removed, and its absence causes SKIP, and this ought to be
treated as a regression, then the feature ought to have its own test
(perhaps, somewhere entirely else than the package whose test is now
SKIPped).

Ian.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.