Re: Etiquette about test regression, bug severities, etc.

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Niels Thykier writes ("Re: Etiquette about test regression, bug severities, etc."):
> Ian Jackson:
> > There are some problems with this, though:
> > 
> >  * The only available bug severity is `serious' which also triggers
> >    testing autoremovals.  [...]
> 
> FTR: If you want to block testing migration, you can simply file the RC
> bug against the unstable version of the package.  When the bug only
> affects unstable, britney counts it as a regression and blocks testing
> migration.  However, the auto-removal only considers bugs that affect
> testing, so the bug is not going to trigger an auto-removal.

Ah, of course.

> If yes, then move the discussion to which package should be changed to
> resolve the situation.  Note that if the reverse dependency is the one
> that need changes to cope, then it often makes sense to file an RC bug
> against *both* packages[1].
...
> For (ii), a regression in the testing will cause a 10 day delay on top
> of the regular urgency under normal circumstances and will eventually be
> a migration blocker.
>   If we need a standardized process for blocking uploads in this
> situation, then it smells like we are doing something wrong (acting too
> slow, or focusing on the wrong aspects, etc.).

I guess what I'm saying is that by delaying testing migrations, and by
planning to block them, we are giving the tests quite a high status.
Nearly as high a status as RC bugs.  I don't think that's
inappropriate.  (And note that the block can be as low as 2 days if
for some reason britney thinks the upload has urgency=high.)

Maybe the right approach is to have a convention that if the
maintainer of the depending package wishes to preserve the status quo
for longer than the current testing migration regression delay[1],
they should file two RC bugs: one against their own (depending)
package, and one against the dependency.  And that the maintainer of
the depending package should usually be OK with that - even if the
only problem is a test failure.

Filing an RC bug against the version in testing, and triggering the
autoremoval clock, is a sufficiently disruptive thing to do one of
one's own packages that this wouldn't be done lightly.  So there would
be some kind of equity.

Maybe I am belabouring all of this rather too much.  But this
autopkgtest testing migration stuff is new and will generate new
behaviours, and we need to establish new social norms.  It's probably
better if we write too many mails about it than too few.

Ian.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.