[-project and -policy, I CCed you because I'm raising issues relevant to
you; *please* honor the Mail-Followup-To: header!]
On Tue, Nov 12, 2002 at 02:28:34AM +1000, Anthony Towns wrote:
> > We therefore recommend that
> >
> > * The bug system administrators and/or the project leader or some
> > other delegates appointed by the project leader decide on the
> > proper use of the bug system features.
Would this include guidance on features like the opening vs. closing of
reports, which territory currently claimed by Ian Jackson's disputes
resolution document?
> Obviously, I, personally, do, and if you want to punt to the release
> manager, you can make use of that. It's not clear that's particularly
> appropriate, since the question is surely as much "was this the right
> thing to do" as "does Anthony have any right to do it".
My feeling as package maintainer is that, if you as Release Manager
don't regard it is a show-stopper for release, then I should be
permitted to prioritize the bug as I see fit.
> (1) All bugs have an objectively defined severity, which can be
> determined by reference to the bug submission guidelines,
> policy, and similar resources. The principle is that bugs
> that have a similar effect should have a similar severity.
>
> Unfortunately, this "objective reality" is somewhat poorly
> documented, and can be readily misinterpreted; eg the policy
> for woody was that as long as the package had ever been
> built on an architecture, "doesn't autobuild" bugs were
> not RC, whereas this is not the case for sarge; without any
> documentation changing. The only particularly effective way of
> finding this out is to ask on debian-devel@lists.debian.org.
This is a pretty low level of objectivity, in practice. So much so that
I think it's overstating the case to assert that "all bugs have an
objectively defined severity".
Furthermore, even excluding your second paragraph above, the definitions
of the bug severities do not apply consistent standards for evaluation.
"critical", "grave", and the first half of "serious" are fairly
objective standards. Not pefectly, though: consider the fact that
folks routinely file grave or even critical bugs aginst X server
packages because the X server doesn't support their video hardware.
For them, "the package in question is unusable".
I worked with Chris Lawrence to sneak slightly improved definitions of
the bug severities into the reportbug package, and this has in fact
helped with severity inflation for X server bugs.
At any rate, we start to slip into subjectivity with the other
definitions.
serious
...or, in the package maintainer's opinion, makes the package
unsuitable for release.
important
a bug which has a major effect on the usability of a package...
What's a "major effect"?
minor
a problem which doesn't affect the package's usefulness...
Some people feel that defects in manpages affect the package's
usefulness (if, for instance, the manpage really sucks). Other people
may feel that no error in a manpage can possibly result in a bug of
severity other than "minor" or "wishlist".
wishlist
for any feature request, and also for any bugs that are very difficult
to fix due to major design considerations.
One man's grave bug is another man's feature request ("I want
accelerated 3D support on my NVidia card!").
Some people have trumpted loud and long about the "objectivity" of
the bug severities in the BTS. I think, with all due respect, that
they're visualizing idealized Debian BTS, not the one we've actually
bothered to document.
Until and unless we have an algorithmic process for determining bug
severity that is largely[1] free of human judgement, we will not have
objectively defined bug severities.
Perhaps we can achieve this. Perhaps we could have a tool like
reportbug ask the user a series of yes/no questions, and set of answers
would map to a severity. To be a reliable mapping, of course, we're
going to have to do away with broad language like "usefulness" and
"major effect".
I propose something different, however. I suggest we split the
difference.
How about having only *some* of the bug severities be deliberately
objective? How about leaving the others to the maintainer's discretion?
I propose something like this:
critical
grave
-------------- objectivity threshold
serious
-------------- "RCness" threshold
important
normal
minor
wishlist
Furthermore, I suggest decoupling policy violations from bug severities.
Make a "policy-violation" tag that can be applied to bugs.
The "serious" definition would thus be rewritten to simply:
serious
A bug which, in the package maintainer's opinion, makes the package
unsuitable for usage in a production environment.
Consider the benefits to be gained by the above proposal:
* We have objectivity where we most need it. We have little time for
arguments about the severity of bugs that almost everyone can agree
are Bad News. Data loss? Security holes? Kernel panics? Resolving
these sorts of problems should be our highest priority.
* We don't try to impose objectivity where we may not really need it.
Why have severity wars over "minor" versus "wishlist"? (Or even, I'd
suggest, "important" versus "normal".) By holding to a principle of
a completely objective set of bug severities, we will front-load a ton
of arguments that we don't really have time for, and that I submit we
don't really need to worry about. I don't really have a problem,
personally, with the existing definitions of "important", "normal",
"minor", and "wishlist". That the declaration is made by some that
the BTS *already* has an objective set of bug severities suggests to
me that other folks are happy with the current definitions as well.
So my proposal is limited in its intrusiveness.
* The package maintainer has a way of flagging bugs for special
attention by the Release Manager, "serious". Once the package
maintainer is satisfied with a package, if there are no grave or
critical bugs, there is a simple mechanism for him to tell the Release
Manager (and the katie suite), that it satisfies his standards of
quality.
* Manoj has, if I recall correctively, derisively complained about usage
of the Debian BTS as a personal to-do list for package maintainers.
But that's what, in part, I think the BTS *should* be. Let the
package maintainer use it to triage problems, focus his energies, and
drop hints to other developers about what he most needs help with. It
is not *necessarily* true that having the Debian BTS function as a
to-do list for package maintainers means that it *cannot* serve other
functions for the broader project as well. Perhaps Manoj fears a
scenario where a package maintainer swats a security hole bug down to
"wishlist" because the maintainer would enjoy working on other things
in the package more. That is not legitimate use of the BTS at present
and my proposal would not render it legitimate.
[The next point is a long one.]
* The current imperfection of the Debian Policy Manual causes us to lend
more gravity to certain bugs, as a result of "a severe violation of
Debian policy (that is, it violates a "must" or "required"
directive)," than they deserve. I know that Manoj has big plans for
the Policy Manual. I also know that some of his plans for the Policy
Manual are not uncontroversial, and this fact plus the constraints on
his time and the lack of other policy editors means that a
re-realization of the Policy Manual as Manoj would like to see it is a
ways down the road. I suggest that until that day comes, or at least
until it's visible on the horizon, we not have the "serious" severity
drafted into service as a maul wielded to enforce Policy. I share
Manoj's concern that developers will scoff at Policy if there aren't
any "consequences" for ignoring it; that's why I suggest a
"policy-violation" tag (which, if we want, can be the sole domain of a
delegated team of Policy Enforcers). Mechanisms independent of the
Debian BTS are thus easily employed to gauge a package's compliance
with Policy, or at least how successful it's been at attracting the
scornful attention of bugs with policy-violation tags. The Release
Manager can cooperate with the Policy Enforcers, or use his own
mechanism, to decide whether or not to exclude a package from the
release based upon the nature of quality of policy-violation tags
against it. It is uncontroversial that the Release Manager has this
discretion, and I don't propose to remove it.
As far as I can recall, I have yet to encounter anyone who can tell me
with a straight face that #97671 is a big deal. It's unappealing and
unesthetic, but its practical consequences are extremely small,
especially given that the functionality in question is virtually
unused. If we have a BTS that leads me, or other people, to spend
time on this bug rather than, say, finding racy symlink-attack
vulnerabilities in xdm, I think that's a misfeature of the BTS. I
*know* policy says I'm a bad person for letting #97671 exist. I plead
guilty to that. However, Policy also mandates a lot of things more
strongly than it should, perhaps. A long time ago, Anthony Towns
spent a lot of time trying to persuade me not to go nuts with "must"s
versus "shoulds" in a policy proposal regarding fonts of the X Windows
System. I was intransigent on the point, and after some other changes
to the proposed policy he withdrew his objection (maybe even seconded
it, I don't recall). The result today, however, is that some font
packages for X might be subjected to severity inflation which is
*objectively* accurate. This makes no sense. A bug should only be of
serious severity if it's a serious bug. Clobbering an existing
fonts.dir file in your postinst is Serious. Failing to put your fonts
in the correct directory isn't (it doesn't break anything, it just
might make your package worthless or tricky to use.) The proper usage
of MUST versus SHOULD in Policy is another long-running flamewar that
hasn't been resolved yet, as far as I know.
Please, let's isolate the damage from Policy flamewars. I don't know
what the Policy Manual will look like in a year or two years, but I do
know that its current imperfections, or the threat of it falling into
an unmaintained state if something bad should happen to Manoj, should
not have disruptive effects on the Debian BTS. With a
"policy-violation" tag, policy violation bugs will be taken as
seriously as the people responsible for managing them are assiduous.
With a disciplined team of policy enforcers working hand-in-hand with
the release manager, egregious policy violations will not be permitted
to creep into a production release. If we have no policy enforcers
and the Policy manual lies fallow, the underutilized tag can do no
harm, and can be categorically ignored by the RM because he knows no
one is keeping up with them. Furthermore, with this tag under the
control of a team, deficiencies in the Policy Manual can be corrected
on a discretionary basis; see my X font example above. If they know
that a "MUST" is in the Policy Manual that shouldn't be, they can act
accordingly.
[Whew.]
> (2) By default, the release manager will consider all bugs that
> have a severity of serious, grave or critical to be
> show-stoppers: either on a per-package basis, or for the
> entire release. This default can and often will be overriden
> by the subjective judgement of the release manager: either
> to say that some less severe bugs will also be considered
> show-stoppers, or that some serious, grave or critical bugs
> will not be considered show-stoppers.
Yup, that's the status quo, and it sounds fine to me.
> (3) As such, the issue in question should remain filed as
> "serious" (as it is a clear violation of the FHS), and
> should not have been considered a show-stopper bug for
> woody's release (based on the release manager's judgement,
> with the support of the package maintainer).
I disagree; see above. :)
I think that because:
* the release manager feels it should not be considered a show-stopper
bug; and
* the package maintainer feels it should not be considered a
show-stopper bug; and
* no one has yet (IIRC) been willing to argue that the bug is more
deserving of attention than several other important (or even normal)
bugs currently filed against the same source package; and
any criterion which compels the bug to be regarded as "serious" in spite
of all of the above is flawed.
Therefore, we should change our criteria. We should not mislead people
into thinking a non-serious (in its standard English usage) bug is
serious.
> Branden suggested:
> ] The Release Manager can strip the "release-critical" tag off of any bug
> ] he wants. This is how things have *always* worked in reality. If a bug
> ] is truly a showstopper for a release, we don't release with it. We
> ] either wait, fix the bug, or drop the package.
[...]
> This doesn't work -- we've already tried it with the "important" severity.
[...]
> What we do now is, effectively, add an additional tag to release-critical
> bugs that aren't going to be treated as such: for potato it was an
> [IGNORE] in the release critical bug list, for woody it should've been
> the same except some scripts were broken, and for sarge it will probably
> a real "ignore" or "allow-release" tag. The benefit of doing it this
> way rather than dropping the severity is that it makes it easy for the
> release manager to keep track of the bugs he needs to be responsible
> for; even the ones he's decided to allow remain unfixed.
Well, why not encapsulate this information into the BTS via tags?
The point I keep hammering on is that "releasability" is in actual fact
orthogonal to severity. There is a strong correlation between high bug
severity and unsuitability for release, but it is no *more* than a
correlation. This is a problem because the Release Manager -- at least
historically -- has not made release decisions based solely on
aggregated data. He hasn't decided ("well, once we get down below 20 RC
bugs, we'll be good enough for release") without letting the nature of
those 20 bugs influence him in anyway.
The Release Manager, historically, has *always* cared about the specific
nature of each release-critical bug, and acted accordingly. I think
this is sound policy, and I encourage you to stick to it (I haven't
gotten the impression you intend to do otherwise).
A high severity is good for getting your (the RM's) attention. You
often want to keep track of a bug so filed, sometimes regardless of
whether the severity was inflated in the first place -- even according
to objective criteria -- or not. The severity may go up or down, but
that's not in and of itself determinative of whether you care about it
or not. Sounds like a good candidate for a tag, to me. You care about
the bug, you put (or keep) the tag on. You stop caring about the bug,
you tear the tag off. Your tag, your call. The package maintainer will
know that your eyes are on that bug.
[Consider our recent discussion regarding 4.2.1-3, #97671, and
propagation testing. Among other things I didn't know, I didn't know if
you were still paying attention to that bug or not. With this tag, I'd
have known. Unless you stopped caring and forgot to take it off. :)
But that's more easily rectified (I mail you about it and you reply "I
don't care about this anymore" and CC control@bugs to strip the tag)
than the converse, where you care about some bugs but the package
maintainer doesn't know you do.]
> ( Hrm, an "allow-release" tag might be a better way of doing things than )
> ( checking there aren't more bugs in unstable than there are in testing. )
I'm not wedded to the name or semantics of a "release-critical" tag.
I think the right thing to do is to equip the BTS with a device that
lets the Release Manager do his job efficiently. I think a tag is it.
What it's named and what it means should be left mostly up to the RM.
(I say "mostly" because I think I might object to a tag called "fucked"
being applied to one of my packages. :) )
> I don't believe there's any real value in changing the BTS to treat
> "ignored" bugs specially in a way that would be useful for Branden's
> triage;
By the same token, I don't think there's any real value in having the
BTS treat policy violations specially for triage purposes. At least not
with Policy in its current form. I may very well feel differently if
Manoj succeeds in thrusting his trident into the sea, threading the
conch shell, and bringing down a new Policy from Mount Sinai that's so
elegant it makes me weep in admiration. :)
> although generalising it to allow the maintainer to arbitrarily
> re-prioritise bugs would probably be a win -- the idea being you could use
> a different URL and get the bugs in the order in which the maintainer will
> be focussing on them. (Which is to say, I won't be writing or applying
> any patches for the former, although someone else on owner@bugs might, but
> I'd probably be interested in seeing patches or good ideas for the latter)
Bugzilla has two different fields. "Severity" and "Priority". I'm
pretty sure I think that's a good idea, but as you say, you're not
willing to do it. I don't know of anyone else who is, either.
Let us not let the vision of an ideal future BTS hamstring its operation
for us now. We should not aggravate our suffering of miserable severity
flamewars as some sort of masochistic punishment for not implementing
certain features in the BTS.
Let the BTS be designed to work well for us today, not against a future
where we have a universally-agreed upon Policy Manual in which "must" is
never used where "should" will suffice. When that day comes, it's easy
to tighten up the definition of "serious" if we want to.
> Assuming the "pending" tag on Bug#143825 is for triage (since it hasn't
> been closed in any of the uploads since the tag was applied) it's probably
> better to add [lowpri] to the bug's subject, then use
>
> http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=xutils&exclude=subj:[lowpri]
>
> to get the sanitized page. An inaccurate "pending" tag probably makes it
> less likely for people to provide patches, etc, which would presumably
> be a loss. Apologies if the assumption's incorrect, of course.
It's not inaccurate; it's there to tell people curious about the bug
that I agree that it's a bug, it's been triaged, I'm aware of it, and
that I intend to fix it. It's just not a high-priority item. I want to
fix it right, which means Imake tweakage. Imake doesn't really
know about FHS[2], which is suboptimal. If it did, this problem, along
with others I dare not point out for fear of more and redundant serious
bugs getting filed, will go away. Such a fix might also make it easier
for Debian to transition to shipping XFree86 in /usr instead of
/usr/X11R6. Ooh, now *there's* a way to get Manoj to support lowering
the severity of this bug. ;-)
It won't be hard to fix Imake in this way. Just tedious. I intend to
get to it before the next freeze.
[1] I qualify with "largely" because there will always be people who
split hairs over the meanings of simple words. In arguing against the
concept of objective severities, I won't be so cruel to my opposition as
to hold them to a standard of *perfect* objectivity. I just think that
it's very hard to attain even a moderate level of objectivity, and that
at present we haven't reached it.
[2] It knows about bits of it. My plan is to have a "UseFHS" expando
and set a bunch of defines, probably in X11.tmpl, if it evaluates true.
Then of course one has to change linux.cf and any other .cf for which
the FHS might be relevant.
--
G. Branden Robinson | If you have the slightest bit of
Debian GNU/Linux | intellectual integrity you cannot
branden@debian.org | support the government.
http://people.debian.org/~branden/ | -- anonymous
Attachment:
pgprhqGi_d2_y.pgp
Description: PGP signature