Re: Debian QA Policy Draft
On Fri, 21 Nov 1997, Vincent Renardias wrote:
> I also post it here so people who are new ( < 1 year ) to the project can
> read it.
Thanks. Yes, I'm new. It looks fairly solid. Some comments follow,
mostly based on my 13 years experience working as a programmer in a
company which has a QA policy that resembles the one we are trying to
write here.
> Aim:
> ----
>
> Try to improve the overall quality of the Debian GNU/Linux distribution.
Can we be more definite about this goal? e.g.
Goal:
-----
It is the goal of DQAG (Debian Quality Assurance Group) to improve the
overall quality of the Debian GNU/Linux distribution.
> Facts:
> ------
>
> - Some packages appear to be orphaned or not maintained enough.
> - Some outstanding and easy to fix bugs are still pending after months.
> - There's a need to check continuously the distribution's stability/quality.
> - Need to have a spool of maintainers to take care of orphaned packages.
^ pool?
> * Takeover uploads delays:
> After a patch is posted by the DQAG to a maintainer, how long
> will the DQAG wait before uploading the package if the maintainer does
> not do the upload himself?
>
> Delays :
> - critical security fix: 2 days.
> - security fix: 7 days.
> - bug fix: 30 days.
> (Fix a bug reported in the bug tracking system or RFE)
> - cosmetic fix: 45 days.
> (typo. in man page, core file in source package,...)
> NB: "cosmetic fixes" are uploaded to "unstable" only; (neither "stable" nor "frozen").
> The above delays are reduced by a factor of 2 in the month preceding a freeze.
> During a freeze: Delay for security fixes (those announced on <security@debian.org>): 1 day.
> Other fixes: 1 week.
> [Are these delays too long/too short?]
It depends. Probably varies by package and circumstances (e.g. if nearing
a Debian release deadline, they may need to be shorter)
> DQAG members making bugfix uploads will need not only to respect the delays above
> but will also have to announce their intent to upload on <debian-QA> with a cc: to the
> maintainer at least 2 days (or 1 day
> during a freeze) before to do the upload, saying which Package_Version they will upload,
> and the serial numbers of the bugs fixed by the upload.
> The 'Maintainer' field of those uploaded packages will stay unchanged.
> However if DQAG members make 3 consecutive bugfix uploads with still no action
> from the package maintainer, then the 'Maintainer' field will be set to
> "Orphaned Package <debian-QA@lists.debian.org>", and the package will be considered as
> orphaned (this will be announced on <debian-devel>).
I don't think the formula should be so rigid. Situations may vary (not to
mention bugfix priorities). Maybe loosen the wording, e.g. just say "if
the DQAG agrees that a package is not being properly maintained, then the
'Maintainer' field will be set ... etc."
> If at least 2 testers report the same installation/upgrade problem, this bug will be marked as
> critical and will have to be fixed before the frozen distribution can be released.
Again, I'm uncomfortable with such formulas as I think they gloss over the
complexities of individual situations. But it is true that some mechanism
is needed for spotting installation/upgrade problems and ensuring they are
fixed before the frozen distro is released.
> * 2nd step:
> -----------
>
> step 1 +
>
> * One current problem with Debian, is that there's only one developper assigned by package.
> [ This part will be done with Dale Scheetz too? ]
>
> In a near future, I propose to have a pair of developpers by package:
> - One doing the 'real work' (ie: what maintainers already do by now with their
> packages)
> - The other one checking each package release for bad dependancies, missing man
> pages, ... If a problem is found a bug report must be opened, and whenever possible
> the tester should try to include a patch to fix the reported bug.
> (Since testers aren't supposed to make uploads, they don't necessarily need to be
> "Debian developpers" having an account on master.)
> For {big,important,bug prone} packages (ie: emacs,bootdisks,gcc,libc6,...),
> there may be several testers.
I'm not sure if this is what you meant, but I don't think a tester should
be 'assigned' to a package in the same way that a maintainer is. It is
harder to coordinate such pairings in a volunteer environment, and it can
slow things down. But the principle is a good one ... have someone
cross-check the work. However, I'd feel more comfortable if the
maintainer just put a call out to the DQAG for someone to check it, and
then have the DQAG assign people as needed (or rather, members of the DQAG
volunteer as needed). If a particular tester works out for a package,
that tester may stay with the package the next time ... or maybe next time
the tester might not be available, and the someone else steps in. In any
event, a 'QA: Tester Name <tester-email@foo.org>' line in the control file
would be a good thing to have. Just don't count on the tester being the
same from one release to the next.
Also, it would be good if the tester were involved as early as possible
in the release cycle (e.g. if you get a new upstream release, ask your
previous tester as soon as possible to assist with testing as you work,
and if you don't get a speedy reply or the tester declines, put out a call
to the QA group)
> * Setup some 'test suites' (and run them periodicaly) ?
> - for the "complete distrib"? (eg: POSIX compliance)
> - for a functionality (eg: java support, TeX).
> - for a precise package: (eg: 'compiler self test' for guavac, bison, ...)
This last one should be called from debian/rules ... whenever a package is
built it should auto-self-test.
> * Open problems:
> [ Feel free to comment ]
> - How to make sure a maintainer doesn't "overwrite" a fix made by the DQAG?
The most straightforward way is for maintainership to revert to the QA
group if the maintainer doesn't make the fix in enough time. Of course,
the time period must be carefully chosen depending on the circumstances.
Then the cross-check to ensure that a maintainer doesn't "overwrite" a fix
is that the maintainer must obtain the package from the QA group again
before further releases are accepted from the maintainer. Maybe this
sounds harsh, but I think it encourages more of a sense of responsibility
than the "3 successive QA group fixes" rule above. I think the QA group
should put off taking over a package as long as possible, and encourage
the maintainer to do the right thing with the package instead. If the
QA group ever needs to make a release of their own, then the ownership
of the package should immediately revert to the QA group.
> [ Ask the maintainer to acknowledge the DQAG fix by email? ]
> - Who will make the final decision when the DQAG and the package maintainer disagree
> with each other on a bug fix?
> (1/ the DQAG, 2/ The maintainer, 3/ The BoD, 4/ Someone else?)
I think there are two classes of disputes ... 'discretionary fixes' that
should normally be left up to the maintainer, and more serious fixes that
the QA group may decide are show-stoppers. If the two parties really
can't sort this out between them, then they should choose an arbitrator
to help them to reach a consensus. At work, I find our department head
works well ... I'm not sure who the equivalent is here (the project
leader?) But it should not be the BoD ... it needs to be a single
individual with good mediator skills.
> IMPORTANT NOTES:
> * The proposed fixes must be taken into account by package maintainers!
> * There already are too many bugs which are outstanding since months
> and are fixable with a 2 line fix provided in the bug report!
> * The DQAG must help maintainers, but does NOT replace them.
I'd go even further and say that the DQAG should really try to help
maintainer develop good skills and procedures for testing their own code
so that things aren't left to be 'caught in QA'. The existence of a QA
group can make developers sloppy because they are not as rigorous in
testing their own code.
> * Also, maintainers must not rely on the DQAG to take care of their packages, or fix their
> bugs...
> * The DQAG is only meant to (in no particular order):
> - take care of orphaned packages.
> - HELP maintainers by providing fixes they should apply THEMSELVES.
> - detect as many bugs as possible (and as soon as possible) in the uploaded packages.
Good except for the last point. It is the maintainer's job to detect as
many bugs as possible *before* the package is uploaded. The goal should
always be that the maintainer uploads bug-free code. The QA group should
only be there to verify that the maintainer has done their job. They
should not be a 'safety net' that the maintainer relies upon to catch
their bugs ... often the QA group will not be as close to the package as
the maintainer, and will not be able to spot many of the potential
problems that the maintainer will be able to see 'up close'.
Ben.
--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org .
Trouble? e-mail to templin@bucknell.edu .
Reply to: