[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Quality Assurance in Debian


I didn't want to announce this today but since Joey has brought up
a hot topic on -devel this morning, I feel that it's time
to make public what some people (me, Christian Kurz, Tortsten Landschoff,
Josip Rodin, Vincent Renardias and Martin "Joey" Schulze) have 
prepared for resurrecting debian-qa.

Please read the text attached (or the html version you can find here :
http://tux.u-strasbg.fr/~raphael/qa/doc/qa.html/). Any comments are

[ Please read below only once you read the text mentionned in the
  paragraph above ]

I believe that the proposed organization (associated with the tools
developed aka 2 mailbots, a database and the website) can be the 
solution for :
- following the Release Goals. If you wonder why I said that check out
  Each of the release-goal can be splitted in simple tasks. Everybody can
  follow what needs to be done on the web and can help.
  It's inspired from the Gnome Status Page.
- correcting buggy packages [more explication below why it could work]
- finding "cruft" and maintainers that have vanished (this is the
  job of the QA committee)

Now some comments why this can work. Many people expressed their interest
to work for Debian's quality but they said it was difficult because :
- they don't know where to begin
- they don't want to work too much (they'd like to grab a little task they
  can do in 2-3 hours and after that forgot QA since the following and so
- they don't want to continue writing patch if they are not used. Sometimes
  patches sent to the BTS are simply ignored (which does often mean that
  the maintainer vanished)
- they don't want to do NMU because they don't want to argue with
  the maintainer
- even if they want to do a NMU, they must contact the maintainer
  and wait for his response. If the maintainer doesn't respond, they
  should wait 3 weeks (it's 3 times enough to forget about the
  NMU they wanted to do) ...

Now, let's listen how those problem are solved :
- they don't need to choose a package and a bug to correct, they simply
  work on what the committe has brought up to them
- the committee has splitted the work in many little task so that everyone
  can help
- once someone has provided a patch for the task, it is marcked as "patch
  available". Those tasks are stored in a special list that the
  committe check from time to time. When a task is marked as patched for
  more than 3 weeks the committee can ask for an NMU in order to 
  integrate the work. Note that the person doing the NMU don't have to be
  the person who wrote the patch.
- people doing the NMU at the request of the QA committee don't have to
  fear the maintainer's reaction. They'll simply respond "I did it
  because the QA committee decided it, check with them if you have a

The main tool for debian-qa will be a new (dynamic) web site where
the tasks to do are listed. The tasks where a patch is available
are also listed separately. And last but not least you can find a list
of meta-tasks. The meta-tasks are big tasks composed of many little
tasks. This is a practical way to follow the release goals and other big
jobs like "Killing release critical bugs" and so on.

The web site can be easily updated by sending mails to 2 mailbots. The
committee only can create new tasks but everybody can update the
information available for each task (setting the task as 'patch
available', closing it, setting the status field, ...).

What can I say more ?

I hope that we can agree on this proposal (i've already spent many hours to
write the tools) and that we can start to try this way of working.

Oh yes, for the QA committee: myself, Christian Kurz and Tortsten
Landschoff are volunteers to begin.

BTW, do we need to vote the proposed text ?

One of the main thing I want to do right at the beginning of our work is
contact all maintainers and ask them to reply (yes a ping message where
you have to reply to explicitely tell that you are still with Debian). They
will be also asked if they don't want to orphan some of their packages,
that they do no more use.

Maintainer that do not respond could be removed from the keyring later
(some months after another try). And the packages orphaned that nobody
wants to adopt could be removed (yes there are some unuseful packages,
like psptools).

Raphaël Hertzog -=- http://tux.u-strasbg.fr/~raphael/
<pub> CDs Debian : http://tux.u-strasbg.fr/~raphael/debian/#cd </pub>
                        Quality Assurance in Debian

                   Raphaël Hertzog <hertzog@debian.org>

                    Christian Kurz <shorty@debian.org>




     With the growing number of developers working on Debian (>500
     registered developers at the moment), the increasing number of
     packages (>3500 already) and the large number of bugs (about 10000)
     Quality Assurance (QA) is urgently needed. This document explains a
     proposed solution for Debian about the quality assurance issue.

Copyright Notice

     Copyright 1999 by the Debian Project

     This text is free software; you can redistribute it and/or modify it
     under the terms of the GNU General Public License as published by the
     Free Software Foundation; either version 2 of the License, or (at your
     option) any later version.

     This is distributed in the hope that it will be useful, but WITHOUT
     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
     FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
     for more details.

     A copy of the GNU General Public License is available as
     /usr/share/common-licenses/GPL in the Debian GNU/Linux distribution.



     1.        Why do we need Quality Assurance ?
     1.1.      Some facts
     1.2.      Goal of the proposed organization

     2.        Overview

     3.        The different entities
     3.1.      The QA committee
     3.2.      The QA core team
     3.3.      The QA members


1. Why do we need Quality Assurance ?

1.1. Some facts

        * The number of bugs is growing as is the number of packages.

        * Some packages have many bugs because the maintainer left Debian
          without orphaning his packages.

        * Some maintainers can't manage all their packages and accept to
          live with buggy packages.

        * Some bugs are sometimes too difficult to correct for the

        * There's also period during which some maintainers can't work at
          all for Debian, but their packages may still need some attention.

        * In general, the quality of Debian can be improved in various way.
          There are still lots of packages not using all the suggested
          tools and integration utilities (menu files, doc-base files,
          alternatives, proper section/priorities, ...).

1.2. Goal of the proposed organization

     Working on Quality Assurance is a big job, in fact it should not be
     the job of some volunteers only. Each Debian developer should take
     care of the quality of his packages. However this cannot be enforced
     by any rules because quality is sometimes a point of view. Therefore
     this document won't define strict rules for the packages (the Debian
     policy already dictates the rules) but a global organization so that
     every volunteer can help by doing things that will concretely improve
     the quality of Debian. One of the main point here is scalability, many
     people must be able to work together on quality assurance without
     interfering too much.


2. Overview

     The work is coordinated on <debian-qa@lists.debian.org>, there's an
     official committee which is in charge of leading the volunteers that
     want to help with QA issues. There's also a QA core team (composed of
     regular QA workers) that can do NMU when the committee request it.

     The QA committe can lead people into different tasks :

        * provide help/patches for particular bug reports (reducing the
          number of bugs in buggy packages).

        * check if release critical bugs are corrected rapidly, if after 2
          weeks one is not corrected, they may contact the maintainer and
          ask him if he wants help or NMU ...

        * help for mass update, maintain list of updated packages, send
          mail to the maintainers, allow NMU after X weeks without

        * check if all packages providing user programs (X11 or not) come
          with menu files.

        * check that all packages that provide documentation register it
          with doc-base.

        * check if package are policy conforming (lintian is very useful
          for this).

        * provide more documentation where it's lacking.

        * check if old packages (ie without recent upload) still work and
          if they are still maintained.

        * check if the informations provided by the packages are accurate
          (dependencies, sections, priorities, description, ...).

        * check if all the packages are linked against the proper libraries
          (so that old libraries may be dropped in favour of most recent

        * check that we have the latest upstream version.

        * any other useful (QA related) job.


3. The different entities

3.1. The QA committee

     It's composed of volunteers designated by the leader. Their members
     would manage a database of QA related tasks. They lead the QA core
     team and the QA members. They contact maintainers which could benefit
     from some help because they have buggy (or difficult) packages. They
     offer the assistance of the QA members. But when the QA group has
     provided help (via patches in the BTS in general), the maintainer has
     to upload corrected packages (considering of course that the patch are
     ok according to him) in a time frame of 3 weeks (or 1 week during a
     freeze). After this delay, the QA committee can ask the QA core team
     to NMU the package in order to integrate the patches and to correct
     all the bugs that can be corrected.

     If the maintainer doesn't respond to the mails from the QA committee
     in a delay of a month (considering that the maintainer did not
     announce that he's in vacation), the committee may first allow a NMU
     to be done to correct the bugs and may later (two months after the
     initial mail) orphan the package if he still hasn't responded.

     The QA committee is also in charge of the WNPP (Work Needed and
     Prospective Packages) list since they are already contacting
     maintainers about packages they might want to orphan.

3.2. The QA core team

     It's composed of QA members designated by the QA committee, only
     people actively involved in QA for more than a month can become member
     of the QA core team. Beeing member of the QA core team doesn't give
     you many power, the reason of the existence of the QA core team is to
     know who can do NMU when the committee request it. However you can be
     proud to be part of it, because this does mean that you're doing a
     great work for Debian and his quality. If a member of the QA core team
     suddenly stops working for QA, he will be removed from the list.
     However he can rapidly integrate it again when he restarts working for

3.3. The QA members

     Everybody who is subscribed to <debian-qa@lists.debian.org> is a QA
     member. The more QA members the better ... the task list will be
     posted to debian-qa by the committe once or twice a week, the QA
     member are encouraged to work on the problems submitted by the
     committee. A QA member who has worked a lot in this area can ask the
     committee to be added to the list of QA core team members.


     Quality Assurance in Debian

     Raphaël Hertzog <hertzog@debian.org>
     Christian Kurz <shorty@debian.org>


Reply to: