Explaining Debian's approach to QA
Over the last few months, I've found myself struggling to find a simple
way to describe our approach to QA to friends and colleagues. I reached
out to lamby a week or so ago and he suggested I brought it to -devel.
I don't think Debian's approach to QA needs to look like that of a
discrete software product or a large development consulting business or
a small team of people sitting next to each other.
Debian, being a universal operating system, is understood to have lots
of software, lots of architectures, lots of use cases, lots of teams,
etc. It does help to keep some analogies in mind, though.
The three main concepts I'm struggling to convey and where I'd love this
community's input are:
(1) QA at Debian is a multi-stakeholder process* where ftp-master, QA,
Release and other teams play a role but core accountability lies with
the package maintainer,
(2) The preferred vehicle for QA accountability in Debian is a bug
report, and,
(3) The QA "bar" is codified in Debian Policy but also in many other
places (maybe not comprehensively) including for example the Release
checklist
I qualified "process" with an asterisk above as I'm not sure if we have
documented a single QA process. It looks like there are places where
Policy is enforced (e.g., ftp-master), places where testing automation
is happening (e.g., lintian, piuparts, chroot/d-i runs in jenkins.d.n),
campaigns (such as the recent NMUs for reproducible builds) and "final
checkpoints" (e.g., release checklist) but I'm not sure if this is all
mapped and documented as a single process (and, if so, if it's updated
since a lot has happened in 10 years)
Of course, for folks that live in a CI/CD environment where the build
log and the stop light are the vehicles of accountability, the concept
of a piuparts run happening after you've uploaded and getting a bug
report that you then go address and "start over" is almost foreign to them.
Also, that Policy codifies many sanity checks but not things that some
organizations do like source code audits, etc., is also novel since many
of those organizations have "single source of truth" docs that codify
the "QA bar". Yet most people acknowledge Debian ships quality releases
even without an approach to QA that "looks like" other use cases in the
industry, so no prejudice there.
Any reactions to educate me and help me formulate a more complete view
would be appreciated.
Thanks,
bureado (not on this list)
PS: it would also be interesting to analyze vs. other distros. Other
distros are less universal, may or may not have highly distributed
teams, may only test for a few use cases, etc., so it's clear it won't
be apples to apples, but with things like OpenQA having some mindshare I
think it's an interesting exercise too.
Reply to: