(subtitled: a tech proposal to make Manoj happy ;) Hello once more, world, This proposal presents a relatively minor change to the archive, but a fairly serious change to the way we actually make use of the archive as we develop a release. The essence of this proposal is that we maintain a distribution somewhat like frozen throughout the release cycle rather than just at the end of it. This "frozen-like" distribution will only consist of packages that are basically ready to be released, but will happily accept improvements in functionality as well as bugfixes. The main focus of this new distribution is to keep a track of what we'd be releasing if we chose to release not just Real Soon Now, but *Right* Now. This is a long proposal. My apologies, but I think the details are important enough to warrant the excessive number of words. (please note, this proposal attempts to focus on the technical issues. However, most of the justification for the changes proposed are philosophical and pyschological, so if you're not already familiar with why a `prerelease' distribution would be useful, this proposal might make more sense if you read the `Philosophy' section first) The dists/ Hierarchy ==================== 0. Introduction --------------- At any one time, there are two distributions of Debian that are relevant for the developers: the current release, that most of our users should be running; and the next release. Further, not every package is to be included in either distribution immediately, and thus there are two incomplete distributions that are also maintained, one that houses uninstalled updates to the current distribution, and the other that houses uninstalled updates to the next release. 1. Terms -------- For some degree of clarity, some terms are used in a more specific manner in this proposal. This proposal doesn't recommend that these are appropriate formal meanings for these terms within the Debian project, just that they are convenient for this proposal. release -- A codenamed and numbered set of distributions of Debian GNU/Linux.. Debian 1.3 (bo/i386), Debian 2.0 (hamm/i386,m68k), and Debian 2.1 (slink/i386,m68k) are current examples of releases. category -- One of 'main', 'contrib', 'non-free' or 'non-us'. section -- eg 'base', 'admin', 'devel', 'libs', 'misc'. stable -- The current release of Debian GNU/Linux. prerelease -- The next release of Debian GNU/Linux. stable-updates -- A collection of packages that may provide added functionality to a system based on one of the stable releases. unstable -- A collection of packages that may provide added functionality to a system based on the prerelease. release-group -- The group of developer's who are most closely involved with making the release. This is intended to include people such as Brian White (the release manager for hamm, who had the final say what made it past the freeze and what didn't, etc), Richard Braakman and Wichert Akkerman (who maintained the "Hamm Bugs Stamp Out List"), and so forth. stable-manager -- The guy who takes care of the horses. Since we don't have any horses, in his spare time he can also fill the release-group's role with respect to the current stable release. 2. Layout --------- Each release of Debian GNU/Linux should be identified by a static codename, such as "hamm" or "slink". Each release should be represented by a directory named by the codename under the dists subdirectory, eg: /pub/debian/dists/ bo/ hamm/ slink/ There should be symlinks "stable" and "prerelease", and they should point to the current and development releases respectively, eg: /pub/debian/dists/ stable -> hamm prerelease -> slink For the stable release, the should be an "-updates" directory, to store updates to that release until they can be included in the release itself. The directory itself should be named after the codename of the release, but there should also be a symlink "stable-updates" for easy identification, eg: /pub/debian/dists/ bo-updates/ hamm-updates/ stable-updates -> hamm-updates There should also be an "unstable" directory, for updates to the prerelease release, and packages that are still in development and not yet suitable for official release. (The intended use of this is for packages such as gettext used to be -- that are not yet stable enough for public release, but are useful enough to be made easily available for testing purposes) /pub/debian/dists/ unstable/ Each release should be arranged as follows: /pub/debian/dists/slink/ main/ contrib/ non-free/ non-us/ disks/ cd-images/ Only the first four directories should be included in the `-updates' releases. In unstable, the disks directory should also be included, for development of as yet unreleased architectures. cd-images should only be provided for codenamed releases. The four categories should have a structure as follows: /pub/debian/dists/slink/main/ binary-all/ binary-*/ -- one for each included architecture source/ These should be subdivided further by section, and each section should contain symlinks for each package to the appropriate package in the package pool. The various binary-* directories should contain in their top level, the files: /pub/debian/dists/slink/main/binary-*/ Packages Packages.gz Release The source subdirectory should contain links to the source of each package present in any of the binary-* subdirectories. This may mean that one package has multiple versions of its source in the same directory if different architectures are not synchronised. The disks subdirectory should contain symlinks to the current architecture specific disks directories in the package-pool, for each architecture the release supports, eg: /pub/debian/dists/slink/disks/ disks-i386 -> ../../../package-pool/disks-i386/2.0.10_... disks-m68k -> ../../../package-pool/disks-m68k/2.1.13_... The cd-images subdirectory should contain the Official CD images for the release. That is, ISO-9660 images containing the binaries for each supported architecture, and images containing the sources for each supported architecture. These should be accompanied by detached GnuPG and PGP signatures of the images made by the release manager and the PGP or GnuPG signature of the developer who created the images. In summary, the dists directory should contain: /pub/debian/dists/ bo/ bo-updates/ hamm/ stable -> hamm hamm-updates/ stable-updates -> hamm-updates slink/ main/ binary-all/ binary-*/ Packages Packages.gz Release section/ foobar.deb -> [package-pool] source/ contrib/ non-free/ non-us/ disks/ disks-i386 -> [package-pool] disks-m68k -> [package-pool] cd-images/ [ISO images, and signatures] prerelease -> slink unstable/ 3. Procedures ------------- 3.1 Uploading a package ----------------------- ...in order to improve on a package in prerelease ...in order to add a package to prerelease In order to generally update a package included in prerelease the maintainer must do two things: * Upload the package into unstable for initial testing. * After the package has received sufficient testing (about a fortnight in unstable), and no release- critical bugs have been found, the maintainer should present it to the release group as a release candidate. NB: automatic reminders should be sent to the maintainers of packages that have sat in unstable for over a fortnight with no release-critical bugs, but have not been submitted as a release candidate. If a package is accepted as a release candidate by the release group, it should be added to prerelease as soon as all its dependencies can be satisfied (That is, a package, even though it is otherwise perfectly ready for prerelease will not be moved until either all the other packages it depends upon are in prerelease, or can be moved with it). [it is recommended that a "staging pool" be used to manage release candidates. This should be maintained on master as an actual distribution so that automated tools like pkgorder can be used to ensure dependencies are met, etc] ...in order to fix a release-critical bug in prerelease The only time when the above, overly cautious, procedure is not workable, is when a package needs to be rushed into prerelease. This is exactly when a release-critical bug is being fixed (whether there be a bug report on it or no). The correct procedure to follow in this case, should be that the package is uploaded to "prerelease" with urgency=medium or high. At this point the release group should be automatically informed that the update is available, and presuming that it hasn't been uploaded to prerelease by mistake, install it into prerlease ASAP. If the release-critical fixes involved security fixes, then once the package is installed, a mail should be sent to the -security-announce list describing the problem, and similar information should be added to the web site. ...in order to improve on a package in stable (new upstram release &c) ...in order to fix a release-critical bug in stable These two events should be similar to the corresponding changes to prerelease, with "prerelease" changed to "stable", "unstable" changed to "stable-updates", and the "release group" changed to the "stable-manager". More vigorous requirements than "a fortnight's testing" should probably be applied before a package is moved to stable, but this is left to the descretion of the stable manager. ...in order to test an alpha/beta package The final case -- where a package needs to be tested, but isn't felt ready for distribution yet (in particular if it crashes all the time, introduces security holes that haven't yet been fixed, or similar) it should be uploaded to unstable, but simply not presented to the release group as a release candidate (The automatic reminder sent by the release group should thus be ignored). This is expected to be the usual place of residence of upstream packages the upstream author does not wish publically distributed. Once the package is ready for release, the release group should be notified that it is now a release candidate, and the procedures for moving a package to prerelease apply. 3.2 Releasing a release ----------------------- In order to make a release of Debian GNU/Linux, the following should occur: * prerelease is frozen. This involves no longer moving release candidates into prerelease. This is in order to allow one or two fortnight's thorough testing -- note that there should be no (or few) release-critical bugs in prerelease to start with, so only *very* minor churn is to be expected. This is the beta release. * prerelease is released. This involves changing the stable symlink to point to the former prerelease ("foo"), creating a "foo-updates" directory, and pointing the stable-updates symlink at it, and creating a new prerelease ("bar"), and pointing the prerelease symlink at it, The foo-updates directory should be initially empty. The bar directory should be an exact copy of the foo directory, with only the codename changed -- that is, we will build the next release starting with the current one. 3.3 Removing a package ---------------------- ...because it's buggy If a package in prerelease or stable has release-critical bugs that cannot (or at least have not, and will not) be fixed (and it's not dpkg), that package should be removed -- either to be replaced with an earlier version, or simply dropped. This should be done by: * notifying the maintainer by direct email * removing the symlink from stable/prerelease * if that package (possibly a later revision) is not in stable-updates/unstable (as applicable) a symlink should be made from that area. * making a post to the appropriate groups as to why the package was dropped, and where it may now be found. ...because it can't be distributed 4. Authority ------------ The archive-maintainers are responsible for incroporating uploads into the archive. As described above, this involves adding that upload into the package-pool, and providing a symlink from stable-updates/unstable to the upload. This should all be automated as much as possible, and where automation is unreasonable, should still be done within 24 or 48 hours of upload. The release group are responsible for moving packages from unstable to prerelease, and as such need to: * send reminders to the maintainers of packages that are apparently in release-worthy condition * verify that release-candidates are indeed release-worthy (which should be basically "the maintainer says so, and no one else has filed release-critical bugs" and hence highly automatable) * send reminders to the maintainers of packages in prerelease that have release-critical bugs, that they need to get fixed *now* or they'll be removed. (given a week's grace?) This should probably also involve maintaining a `Bug Stamp Out List' for packages in unstable and prerelease. * remove packages from prerelease and move them (back) into unstable. * determine an appropriate point at which to freeze prerelease and go into beta testing. * determine the appropriate point at which to stop beta testing, and release. 5. What does it buy us? ----------------------- There are two main improvements this proposal should bring about: * First, releases should be *very* easy, yet remain at the high standard Debian expects. * Second, the prerelease distribution will form a far more reliable base for people who want up-to-the-minute packages, but can't afford the unreliabilities that following either the current or the proposed unstable entails. 6. Philosophy ------------- The philosophy behind maintaining a prerelease distribution, that is forced to stay reasonably bug free in spite of constant updates, is the zero defects policy Steve Maguire claims Microsoft follows in his book _Debugging the Development Process_ (Microsoft Press, ISBN: 1-55615-650-2). The theory is that developers should be forced to consider and fix bugs throughout the entire development cycle, rather than simply at the end. This is to avoid the scenario of gradually accumulating a bunch of bugs that need to be worked on just when you think you've finished the project. Now obviously, there's no point trying to force anyone to do anything with Debian, however the one thing we *can* do, is to make it clear sooner which packages are going to fall under the `if there are release-critical bugs, it doesn't get released' hammer. Additionally, the classification of Debian users into the following three groups was taken into consideration: * those who require a robust, bug-free system and do not wish to be bothered by constant change (due to the cost or difficulty of obtaining updates, the difficulty of verifying large upgrades still do what you want on mission critical systems, or otherwise). Those who want security updates made available, but would rather keep the wholesale upgrades down to a couple of times a year at most. * those who want to keep up with the latest packages, but don't want to run into the more severe bugs that can come up with actual development releases. * those who want to run the absolute latest of everything, and are willing to put up with some, possibly severe bugs, to get it. This doesn't consider people who want to install something then keep getting security updates for the next five years, or people who want the absolutely latest versions of everything, but no bugs whatsoever, for what are probably obvious reasons. 7. Considerations ----------------- 7.1 Intrusiveness of Automated Reminders ---------------------------------------- One concern might be that the automated reminders would become unduly annoying for alpha packages intended to be left in unstable. The way these should behave, however, should minimise such intrusiveness fairly successfully. In particular, if updates are uploaded within a fortnight, then no reminder should be sent (although, if the maintainer wishes one of the interim releases to be placed in prerelease, then the maintainer can explicitly nominate that version as a release candidate without having received a reminder). If a package nominally under development isn't being updated, and doesn't have any bugs, however, the occassional (fortnightly or monthly) nag doesn't seem entirely inappropriate. 7.2 Foo Must be Removed!! ------------------------- If a release-critical bug is found in a package that really can't be removed, the release group should, of course, use there discretion in acting on the above. Where possible it is probably best to revert back to an older version of the package, but where it's not, living with the bug may be more reasonable than harming the distribution. 7.3 Submitting a Package as a Release Candidate ----------------------------------------------- In the usual case, the maintainer of a package should be the only one submitting a package as a release candidate. For packages that have been orphaned (Maintainer: set to debian-qa), this means anyone, and for packages maintained by multiple developers, this means either of those developers. However, for non-maintainer uploads, and cases where the maintainer is unavailable, any developer may submit a package as a release candidate. In this case, the package maintainer should be notified of this happening, and the release group should give the maintainer a reasonable amount of time to object, if necessary (24 hours? a week?). Note that this does not apply to release-critical updates, which are submitted straight to prerelease, rather than going through the staging area. 8. Acknowledgements ------------------- The grouping of users presented in the Philosophy section is due to Bdale Garbee. The details and name of the "staging area" are due to Manoj Srivasta. The term "category" for the main/contrib/non-free/non-us `distributions', is from Jason Gunthorpe's "Release" files. The name "prerelease" is due to Joseph Carter. Most of the ideas in the proposal are due to discussions with: Bill Mitchell <firstname.lastname@example.org> Bdale Garbee <email@example.com> Brian White <firstname.lastname@example.org> Craig Sanders <email@example.com> Dale Scheetz <firstname.lastname@example.org> David Engel <email@example.com> Guy Maor <firstname.lastname@example.org> Ian Jackson <email@example.com> Klee Dienes <firstname.lastname@example.org> Manoj Srivastava <email@example.com> Philip Hands <firstname.lastname@example.org> "Rev. Joseph Carter" <email@example.com> Richard Braakman <firstname.lastname@example.org> Raul Miller <email@example.com> Errors, ommissions and any lack of clarity or forethought in the above are of course mine. Awaiting your comments. Cheers, aj -- Anthony Towns <firstname.lastname@example.org> <http://azure.humbug.org.au/~aj/> I don't speak for anyone save myself. PGP encrypted mail preferred. Remember to breathe.
Description: PGP signature