Hello world, So, I'm going to try breaking my -devel-announce habit. I wonder if this means nobody'll notice this mail. I guess I can hope, hey? As promised: I hereby boldly predict that Debian GNU/Linux 3.0 (woody) will be released before the end of July, 2002! And now, a brief musical interlude. The battle's done, and we kinda won, So we sound our victory cheer: Where do we go from here? Anyway, as the dust settles, it's probably about time to work out what just happened and figure out, well, where *do* we go from here? So what did just happen? Well, we just tried a release with a completely new methodology: rather than hack on stuff for a while, then spend however long it takes making it reliable, we tried to keep woody bug free for the entire time we worked on it. It didn't quite work out that way, of course. We had a whole bunch of problems: 2000/01/16 - 2000/12/19: woody and unstable were literally the same thing, so woody certainly didn't have minimal bugs by any means. on the 19th, woody got rolled back to equal potato, losing X 4.0, and bunches of other stuff. it took a few more months to get those packages back in in a way that didn't break things too badly. Up until the first few months of 2001 (ie, around six months after potato's release), it required pretty constant attention, all of which did a good job of distracting me and presumably others from getting on with the other parts of releasing woody. 2000/01/16 - 2002/04/09: woody missing functional boot-floppies. in more detail: - 2001/04/08: no boot-floppies at all - 2001/06/21: boot-floppies that didn't work even on i386 - 2001/08/24: boot-floppies only built on some of the architectures that released with potato (alpha was the last one) - 2001/10/18: boot-floppies not built for all the architectures that ended up releasing with woody - 2002/03/14: debootstrap has RC bugs of one sort or another - 2002/04/09: b-f's built everywhere with releasable components (there was an additional b-f's build done between May 16, and May 21st) 2000/01/16 - 2002/04/16: missing CDs for some/all architectures (we had unofficial i386 CDs since 2001/07/20 and probably earlier, we didn't get alpha CDs until some time after 2002/04/16, sparc CDs weren't bootable at least until the most recent b-f's upload in May) On the other hand, we didn't really make much of an effort towards getting official woody CDs until early this year, either. 2001/10/19 - 2002/06/21: more architectures in woody than the security team were able to support. (mips, mipsel and s390 were added on the 19th. hppa and ia64 had been added earlier than that, so it's possible that it would've been too much effort to support those architectures too) the security team had been trying to improve the situation for quite a while on general principle, but nobody realised just how much of a problem it was until March or April this year. everyone probably remembers how the story went from May onwards. Those issues alone were enough to make woody not be released before June: releasing without CDs or any way of installing particular architectures is unacceptable, as is releasing software we can't do security updates for. And given that July was largely spent making sure the new security infrastructure worked, and fixing bits that didn't, it pretty well means that we couldn't have released _any_ earlier, if we hadn't done the above substantially better, somehow. What's particularly interesting is that we *didn't* end up spending the last few months before the release with everything else ready, trimming down the release critical bug list. By contrast, even though potato had working boot-floppies and CD images at the end of the second test cycle , we still had to make changes to dpkg, gcc, postgresql, sysvinit, xfree86 and so on. There were about 150 packages updated all up in the month between test cycle 2 and test cycle 3 (test cycle 3 took another month, with no changes). In the three months leading up to woody's release (May - July), there were about 100, which compares fairly favourably, especially when you also consider the increase in both packages (3900 to 8300 in main/i386) and architectures (6 to 11). Also fairly indicative of the decreased effect of the release critical bugs on the release process overall is probably the "spike" in mid-February on the RC bug graph: http://bugs.debian.org/~wakkerma/bugs/graph.png It drops from a peak of about 442 on 2002/02/15 to a low of 95 on 2002/03/23. This coincides most notably with the end of linux.conf.au 2002 on the 9th (which freed up a fair amount of time for me and thus resulted in a fair number of packages getting dropped from woody and bugs getting downgraded), and also with a couple of bug squash parties and probably some renewed vigour amongst NMUers in general . Those were the big ticket items, anyway -- at least as far as I'm concerned. It wasn't all that happened by any means. We also had: crypto-in-main: This took absolutely ages to get done. Finding a lawyer to get legal advice from, working out what exactly we wanted advice about, making sure we understood the advice, working out what we'd do about it, and then doing it all took months. It was probably important to get this done: a fair chunk of software -- most notably everything postgresql related -- that was in main in potato had started depending on the SSL libraries by mid last year, and having to remove that from main would have been fairly unpleasant in a bunch of ways. This was all finished reasonably well by the end of April, though. the gradual freeze: You'll probably remember that for most of last year I was advocating a gradual freeze: first we'd freeze policy, then base, then boot-floppies, standard and base, and finally optional and extra, then release. This was a complete flop. We got away with freezing policy reasonably well (although even it had to be "corrected" with a helpful little webpage ). We tried freezing base at the end of November, but by January we still had a fair number of severe bugs in base (glibc alone was still having packaging updates in late March and April), and base uploads started getting automatically promoted to testing again, albeit with twice the delay. The only part of this that probably *was* effective was the "NO MAJOR CHANGES!!" policy, which has been more-or-less in effect since last late last August (11 months prior to woody's release, although I guess most of you probably could've worked that out yourself :). It was *probably* more effective than we needed, in fact. the RC bug list: Unfortunately, we pretty majorly screwed this one up. The bug tracking system has never been particularly good at handling multiple distributions. This hasn't been too much of a problem when we just had stable and unstable, since we've never worried too much about tracking the bugs in stable, but it's a fairly serious problem with testing, since it makes it hard to get a grip on what RC bugs remain in testing: some RC bugs are new in the unstable version, others have been fixed in the unstable version, and so forth. This needs some fairly significant changes to the BTS before we can fix it though (which was why it hasn't been done already). There's probably more, but given that it'd be nice to finish writing the email before sarge is released, let's move on. Hopefully that covers the most significant parts of "what just happened?" on the "release management" side of things. That, naturally enough, leaves us with "where next?". Well, what's next is another distraction: http://lists.debian.org/debian-devel-announce/2001/ debian-devel-announce-200104/msg00004.html Grep for "realistic schedule". Doh. I think there're probably three things we want out of sarge's transition to stable: (1) Speed. (2) Less wasted time. (3) Better communication and more transparency. The main things blocking (1) were: (a) the six month delay before we started trying to release (b) boot-floppies taking twelve months to develop (c) CDs not being ready (d) not noticing that security wasn't ready, then getting it ready By (2), I'm referring to things that could have been ready by the time we released (or that *were* ready), but that nevertheless didn't make the release. dpkg 1.10, xfree86 4.2, better i18n in a number of places, and a few other things. The things blocking this were (probably): (e) the "NO MAJOR CHANGES!" freeze, 11 months prior to release (f) the staged freeze, from a similar time (g) the real freeze, from 3 months prior to release The final point, (3), was probably the most irritating for a bunch of people. Particularly from January 2002, it really wasn't made particularly clear where we were going, or how were going to get there. In most ways, there was little to be done about that for woody: it's difficult to tell people what's going to happen when, if you don't know yourself; and that's naturally the case when the things that need to happen involve significant development, which each of boot-floppies (from January 'til April or so), CD images, and the security buildd stuff did. OTOH, we can probably do a better job of avoiding getting ourselves in that situation in future. So anyway, that's my take on what happened and what the problems are. I mightn't be the most unbiassed observer, of course. There might be other problems that are more significant than those above: delays getting packages into testing because other packages in unstable are broken can be annoying -- and maybe it's more important to fix that than to release quicker; the RC bug list might still be a bottleneck, in spite of "testing" or the above arguments. I don't think so, but you might. My opinion on what we should do about this hasn't changed much: I still think the best way of getting consistent, controllable is to maintain a candidate distribution in a releasable state permanently. Where we failed in doing that for woody is probably pretty obvious: we didn't have releasable boot-floppies, official CDs, or a security updates repository for woody for most of its life as "testing". As such testing *wasn't* releasable for quite a while, and we had to wait ages before we did actually release. Time for another digression. I've been asked a few times if I intend to continue as release manager for sarge, and I've been fairly coy and noncommital about answering. Nominally, there're only two things you need to be release manager: the support of the DPL (to get your name up in lights on the "organisation" page), and the support of ftpmaster (so that when you say "it's released", it actually is). In reality, at least in my experience, it's at least as important to have most people either to listen to what you say and to head in that direction, or to give you an equally or more effective alternative. I can sing and dance all I like about having everything releasable at all times, but if no one else gives a damn about it, we're not going to get anywhere. Now, I don't really know how to judge the second part. I'm concerned that there've been comments like: <moshez> aj certainly has given the impression he wishes to continue to be RM, and from the "vote  bdale" I surmise he has had some understanding with bdale...'' (this isn't the case, remotely) or that I have to go through huge flamewars fairly regularly about release oriented matters . Now, maybe that's just a minority thing, or maybe it's just indicative of the project as a whole at the moment, but it's not helpful for me to be feeling nervous about raising these matters, least of all when *more* transparency and communication is what we're aiming for. Now, like I said I don't really know how to judge this well. Maybe the easiest way to start is by seeing if the hystrionics in this thread can be kept to a minimum. Anyway, I'd like to continue bossin' y'all around as RM, but it's possible everyone's sick of me doing that and would rather it come from someone else. I've no idea. But presuming that there's at least some curiousity as to what I see happening for sarge, I'll keep on rambling anyway. The way I would like to see sarge run is basically to get it to "roughly" releasable status ASAP then keep it there. By that I mean getting some official CDs made for testing quickly, and automating them so they get updated every week or so. I mean getting some installers for sarge into the archive and maintained across all architectures as sarge develops. Likewise for security updates. And, in general, likewise for anything else that's important to have working when we release. The easiest one of these to get and keep working is probably the CD images. I've already talked to at least Phil Hands about this, and it looks like we should be able to setup raff.debian.org to serve jigdo images for sarge (and some isos) updated on anything up to a daily basis. Getting them to be bootable obviously depends on us getting some installation tools that work with sarge, but hopefully that can be worked around in the meantime -- CDs that're only useful for upgrades are better than none at all. Security updates are probably not too difficult to manage either. The hardest part is probably keeping track of which packages need updating and what the fixes are. On the upside, the current security buildd stuff that we just did for woody should work with sarge already. It's not clear if we need a separate security team to manage updates to packages in testing, or exactly how much effort it will be. (And before suggesting that we've already got this problem licked with the new security infrastructure, please realise that we've got the possibility of releasing a Hurd or BSD based distro in the not too-distant future, either of which have reasonable odds of introducing new complexities to maintaining security updates; and, in any case, it's the unforseeable problems that we're aiming to avoid) Getting the installer right, however, could be quite difficult. We've never done it before, so who knows? There are many problems with boot-floppies, but the only ones that're relavant here are the ones affecting their maintainability. In particular it's proven exceedingly difficult to get boot-floppies to build across all architectures, and to update it from one release to the next. One set of problems is related to just how difficult it is to get b-f's to build. Unlike just about everything else in Debian, b-f's doesn't autobuild. Try it, you won't like it. Building involves getting an up to date checkout from CVS, making sure your kernel has the appropriate options available, getting various .debs from the archive to flesh out the installation disks themselves, trimming unused symbols from any libraries to make sure they fit in 1.44MB, and so on. Kernel updates, program updates, just about everything tends to knock b-f's off balance and require a fair degree of thought and effort to make them work again. The official solution to this is the debian-installer project. It's a rewrite of the installation system from the ground up, with two important features. The most straight forward is that it uses debconf for its user interaction. The more important, as far as ongoing maintainability is concerned, is that whereas the woody installer is built as a single "chunk" (ie "boot-floppies"), debian-installer is made up from many "udebs" (micro-debs) which, generally, are smaller counterparts to real debs. For example there is a debootstrap.udeb, which provides the installer with the tool to actually unpack the base system. It's built as part of the regular debootstrap upload, and autobuilds quite successfully. This applies to the entire d-i installer: all the tools, busybox, parted, kernel-image, whatever are all built as .udebs and that's all there is to it. Well, actually it's not quite all, and this is the first problem with d-i that probably needs addressing. You can't put a udeb onto a floppy disk and expect it to boot. You have to run a special script that gathers all the udebs you want to boot with (a kernel, a UI of some sort, a tool to let you get any other udebs you might need and aren't including from the net or a CD or similar), makes a filesystem out of them, and makes it bootable. This is fine and necessary, the part that's a problem is that this script is likely to need to be run by hand (not autobuilt) on a machine of the target architecture, probably as root. While this is better than the b-f's situation, it's not particularly efficient or scalable. What we want is to be able to upload a new version of debootstrap, and then have udebs automatically built, and floppy and CD images automatically constructed for all architectures. Probably the most difficult part of this is making, say, a bootable powerpc image on an i386 host. The second issue, which isn't really a problem, but could use addressing anyway, is that as a complete rewrite of just about everything, d-i is a pretty risky project. At present, even getting cfdisk to work with d-i is a major undertaking, since it has a UI that's not debconf. However, the nice thing about .udebs is that, being modular, you can mix and match them however you like. As such, it should be possible to construct a dbootstrap.udeb or a pgi.udeb, that uses the boot-floppies UI or PGI's UI to install a Debian system, while retaining most of d-i's maintainability benefits. This isn't likely to be easy on two grounds. First, d-i fairly fundamentally assumes you're using debconf, and neither dbootstrap nor PGI are likely to sit well with that. Avoiding that might be as simple (if inelegant) as just not running the postinst's of any udebs when you "install" them, or it might be more complex. The other problem is that, for obvious reasons, dbootstrap and PGI don't have any code to download and install new udebs. That's probably a necessary feature if you want to allow floppy installs. PGI probably has the additional problem that it'd need xfree86 udebs before it could really work -- if anyone wants to try making PGI.udebs, it might be a good idea to get the text-only mode working first. The third issue, just for completeness, is that d-i still only works in a very limited sense. It's not ported to non-i386 architectures, and it isn't remotely as flexible as woody's installer from a user's POV yet. There's no big surprise there though, really. So anyway, that's my theory. What I'd like to see in the next month or so are some official "testing" CD images (jigdos for everything, .isos for at least CD#1 and CD#2 of i386, I guess) on a .debian.org site that're getting updated regularly (at least every two weeks). I'd like to see some limited, but working d-i images for i386 in a similar time frame; for bonus marks adding some bootable 50MB (or so) testing CD images along with the others. I'd also like to see some some sort of movement towards getting proper security support for testing, but real milestones for that are hard (for me) to pick. (Bdale's probably going to send out a mail about all the /real/ features we're aiming towards for sarge sometime soon, so happily I don't have to :) Cheers, aj  Probably worth reading again, in case all the excitement since has let you forget. NMUs are good, mmmkay? http://lists.debian.org/debian-devel-announce/2002/ debian-devel-announce-200201/msg00014.html  http://people.debian.org/~ajt/woody_policy_addenda.txt  eg, Bug#97671 which went all the way to the tech ctte, or things like getting b-f's to stop adding new features or changing kernels and so on while they don't work everywhere, or on what hurd should do to be suitable for a stable release. -- Anthony Towns <firstname.lastname@example.org> <http://azure.humbug.org.au/~aj/> I don't speak for anyone save myself. GPG signed mail preferred. ``If you don't do it now, you'll be one year older when you do.''
Description: PGP signature