Re: bits from the release team
Sorry for the long mail, but i believe there is something important all the
way done, so if you cannot be bothered to read it all; please go down to the
point marked *IMPORTANT*.
On Tue, Jan 03, 2006 at 10:05:04PM -0800, Steve Langasek wrote:
> You have been harranguing the ftp team to approve new upstream kernels
Wrong, i have asked Gannef to do it quicker NEW handling, and i told him about
this as soon as i found out about the 2.6.15 kernel release, since i believe
infomring folk early is good courtesy if you want them to do stuff for you, as
they can then more easily fit it in their schedule.
Notice that NEW handling is also important in this case, since the
autobuilders will build out of incoming, but not out of NEW, and since kernels
are rather long in building, this is an additional delay.
I may have been more insistent, and thus more dissapointed when it failed to
happen the same day, for the 2.6.14 release, since this one culminated an
intense work session of the whole debian-kernel team to make the
2.6.12->2.6.14 transition with initrd-tools going away and co happen. We are
speaking of 2 to 3 weeks of intense work on the -rc serie, which culminated
in the release, with the claimed aim to do 0-day uploads which many many
believed was not possible, and it didn't happen not for technical reasons, but
for bureaucratic details. You would feal the same, but on a larger scale when
you where about to announce the release, and you couldn't for some stupid
reason fully not under your control and you believe it is an unnecessary
issue. Can you honestly say we wouldn't hear you rant about it afterward ?
But again, this is the second time this happens, so as soon as i knew, i
informed Ganneff, and didn't harrangue or demand or whatever, just informed.
I _did_ rant about the not-really-need for NEW in these cases, but that is
unrelated.
> through the NEW queue before they've even been uploaded -- for an amazing
> false optimization that burns good will with your fellow developers. Even
Yes, and we are all volunteers, the preparation of a release like the 2.6.14
one had demanded effort and sacrifice from about a dozen persons, do you not
think it burns good will to work on this if you fail to achieve your stated
goal for bureaucratic details, i know my motivation fell a bit when we did
indeed miss dinstall.
> if udebs *were* being built from the same linux-2.6 source package, this
> doesn't address the real reason why it's important to freeze the kernel
> early: *the kernel is a core component of everyone's system and detecting
> regressions takes a long time*. Anything that requires a reboot cycle or an
How long a time ? A month, two month, four month ? And what is the cost
balance between backporting all those fixes from the new version, and simply
using the new version.
Also, with the new approach of building -rc releases in experimental, we have
easily another month or so of time to test the kernels, and the possibility to
correct at least part of the issues in upstream before even the release.
But sure, there are issues, but i don't believe they are as time consuming as
you make them.
> installation test in order for users to detect bugs is going to need a
> longer testing period than other packages; the only way to ensure this
> happens is by freezing it early, i.e., around the same time as the toolchain
> packages for which we have the same problem of figuring out whether a new
> version is better or worse.
Bah, ... I can guarantee you users are quick in testing new kernels, part of
them are, and we test them ourself also and run them, so major issues are
found early. we knew about the powerpc debconf/postinst script fuckage before
he package left incoming, altough there was no way to stop it from entering
the archive, and the bug reports came in very very shortly after dinstall.
> The underlying assumption in your plea for a shorter kernel freeze is "newer
> is better". But people who accept this assumption unconditionally don't
> *run* Debian stable; so neither should we base our freeze timeline on an
Bah, they run stable with sid/experimental or self built kernels, which is the
sanest thing to do, especially given the messy kernel stable security
situation, which i warned you about in april.
> unconditional acceptance of it. Newer isn't necessarily better, but it is
> necessarily *different*, which is the enemy of stability.
No. the kernel evolves, but more importantly issues get fixed. There are some
newer breakage that may slip in, but all in all the fixing amount overweights
the breakage introduced, and this breakage can be fixed, so i will have to
disagree with this.
But i believe the point is elsewhere, with our current plan, we will always
have two release candidates in the archive, or a single good quality release
candidate. All i ask, and i believe all agree on that, is that the freeze date
and the choice of kernel is done solely on the quality and testing of the
kernel, and not like it was done for sarge, forced on us by the technical
burden of the messy packaging infrastructure which means switching kernels
means a month long delay.
*IMPORTANT*
So, i am not asking that we upgrade the kernel in 48 hours after the release
always, but that we have the possibility of doing it quickly if we see the
need of it. This includes a new kernel version, but also an abi changing
security updates, which as far as the hurdles involved are exactly the same.
And right now the situation is as follows :
1) the linux-2.6 upload. It can happen, in 24-48 hours provided availability
of a ftp-master to do the NEW handling and load on the autobuilders, altough
the latest could be supplmented by developper builds if needed.
2) the d-i kernel module .udeb build. This happens relatively quickly for
x86, and could also be done fast for the rest of the arches, but experience
proves that it takes a week or two still. Also NEW handling is needed for
this one again, and independently of all arches, so more delay and uneeded
work. The d-i team bounces the responsability to the porters with rather
strong words, and the porters (or at least taking me as example for the
powerpc port, or you for the alpha one) have really no much interest in
doing this, since it is unecessary and painful drudge work, which is time
consuming, and using unfamiliar infrastructure. Furthermore, they need to
redo exactly the same kind of work a second time they already did for 1).
3) the d-i image needs to be rebuild, which involves a third run through the
autobuilders, which is timeconsuming.
4) out-of-tree modules need to be rebuild. Right now this doesn't happen,
and the situation is a mess, there is no way to found out all such packages
easily, and most of their maintainers are outside the area of influence of
the kernel team.
5) third-party patches have to be checked and rebuilt. This is the only one
which involves real work and cannot be easily automated.
1) to 3) used to take a month in sarge, but i believe we seriously took this
down, since 1) was a big part of the time involved.
Now, my proposal is to continue the streamline process, to diminis the needed
time and effort for all involved. I believe (and i am not alone in that), that
by incorporating the .udebs into the linux-2.6 package, and providing an
automated set of helpers for the porters to handle the kernel configuration
time more easily, and at the same time handle the added/removed modules (since
new modules are created from new config options), the step 2) can be
eliminated altogether without any negative influence on the rest of the d-i
process. d-i and porter developper time is scarce ressource we have no excess
of, so let's use it efficiently instead of wasting it away in drudge work as
done right now, and laying blame, and agitating the stick of status loss for
the whole port.
4) is also an issue, and probably a worse one, but my believe is that we
should set a policy which would allow a binNMU rebuild in case of new version
and abi change, and making it RC critical for any package that fails this
policy and quick them out of the release, will allow in the long run for a set
of release modules that can be rebuilt after a new version or abi change with
only a single release manager intervention and the unavoidable buildd delay.
5) is human intervention, and not much can be done there, i propose removing
the patches that have no active maintainers, as nothing else is really
possible.
> There is still room for targetted fixes to the kernel after the freeze date;
> backports of new drivers, or backports of specific bugfixes, are certainly
> fair game. Taking a new version of whatever upstream happens to have
> released would not be.
But we will back out from abi changing security fixes, like we did for sarge,
because people are not interested in fixing the infrastructure in such a way
as to allow it, and resist with all fours and arrogant words any kind of
progress, for reasons i cannot start to understand.
> > My believe is that this kind of thing should be as much automated as possible,
> > to let the few ressource we have be used where best it should, a little work
> > at the start which will pay off forever after, this is what computers and
> > programming is for, to make the task of the users and programmers easier, i
> > think we all agree with that, or we would still be using boot-floppies :)
>
> I'm all in favor of streamlining the integration of new kernel versions into
> the installer, but I don't believe that the majority of the work involved
> falls into the "automatable" category.
Ah, yeah ? tell me exactly what in the module udeb creation cannot be
automated ? I know joeyh has some trouble with it, and gave a long list of it,
but this is only valid because he lacks the internal view of it, and cannot
rely on helped tools during the -rc in experimental config file fixing. He is
not a kernel team guy, and lacks the vision necessary to fix it as it should
be fixed. I would defer to you for (most) release issues, and to joeyh and fjp
for most d-i issues, but i expect that you extend the same courtesy to the
kernel team, as i believe we have the best expertise on this issue, and always
being second-guessed and criticized by external folk who think they know best,
is definitively something i am not happy about, and i guess that may probably
show in the tone of those mails.
Damn it, we are trying to make live easier for both the release team and the
d-i team, and the common infrastructure package and the way we handled the
devfs/ramdisk mess gives us some background that shows we know what we do, so
why are you all making it so difficult ?
Friendly,
Sven Luther
Reply to: