Re: Etch timeline is unrealistic because non-free firmware is NOT being dealt with
On Sun, Aug 13, 2006 at 07:14:02PM +0200, Goswin von Brederlow wrote:
> Sven Luther <sven.luther@wanadoo.fr> writes:
>
> > On Sun, Aug 13, 2006 at 03:47:13AM +0200, Goswin von Brederlow wrote:
> >> > The whole concept is of an extreme fragility, prone to break again and again,
> >> > cause lot of work for all involved, who become irritable, and bash on you when
> >> > you even mention it.
> >>
> >> I did it when working on the amd64 D-I for sarge. I found it quite
> >> trivial to do, a matter of minutes. In fact a hell of a lot simpler
> >> and way faster than getting the linux-2.6 source package to do
> >> something for me.
> >
> > Naturally, amd64 being one of the mainstream arches the d-i team cares about.
>
> No they didn't since amd64 was not official or even liked. It didn't
> even exist before I cloned the i386 config and adjusted them to fit.
Let's say that the amd64 arch is very very near the x86 one in design. After
all it is just a followup of the later.
> Into the SuSe/RH source? Not likely. And kernel-package doesn't deal
> with prebuild binaries.
kernel-package deals with creating kernel .debs. You need kernel .debs to use
kernel-wedge, at least this is the way it is designed to work, so you lose
nothing there.
> >> You do have raised some points of improvement for kernel-wedge. Like
> >> automaticaly adjusting to changed .config files or some other magics
> >> you mentioned. And some points for building module udebs from within
> >> linux-2.6 source although I don't agree with you there.
> >
> > Why not ? It is the only way we can bring down the delay for security builds
> > with abi change to the minimum time, and the less human intervention.
>
> I don't see security for the installer as a top priority. It is not
> like people will burn their daily security patched D-I image. It is
> not like the security team does do security releases for D-I. For
> stable that seems to get totaly ignored.
Indeed, but this is mostly because due to those issues they are incapable of
handling those security issues in a timely fashion.
> D-I in testing/sid, especially the daylies, are for developing. Not to
> run a secure and stable system. The have bugs, they have security
> flaws, they might not work at all. That is what unstable is.
My main problem is that sarge released with a remote root security hole,
which was known and publicized a few month before the actual release, which
means all sarge systems are potentially unsecure.
> >> > The d-i team only needs to handle a list of the modules it want to include in
> >> > this or that ramdisk, and all will work automatically.
> >>
> >> Which has the same problem as the kernel-wedge configuration has now
> >> when splitting the modules. When a module gets added or removed the
> >> list has to be adjusted.
> >
> > Err, no, because the list would be handled in the debian-installer package or
> > its devel tree in the svn repo, and not inside kernel-wedge.
>
> Moving the problem from A to B. Doesn't matter what svn repository it
> is in.
Yes, it does matter. If it is inside kernel-wedge, you need to upload kernel
wedge to benefit from any fix, while if it is in d-i, you can just build the
out-of-svn tree and be happy.
> > Notice that we already have the mapping of module .udeb packages to d-i images
> > in d-i proper, which needs to be done anyway in addition to the
> > kernel-wedge/linux-xxx-di work.
> >
> > Furthermore with the current situation, and given that not all modules are
> > built the same on all arches, and that binary size vary per arch, you end up
> > in an unholy mess when trying to build space constrained ramdisks, like the
> > floppy ones for example.
>
> That just means that you need per arch lists of modules. And
> kernel-wedge has that. Having each module as udeb doesn't change the
Kernel-wedge has no per arch list of modules, those are in the 12+ individual
linux-*-di packages.
> fact that you have a unholy mess what modules to put in the image.
You don't have an unholy mess, since you can select exactly what modules need
to go into each image, a simple list or a hierarchical per
arch/subarch/flavour list. Compared to the current case, where you have to
upload two packages to be able to modify the list included in d-i.
> > Furthermore, the build, failure, check what the heck kernel module has changed
> > name or dissapeared, or whatnot, is a *ROYAL PAIN*, and very much time
> > consuming., especially on slow arches with lot of flavours.
>
> A factor of 10000 or so faster than building the linux-2.6 source. As
> combined time the kernel-wedge time is neglible.
Ah, but the beauty of my proposal is that you have tool which handles the
config files and module-as-udebs without needing to do a full build. Like
Bastian created the nice debian/rules target to check all config options.
> > Furthermore, given that you need to install the kernel image package, you
> > can't even start to autobuild that without hosing the autobuilders.
>
> Last I use kernel-wedge, and this is ~2 years ago now, you could build
> all kernel udebs on a single arch with the help of a little wrapper
> script. Did somone break that?
This has been rumored, but i have not seen this script, and the latest round
of discussion during debconf/mexico, Frans rejected the idea of doing a single
kernel .udeb package for all arches, claiming this was not possible, that it
would break autobuilders who wanted to install the kernels and so on.
But then, maybe he was isssuing clueless suppositions ?
> Rewrite that script.
Get ride of it and build the .udebs from the kernel packages :)
> >> > modules which need to go into each ramdisk flavour, not mentioning that the
> >>
> >> Which is pretty constant from my experience since the modules are
> >> already split right into udebs.
> >
> > They are split into the right .udebs for the x86 and other mainstream cases
> > the kernel-wedge maintainer cares about.
>
> Even if they are not split right you don't change what udebs go into
> the ramdisk. You fix the split itself (which you already have to do,
> that work is already accounted for). So this part of the job is a NOP.
Ah, but the interesting part of it comes when the constraint for x86 floppies
and powerpc floppies for example impose different distribution of modules.
As an example, i was not able to build apus flavoured .udebs, since i don't
remember which .udeb included a module not built on apus, and which needed
kernel-wedge modifications, which Frans forbade me to make.
> >> > ramdisk package list has no support for per-flavour module selection, and you
> >> > have to end up with stuff like the netboot64 list, which has as sole usage to
> >> > add the ibm power hypervisor and virtualization modules, all an ugly mess.
> >>
> >> Something to improve. No argument for or against your proposal.
> >
> > Because you miss that my proposal makes this whole issue obsolet and
> > non-existent.
>
> No it doesn't. You still have to list the modules that get into the
> ramdisk. That list doesn't magically build itself just because you
> parse .config.
Again, you fail to understand. In the current case, you do the work 3 times :
1) the kernel team choses the configuration option for the kernel, and thus
chose which modules to enable or not.
2) the kernel-wedge contains a list of modules per .udeb, and is completed
by the linux-*-di packages, all 12+ of them.
3) the d-i package contains in the pkg-list a list of modules .udebs per
image type and arch.
My proposal gets ride of 2), including in 1) only a flat list of which modules
to build, and allows for finer grained control in 3).
> > Indeed. The difference is that the knowledge about this is disseminated in at
> > least 3 different places, and that the kernel-wedge maintainer cares about no
> > more than 3-5 arches compared to the 12 or so official debian arches (at least
> > the powerpc case repeteadly broke every so often).
>
> I only konw about one place: kernel-wedge.
>
> Where else?
So, why was the proposal i made during discussion at debconf/mexico, that now
that the package_number/line_size issues is solved, to revert to a single
package for all arches summarily rejected ?
> > The reasoning behind this analysis is that the kernel porter need to make the
> > decision about each new config option anyway, that the kernel has good
> > infrastructure in place for separating that choice in between debian-global
> > choices, per-arch choices, per variant (historically called subarches), and
> > per flavour.
> >
> > The decision on what modules to offer as .udebs for d-i is rather trivial,
> > once the decision on how to group them in .udebs is out of the way, and the
> > kernel team is as well if not better placed to take that decision. You need
> > the network modules, the disk modules, the display modules and the input
> > modules, and another set of assorted modules needed for each machine to run
> > correctly, like the fan control modules on apple hardware for example.
>
> Sure. That part is easy. So easy it is merged into kernel-wedge
> because D-I can just as well decide that too.
Yeah, so what do you think we have in all those linux*-di packages ? and why
does it take hours to upgrade those packages ?
> > Then the d-i team can cherry pick among those .udebs and put into each flavour
> > those they need, being able to do different variants for different hardware
> > much easier than the current step which takes a couple of week or so.
>
> Which is what kernel-wedge does now. You still have that decision at
> exactly the same spot you have it now. You just moved it from a
> kernel-wedge upload to a d-i upload. Ok, 1 upload instead of 12 Some
> progress. But still, all 12 ports have to change D-I first before the
It is not 1 upload instead of 12, it is 1 maintainer takes the decision to
upload and does it, compared to doing a kernel wedge upload first, and then
12+ porters need to go over the linux-*-di packages, without help or
coordination, and upload them with each a random delay due to busyness or
whatnot. Please look at the regular call to porters that the d-i leadership
has to issue, saying things like "arch foo and bar have not yet upgraded their
linux-*-di packages", often repeated during various weeks.
> upload. So in fact you haven't saved anything but just made every
> single arch block the next D-I upload instead of their own
> kernel-wedge.
Ah, yes, exact. This is a feature, it is the same thing we have the testing
scripts for, each arch has to be in sync (well, each RC arch).
And this is exactly what i reproach to the d-i team, and why they dislike my
proposal, because they only want to care about x86 and their random pet
arches.
> >> You didn't eliminate the need for a manualy tuned list of modules to
> >> put into udebs. You just shifted the problem from A to B.
> >
> > Indeed, i split the problematic into two different cases, and assigned those
> > cases to the people most competent in solving them, in a way which cuts the
> > requirement for human intervention to the minimum, both time wise and with
> > regard to the number of actors, as well as simplifying the whole issue by an
> > order of magnitude.
>
> You have split the work from one team to two. You have not reduce the
> work. One could say you have double the work because now twice as many
> people have to look at every change.
And guess who it is who does the d-i work ? Is it frans or joeyh ? No, it is
all those porters, and they are often the same who also do the kernel porting
job.
I can guarantee you that it is light-years easier and more friendlier for me
as porter to do those things inside the kernel team, than to have to do it for
d-i, and get insulted and bashed too.
> A new module gets added, kernel team has to decide if it becomes an
> udeb, D-I team has to look again and decide where to put it. And so
> on.
Exact, compared to now, the kernel team has to decide if we build it, then it
has to be added to the linux-*-di packages and kernel-wedge, and then you have
to add it to d-i.
So, unless you live in some alternate universe where mathematics work
differently, that is 2 actions for my proposal and 3 for the status quo.
> > The cost is small, the only drawback would be the relative augmentation of the
> > number of .udebs packages, but not the global size of them, and a little work
> > on the infrastructure side, which is out of our control.
> >
> > Even the package size augmentation is minimal, since you said yourself that on
> > most mainstream machines, bandwidth and memory size are perfectly able to
> > handle it, and on those arches which have problems with memory size, most
> > modules will probably be builtin, and thus the amount of needed .udebs is also
> > minimal.
>
> >From what I understand all your proposal does is shift work around,
> splits it up, fragment it. I don't see any work reduction.
>
> On the other hand I see drawbacks for prebuild kernel images, the DAK,
> the ram requirement for anna/udpkg (a factor of 8-40 in tripple
> replica of the index files) and most of all the user when selecting
> modules.
>
> So I have to agree with the D-I team. It isn't a viable proposal.
>
> > But i think we can agree that nobody regrets the current state of the kernel
> > packages, nor of the single day uploads, which has brought debian out of the
> > always-months-outdated kernel era into a more modern and responsive one.
>
> I do. It is a mess that nobody outside the kernel team understands.
> The old way was way easier to grasp. Maybe not better but simpler.
I hope you are joking ? Please ask the security team how they liked to do
security uploads, uploading 15+ packages, including 4+ different upstream
versions. Also, please go look at the pathetic calls for kernels sync for d-i
builds and releases of april last year or so, and the month long processus
this was.
> Especialy the incompatibility with make-kpkg to build custom kernels
> sucks. You can't build any kernel that requires an *-extra patch with
> make-kpkg.
Err, if this is so, file a bug report, and i am sure Manoj will fix any such
issues. It is a design goal of both the kernel team and Manoj to maintain full
compatibility, and we all invested over 6 month of effort to make it so.
I guess you have no clue about what is going on here, and thus make uninformed
and clueless assertions, please inform yourself.
Friendly,
Sven Luther
Reply to: