Re: Etch timeline is unrealistic because non-free firmware is NOT being dealt with
Sven Luther <sven.luther@wanadoo.fr> writes:
> On Sun, Aug 13, 2006 at 07:14:02PM +0200, Goswin von Brederlow wrote:
>> Moving the problem from A to B. Doesn't matter what svn repository it
>> is in.
>
> Yes, it does matter. If it is inside kernel-wedge, you need to upload kernel
> wedge to benefit from any fix, while if it is in d-i, you can just build the
> out-of-svn tree and be happy.
And I can just as easily build the kernel-wedge out-of-svn. That
really makes no difference.
>> > Furthermore, the build, failure, check what the heck kernel module has changed
>> > name or dissapeared, or whatnot, is a *ROYAL PAIN*, and very much time
>> > consuming., especially on slow arches with lot of flavours.
>>
>> A factor of 10000 or so faster than building the linux-2.6 source. As
>> combined time the kernel-wedge time is neglible.
>
> Ah, but the beauty of my proposal is that you have tool which handles the
> config files and module-as-udebs without needing to do a full build. Like
> Bastian created the nice debian/rules target to check all config options.
Which helps exactly 0 if you build all the kernels and then detect
that a module was not manualy maintained in the "to be udeb-ed"
list. Since not all modules are to be put in udebs I see no way how
you will automate the list.
And instead of having a 1 minute run of kernel-wedge to build the udeb
for the one generic kernel of most archs you have a 6h build of all
the (non D-I except for generic) flavours. Getting a module into an
udeb would be a hell of a lot more time consuming.
>> > Furthermore, given that you need to install the kernel image package, you
>> > can't even start to autobuild that without hosing the autobuilders.
>>
>> Last I use kernel-wedge, and this is ~2 years ago now, you could build
>> all kernel udebs on a single arch with the help of a little wrapper
>> script. Did somone break that?
>
> This has been rumored, but i have not seen this script, and the latest round
> of discussion during debconf/mexico, Frans rejected the idea of doing a single
> kernel .udeb package for all arches, claiming this was not possible, that it
> would break autobuilders who wanted to install the kernels and so on.
>
> But then, maybe he was isssuing clueless suppositions ?
The script was in the repository and was never for auto-building.
Buildds must install the kernel image package because a Build-Depends
is the only way to download a deb on a buildd. You can not rely on a
network connection during build so you can't download the debs some
other way. And since you can't install a s390 kernel on arm that makes
building for all archs impossible for buildds.
With a manual build you can just apt-get the debs and unpack them on
the spot without having to install anything.
>> Rewrite that script.
>
> Get ride of it and build the .udebs from the kernel packages :)
>
>> >> > modules which need to go into each ramdisk flavour, not mentioning that the
>> >>
>> >> Which is pretty constant from my experience since the modules are
>> >> already split right into udebs.
>> >
>> > They are split into the right .udebs for the x86 and other mainstream cases
>> > the kernel-wedge maintainer cares about.
>>
>> Even if they are not split right you don't change what udebs go into
>> the ramdisk. You fix the split itself (which you already have to do,
>> that work is already accounted for). So this part of the job is a NOP.
>
> Ah, but the interesting part of it comes when the constraint for x86 floppies
> and powerpc floppies for example impose different distribution of modules.
That is why each arch has possibly different modules listed. And as
Joeyh posted even per flavour. This is really a non problem solve
years ago.
> As an example, i was not able to build apus flavoured .udebs, since i don't
> remember which .udeb included a module not built on apus, and which needed
> kernel-wedge modifications, which Frans forbade me to make.
You were forbidden to do it. It wasn't impossible or even difficult.
>> >> > ramdisk package list has no support for per-flavour module selection, and you
>> >> > have to end up with stuff like the netboot64 list, which has as sole usage to
>> >> > add the ibm power hypervisor and virtualization modules, all an ugly mess.
>> >>
>> >> Something to improve. No argument for or against your proposal.
>> >
>> > Because you miss that my proposal makes this whole issue obsolet and
>> > non-existent.
>>
>> No it doesn't. You still have to list the modules that get into the
>> ramdisk. That list doesn't magically build itself just because you
>> parse .config.
>
> Again, you fail to understand. In the current case, you do the work 3 times :
>
> 1) the kernel team choses the configuration option for the kernel, and thus
> chose which modules to enable or not.
Which is a trivial step as they basicaly enable all modules and hardly
ever disable one again.
> 2) the kernel-wedge contains a list of modules per .udeb, and is completed
> by the linux-*-di packages, all 12+ of them.
And per flavour.
That list changes on kernel updates (not so often) and much more
frequently on D-I updates as the amount of free space on the different
ramdisks changes.
> 3) the d-i package contains in the pkg-list a list of modules .udebs per
> image type and arch.
Has that changed even once in the last year?
> My proposal gets ride of 2), including in 1) only a flat list of which modules
> to build, and allows for finer grained control in 3).
No.
Since you want to build one udeb per module you are not eliminating
step 2, you are merging step 2 and step 3. Instead of a list what
modules to put in the core udebs you have a list of what single module
udebs to put onto the images. Same amount of work. Changes in the
amount of free space will need constant meddling with those lists just
like now.
>> > Indeed. The difference is that the knowledge about this is disseminated in at
>> > least 3 different places, and that the kernel-wedge maintainer cares about no
>> > more than 3-5 arches compared to the 12 or so official debian arches (at least
>> > the powerpc case repeteadly broke every so often).
>>
>> I only konw about one place: kernel-wedge.
>>
>> Where else?
>
> So, why was the proposal i made during discussion at debconf/mexico, that now
> that the package_number/line_size issues is solved, to revert to a single
> package for all arches summarily rejected ?
How does relate to where else that knowldege is?
>> > The reasoning behind this analysis is that the kernel porter need to make the
>> > decision about each new config option anyway, that the kernel has good
>> > infrastructure in place for separating that choice in between debian-global
>> > choices, per-arch choices, per variant (historically called subarches), and
>> > per flavour.
>> >
>> > The decision on what modules to offer as .udebs for d-i is rather trivial,
>> > once the decision on how to group them in .udebs is out of the way, and the
>> > kernel team is as well if not better placed to take that decision. You need
>> > the network modules, the disk modules, the display modules and the input
>> > modules, and another set of assorted modules needed for each machine to run
>> > correctly, like the fan control modules on apple hardware for example.
>>
>> Sure. That part is easy. So easy it is merged into kernel-wedge
>> because D-I can just as well decide that too.
>
> Yeah, so what do you think we have in all those linux*-di packages ? and why
> does it take hours to upgrade those packages ?
The config for kernel-wedge. I count them as part of kernel-wedge in
the bigger picture. Imagine kernel-wedge/linux*-di everywhere I say
kernel-wegde.
>> > Then the d-i team can cherry pick among those .udebs and put into each flavour
>> > those they need, being able to do different variants for different hardware
>> > much easier than the current step which takes a couple of week or so.
>>
>> Which is what kernel-wedge does now. You still have that decision at
>> exactly the same spot you have it now. You just moved it from a
>> kernel-wedge upload to a d-i upload. Ok, 1 upload instead of 12 Some
>> progress. But still, all 12 ports have to change D-I first before the
>
> It is not 1 upload instead of 12, it is 1 maintainer takes the decision to
> upload and does it, compared to doing a kernel wedge upload first, and then
> 12+ porters need to go over the linux-*-di packages, without help or
> coordination, and upload them with each a random delay due to busyness or
> whatnot. Please look at the regular call to porters that the d-i leadership
> has to issue, saying things like "arch foo and bar have not yet upgraded their
> linux-*-di packages", often repeated during various weeks.
One maintainer will never have all the knowledge to make the decisions
for all 12 archs. Otherwise one single maintainer could just upload 12
linux*-di packages on update.
>> upload. So in fact you haven't saved anything but just made every
>> single arch block the next D-I upload instead of their own
>> kernel-wedge.
>
> Ah, yes, exact. This is a feature, it is the same thing we have the testing
> scripts for, each arch has to be in sync (well, each RC arch).
>
> And this is exactly what i reproach to the d-i team, and why they dislike my
> proposal, because they only want to care about x86 and their random pet
> arches.
There is a difference between testing and sid. In sid versions can
differ. The testing script resolves all your out-of-sync version
concerns that you have already.
...
> I can guarantee you that it is light-years easier and more friendlier for me
> as porter to do those things inside the kernel team, than to have to do it for
> d-i, and get insulted and bashed too.
Yep. for YOU. But for everybody else the current system seems work
just fine. Through all this discussion i haven't heart about any
supporters for your proposal (which might just be because it is just
us two). Are there any?
>> A new module gets added, kernel team has to decide if it becomes an
>> udeb, D-I team has to look again and decide where to put it. And so
>> on.
>
> Exact, compared to now, the kernel team has to decide if we build it, then it
> has to be added to the linux-*-di packages and kernel-wedge, and then you have
> to add it to d-i.
>
> So, unless you live in some alternate universe where mathematics work
> differently, that is 2 actions for my proposal and 3 for the status quo.
Now:
- kernel team adds module
- D-I team adds it to kernel-wedge to one of the existing udebs
(- D-I automaticaly includes it since it is in an existing udeb)
2 manual decisions
Your proposal:
- kernel team adds module
- kernel team decides the module should be an udeb
- D-I team decides what image should include the new udeb
3 manual decisions
I must be living in an alternate universe.
...
>> > But i think we can agree that nobody regrets the current state of the kernel
>> > packages, nor of the single day uploads, which has brought debian out of the
>> > always-months-outdated kernel era into a more modern and responsive one.
>>
>> I do. It is a mess that nobody outside the kernel team understands.
>> The old way was way easier to grasp. Maybe not better but simpler.
>
> I hope you are joking ? Please ask the security team how they liked to do
> security uploads, uploading 15+ packages, including 4+ different upstream
> versions. Also, please go look at the pathetic calls for kernels sync for d-i
> builds and releases of april last year or so, and the month long processus
> this was.
The security team has inside knowledge into the code and doesn't do
much more than dumping an extra patch in there. They don't count as
"outside the kernel team" in that regard.
They also don't like the new code because it is simpler. They like it
because they have one source package with a single fix instead of one
per arch.
Please also don't forget that we currently can't add any new flavours
(even some sarge had) because build times for the is too long. Imagine
the impackt your proposal has when adding a module to the (then)
scsi-drivers.udeb meta package causes a new upload and full build of
the kernel.
>> Especialy the incompatibility with make-kpkg to build custom kernels
>> sucks. You can't build any kernel that requires an *-extra patch with
>> make-kpkg.
>
> Err, if this is so, file a bug report, and i am sure Manoj will fix any such
> issues. It is a design goal of both the kernel team and Manoj to maintain full
> compatibility, and we all invested over 6 month of effort to make it so.
>
> I guess you have no clue about what is going on here, and thus make uninformed
> and clueless assertions, please inform yourself.
I did file a report and the problem is the kernel teams patch and not
make-kpkg. Took an awfull long time to dig through the build process
with debug info to find out where the problems lies. The old packaging
that simply called make-kpkg could never had such a problem.
> Friendly,
>
> Sven Luther
MfG
Goswin
Reply to: