Re: Proposal - Defer discussion about SC and firmware until after the Etch release
Note: for purposes of discussion, I will use the term "firmware" to refer to
the binary executables loaded into various peripherals by the Linux kernel.
Properly speaking these are *not* firmware -- if they have to be loaded at
every boot, they're not firm, they're just software -- but that's another
matter, and since everyone is saying "firmware" and I don't have a better
term, I'll use it.
Frans Pop wrote:
<snip>
> I will start with an alternative GR proposal based on the one from aj.
It looks good.
<snip>
> On Tuesday 05 September 2006 09:44, Anthony Towns wrote in
> <[🔎] 20060905074404.GA8506@azure.humbug.org.au>:
>> It therefore seems to me as though we're going to be failing to meet the
>> social contract again, and as a consequence I think we should seriously
>> reconsider whether the change we made in 2004 was the right one. So I'd
>> like to propose the following course of action for consideration:
>
> I rather like the basic idea behind this proposal and I applaud the
> personal commitment the DPL is willing to make on this issue,
Yes, applause!
> however I
> feel it is wrong to do things in this order: we should not rush a change
> in the SC in order to get a release out on schedule only to have to
> discuss another change in the SC immediately after the release.
>
> I would suggest to not decide on a), b), c) and d) now but rather shelve
> those until after the release and instead do something like:
>
> The project acknowledges that a lot of progress has been made with regard
> to the removal from the distribution (main) of "software" that could be
> considered non-free given the current wording of the Social Contract.
You mean "...that is non-free according to the Social Contract."
I tire of hearing the completely invalid claim that the Social Contract
as written, now or before, can possibly allow non-free programs in main.
> However, in some cases for valid reasons, this work is not finished and
> requiring this to be finished before the release of Etch would result in a
> serious delay of the planned release.
>
> There are also indications that a significant group of people within the
> project feels that the current Social Contract does not meet the best
> interests of the project in that the current wording is too restrictive
> and
> that a limited and conditional inclusion/support of some types of
> "software" should be possible. Example: support for loading sourceless
> firmware during installation.
>
> The Debian Project resolves that:
>
> (a) The inclusion in main of sourceless firmware and support in Debian
> Installer is not a release blocker for the release of Etch.
>
> (b) For the release of Etch, the Release Managers are given discretion
> to waive RC issues in other cases where the letter of the Social
> Contract is currently not being met, provided there is no
> regression relative to the Sarge release
FYI:
This will require at a minimum (assuming undistributable firmware is allowed
in general) removal of lumps of hex from 13 files in the kernel. The
'new' ones are noted at ldoolit's page:
http://doolittle.icarus.com/~larry/fwinventory/2.6.17.html
If undistributable firmware is not allowed in etch, then this will require
the transfer of the tg3 and qla2xxx firmware to non-free; the other 'new'
ones are all non-distributable anyway.
Frankly, I've been waiting for the tg3 firmware to be (re-)removed from
Debian's kernel before I do any additional work, as a sign of "good faith".
> and that waivers are done
> consistently and with proper consideration of past resolutions
> (e.g. GDFL) and work already done on other (comparable) packages.
>
> (c) Following the release of etch, the Debian Project Leader shall:
> i. ensure that the Debian community has a good understanding
> of the technical and legal issues that prevent the Debian
> Free Software Guidelines from being applied to logos
"We can't get legal advice from a trademark lawyer, and some people think
that keeping logos non-free will somehow help make up for that."
That's basically it for logos. If someone pays for a trademark lawyer to
consult with Debian, that would be a great benefit all around.
> and
> firmware in a manner that meets the needs of our users;
> ii. ensure that project resources are made available to
> people working on addressing those issues;
To repeat:
Frankly, I've been waiting for the tg3 firmware to be (re-)removed from
Debian's kernel before I do any additional work, as a sign of "good faith".
> iii. keep the Debian community updated on progress achieved
> in these areas.
>
> (d) Following the release of etch, the Debian Project as a whole shall
> reopen the question of which commitments should be codified in the
> project's Social Contract. This shall include both an online
> consultation with Debian developers, users, Debian derivatives and
> the free software community, and a public in-person discussion at
> DebConf 7 in Edinburgh in honour of the 10th anniversary of the
> original publication of the Social Contract on the 4th of July
> 1997.
As long as it's recalled that supermajority requirements are needed to
change the SC, great!
And as long as it's recalled that the previous text clearly prohibited
non-free software, and that it, albeit very unclearly, prohibited non-free
non-software. People need to actually propose a *new* change which
will make it say what they *want* it to say rather than dragging Debian
further into the muck of unclarity. If the majority wants to allow certain
non-free materials in main, great, but I haven't seen a single proposal
which is really honest to Debian's users yet.
<snip>
> My rough summary:
> - (almost) everybody agrees that non-free drivers don't belong in main;
Yes.
> - (almost) everybody agrees that sourceless firmware at least needs to be
> distributable before any kind of support can be considered;
I think you're wrong here, unless you're using an unusual definition
of "distributable". The usual definition used by debian-legal is "We have
explicit legal permission to distribute it." If you were right, we wouldn't
have 46 undistributable files in Debian's Linux kernel packages today.
Should Debian release with those files (again)? This is a very, very
important question. Currently Debian is on track to release with 46
undistributable files.
> - most people agree that Etch should not be delayed for a solution to the
> sourceless firmware issue;
Yes.
> - a fair number of people (though a percentage is hard to estimate) seem
> to
> feel that the current Social Contract is too restrictive when it comes
> to some types of files, forms of documentation and sourceless firmware;
Certainly.
> - probably a larger number feels that we should not kill the project
As if anything suggested here would "kill the project". Hyperbole?
> by
> scaring away users with hardware that depends on sourceless firmware and
> is willing to consider solutions for that while still making the
> projects preference for firmware _with_ source clear to users and
> others.
Yeah, definitely.
> There is a weird mix of policy/politics and implementation in this
> discussion, especially by the people in favor of resolution of the
> firmware issue before Etch. IMO d-vote is _not_ the correct platform to
> discuss implementation, that should be done on either d-kernel/boot or
> d-devel, but there has been a lack of real discussion there (except for
> one short recent thread started by Wouter Verhelst on d-boot).
> Also, the proposed implementations are at the level of a rough idea
> without any details worked out or research done regarding feasibility.
Apparently not. See below for the reference to Godwin's message.
Implementations of firmware loading in the kernel are straightforward
at this point and require no discussion. They are all feasible, and
some are completed.
Details have not been worked out for d-i, but the implementation appears to
be 90% complete, and feasibility is definite. It's time for testing, to
find the "weak points".
> As indicated above, I also feel that the pressure of the Etch release is
> tempting people too hurry decisions without really discussing the issues
> and alternatives, which can only lead to poorly drafted ballots with a
> bias to one solution or another (as IMHO the past ballots on the SC change
> and the waiver for Sarge have been).
<snip>
> The position of the d-i team has so far been presented by Joey Hess and
> there have been some contributions from other d-i team members (Yoe,
> p2-mate, bubulle). Joey has given an overview of the impact of removing
> firmware from main on the installer with an estimate of required effort.
Which estimate is simply wrong. See below.
> The position of the kernel team in this is interesting. AFAIK the kernel
> team _is_ unanimous in the proposal to postpone the firmware issue until
> after Etch (which has consistently been Sven's position in the
> discussion). However, from discussions on IRC and private conversations
> with several kernel team members I know that the personal opinions of its
> members vary
> wildly. At least one is radically opposed to removing firmware from main
Hypothesizing, perhaps the one who reintroduced the tg3 firmware? :-/
> and a few others are much more open to further discussion or favor a much
> more liberal long term solution than proposed by Sven. IMO it is a real
> pity that not more kernel team members have spoken up in the discussion.
>
> I was appalled at the suggestion that surfaced in the discussion that the
> kernel or d-i team have actively refused to work on or delayed a solution
> for the firmware issue in time for Etch. The facts:
- Sourceless firmware was removed for the tg3 driver. It was *reintroduced*
by the kernel team.
Any wonder that I thought there was an active refusal to work on a solution?
Perhaps this was just a matter of stuff getting "lost" during the transition
to the kernel team, however.
- Joey Hess has claimed that it will take upwards of six months to ready the
installer for loading firmware from non-free. However, it turns out the
majority of the work which he listed as necessary has *already been
done* and requires *no changes*. See Goswin von Brederlow's message at
http://lists.debian.org/debian-devel/2006/09/msg00017.html .
Charitably, I would like to assume Joey Hess was just ignorant of
the status of d-i. (I didn't know either.)
Unfortunately, I am probably sensitized and am seeing obstructionism where
there is simple incompetence, because some of the *upstream* kernel
maintainers have been definitively and clearly obstructive, because there
was substantial obstructionism during attempts to remove non-free GFDL
material with invariant sections from Debian, because there was
obstructionism in response to the attempts to remove the non-free material
from the emacs packages, and so at this point I'm expecting it.
But everyone is incompetent sometimes (consider my efforts to find the d-i
repo), and it's more charitable to assume incompetence.
> - both the kernel and d-i team consist of volunteers, so the basic rule
> is:
> members are masters over their own time and effort;
> - in general there has been good coordination and cooperation between the
> kernel and d-i team;
> - the kernel team has done a major job on the integration of kernel source
> for all arches for 2.6 _and_ making the switch from initrd to initramfs
> while keeping support in kernel-package; a lot of time and effort has
> gone into this and as this was unavoidable to support upstream releases
> it took precedence over the firmware issue;
Yes, perhaps this caused them to accidentally *reintroduce* firmware which
had been removed. The regression still makes me suspicious though.
> - between Sarge and Etch there has been major development on the installer
> even though the number of active team members has shrunk somewhat,
> mainly because the installer had reached maturity with the Sarge
> release; a lot of the work had to be done to keep up with changes in the
> kernel and environment; other major projects: integration of base-config
> into the installer, graphical installer and partman-crypto (last two
> mostly by "new" members in the team);
> - implementation of a solution for firmware/non-free drivers in d-i has
> been discussed
And apparently 90% implemented.
> but consensus was that there was not much point in
> working on it while there was no separation in the kernel;
Which was being blocked. I find a strong disincentive to submit patches
when functional ones have been reverted for no good reason.
> - when that was done a change in the archives was requested to support
> loading udebs from contrib/non-free (and experimental); this was not
> picked up for a long time which led to loss of motivation and time;
Yes, that did lead to loss of motivation and time. Why did this relatively
simple change take so long? Lack of time on the part of the very
small number of people in a position to implement it, I suppose. Oh well,
it happens.
> - the person implementing the kernel part of the firmware solution happens
> to also be the person most suited to make some necessary changes in the
> installer;
Huh? Who you talkin' bout? There are multiple volunteers to implement
the kernel part of the solution. And it turns out there are few
necessary changes in the installer. :-P
> - the first separation in kernel packaging happened only recently; at that
> time it was really already too late for a solution in time for Etch in
> the installer.
More importantly, all but 12 of the non-free firmware files are also
nondistributable. This means that this entire discussion is a pointless
sideshow at the moment.
> VISION FOR THE FUTURE
> =====================
>
> My _personal_ preference at this point would be a solution as
> etch-a-sketched below. I have seen several people suggesting elements from
> this during the discussion.
>
> - Creation in the archive of a new, separate section ("restricted"?) for
> packages that can be used for new installations but are not suitable for
> main (alternatively they could be left in main but specially tagged in
> the control file). Main reason: you not only need such packages during
> installation, but also for the installed system; loading such packages
> from contrib/non-free would imply that these sections have to be added
> by default to the sources list for these users which is undesirable
> given the aims of the project [1].
> - Allow inclusion of these packages in installer images and on
> installation
> media.
> - During installation, present a message to the user every time such a
> package is actually used; user has to acknowledge.
> - Also add support for loading non-free/third party drivers into the
> installer, but _without_ including such drivers in installer images or
> on installation media.
Apparently already present; see Godwin's message linked above.
This is actually very important because:
- 46 files in the Linux kernel
*already* load their firmware from userspace, and the firmware is *not*
in most cases included in Debian or in the non-free archive. Perhaps
a recipe could be provided for users for each case, explaining how to get
the firmware and wrap it in a udeb.
- 46 additional files contain *nondistributable* firmware; the best scenario
for the users needing these (if relicensing can't be achieved) is to make
it easy to load them from an external source. Perhaps recipes could be
provided as above.
- only 12 files currently contain firmware which is non-free and legal to
distribute.
> These are invasive changes
Or are they? I argue that they are not.
> and require much more thought and careful
> implementation in kernel packaging, the installer and debian-cd; the
> required support in the archive will probably also not happen overnight
> ;-)
>
> I am strongly opposed to suggestions of creating a "non-free" version of
> the installer besides the regular "free" version. This would lead to an
> explosion of the number of images, a waste of bandwidth and mirror space,
> severely complicate building, testing and release management and will only
> confuse users who will probably only know they need the non-free version
> after a failed installation.
Agreed.
> [1] This is also why I object to MJ Ray's amendment in
> <[🔎] 20060905122545.1178DF6CBE@nail.towers.org.uk> as it codifies the solution
> without having checked its practical implications.
>
>
> STANDPOINT AS D-I RELEASE MANAGER
> =================================
>
> Frankly I am amazed that some people so easily dismiss Joey Hess' estimate
> of the effort required for proper implementation of support for firmware
> (and non-free drivers).
You should also dismiss his estimate. Read Goswin von Brederlow's message
linked above.
> There are probably only two people who come close
> to his level of overall understanding of the installer (no, I'm not one of
> them, I rank myself about fifth).
Apparently his understanding was *wrong* in substantial and clear ways.
See Godwin's message. Joey Hess thought things were
unimplemented which have been implemented for over five years.
I guess *nobody* understands the installer. That sucks.
I would love to see Joey Hess's estimate of the required effort *after*
Joey Hess actually checks what's already been implemented.
> The installer is a complex beast with loads of subtle dependencies, space
> issues and a tight interdependency with debian-cd. My experience with any
> structural change is that there is always unexpected breakage that only
> shows sometime _after_ the next release.
Luckily, there are no structural changes needed. See above.
> Besides that there are a load of issues that people fail to consider when
> proposing solutions:
> - the way non free kernel udebs are currently built (as by-product of the
> build of the deb) does not fit with d-i release management and the skew
> we habitually see near a d-i release in unstable between the kernel
> version the installer uses and the kernel images in unstable; it would
> be better to create the udebs using kernel-wedge;
This should never be an issue for firmware files which are dynamically
loaded.
Firmware versioning is essentially independent from the kernel versioning.
It also changes much more slowly. A non-free *firmware* udeb would contain
the loadable *firmware* file. Version skew between the kernel driver image
and the firmware image is just fine. (And would happen rarely; new driver
versions generally use the same old firmware.)
It would be an issue for entirely non-free drivers.
> - if firmware is used in the installer, the installer also has to make
> sure
> the corresponding regular package is installed for the installed system;
> so, as already indicated above, a solution needs to be found for both
> the installer _and_ the installed system, preferably without forcing the
> user to add _all_ of contrib/non-free in his sources list just because
> he/she needs one sourceless firmware file;
How sweet. I think it's just fine to force the user to add non-free to his
sources list if he needs one non-free file. What the hell is non-free for
anyway? Non-free shows up prominently as a separate section in aptitude
and dselect, so people aren't going to install non-free packages by
accident.
Howeveer, if you want to splinter non-free into multiple different non-free
archives (non-free-firmware, non-free-doc, non-free-other,
non-free-pumpkins), I really don't care at all. It's just naming, and it's
not deceptive in any way. It's just silly.
I think this is a matter of not wanting to admit that you have non-free
stuff on your system, while still having non-free stuff on your system.
I prefer to admit that I have non-free stuff on my system, although not
very much.
> - adding separate non-free installer images requires extra mirror space
> and
> changes in BYHAND processing and thus needs approval from FTP-masters;
This could be implemented entirely separately by any
volunteer. FTPmasters would be needed in order to give it an official
location on an official Debian server; period. If that is too hard for
FTPmasters to do, then we don't have enough FTPmasters.
The preferred design is to have an all-free installer and CD which has the
ability to load additional udebs from a separate CD, floppy,
or network source. If the CD drive, the floppy drive, all USB keys, and the
network card *all* need non-free firmware loaded in order to operate, then
the user is screwed, but frankly the user is pretty screwed in that
situation anyway, and I've never seen it happen. (Have you?)
The extra mirror space needed is tiny, amounting to the additional udebs,
not a duplicate of the entire installer. With the current licensing
situation, this is going to total 12 files (plus
control file stuff). It could expand in future if relicencing is
successful, of course.
> - adding separate non-free CD images would mean a substantial hit on the
> debian-cd (mirror) infrastructure and is currently probably not even an
> option.
As above.
> The current status of the installer is that, after the recent Beta 3
> release, it is basically frozen for new development (with a few exceptions
> agreed on before the release of Beta 3). There is also quite a long list
> of RC and minor issues, polishing and testing to be done and we still have
> the upgrade from 2.6.17 to 2.6.18 coming. I estimate we will have at least
> 3 RC releases in the run up-to 4 December.
> There is currently no support whatsoever for loading firmware in the
> installer
Or is there?
Consider those kernel drivers which already load their firmware dynamically.
Find one of those pieces of firmware. Put it in a udeb (under the correct
name, in /lib/firmware). Put the udeb in a network location with md5sum
etc., or on a floppy. Specify that location when asked for a source of
additional installer modules, and use that udeb. Do this before loading
the udeb containing the kernel module which loads the firmware. Or modify
the udeb to unload and reload the kernel module. (Or, in the cases of the
best-written drivers, send the 'reload the firmware' ioctl.)
> (as so far there has been no need for it).
>
> So, my formal position as D-I release manager has to be:
> "We will not accept any structural changes to support loading firmware in
> the installer (not from main nor from elsewhere),
> - if the release plan for Etch remains at 4 December 2006;
I guess "Debian releases when it is ready" has been dropped.
> - unless both Joey Hess and I, after careful review of a finished and
> tested proposed solution, decide that 1) it provides an acceptable
> solution for all installation methods and architectures,
If you require every possible installation method to work on every person's
system, you are setting a higher bar for the firmware-loading work than
is reasonable. This is unreasonable because on most systems *today* only
some of the installation methods work. Asking for the firmware support to
work *better* than the rest of the system is not reasonable.
If you instead use the reasonable standard that some installation method
works for almost everyone's system, and that it is clear from the
documentation which installation method to use, then it's possible to
make a good implementation quite quickly.
> 2) it poses no
> risk of regressions or delays in the run-up to the release."
In other words, no changes at all will be accepted. All changes pose
risk. Some pose very low risk, but all changes pose at least a minimal
risk. OK.
> The initiatives started during the discussion on d-vote to implement some
> kind of support are appreciated, but they are too late for Etch!
> In my personal opinion they are also premature given the controversy that
> exists over the question if firmware should really be treated as totally
> non-free or rather be supported to some extend.
Don't misrepresent the controversy, please. Too many people are already
doing that.
Of course sourceless MIPS binaries and the other stuff constituting
this "firmware" are totally non-free; nobody's ever made a serious argument
that it was free software. And it certainly should be supported, and
everyone agrees. Just like other non-free software people need is
supported. Namely, in the "non-free" section.
The controversies are multiple:
(1) Should Debian allow non-free material in main if it "doesn't really
matter" for some reason that it's non-free, or if it's technically very
desirable, or if it's not a traditional program, etc.?
There are strong
arguments for each of these; however, nobody arguing for this has yet been
willing to propose an amendment the Social Contract to clearly say what they
want it to. (It is *extremely* hard to write a clear amendment which allows
non-free firmware and doesn't allow other non-free software, because there
is no really clear distinction between a CPU on a peripheral and
a "main" CPU. But if anyone manages to, it would be a good thing.)
(2) Does it really matter that the firmware is non-free? (Many people argue
that it doesn't, and they have some strong arguments, though they didn't
convince me.)
> Thanks for hearing me out,
> FJP
You're welcome. Thanks for listening to what I wrote: hopefully it will
leave you with more understanding of the situation.
*My* personal preference?
-------------------------
(1) Nondistributable material -- mostly "GPL without source" -- removed from
the kernel packages and installer entirely, until it can be relicensed.
Yes, this will hurt users. (The alternative is to come up with a coherent
policy of trying to guess the intent of licensors rather than reading their
actual licenses. It would be unfair to allow mislicensed material for the
Linux kernel but not for other packages.)
(2) tg3 firmware removed from kernel packaging, replaced with userland
firmware loading and a firmware package in non-free. Tested and
functional, y'know.
(3) Testing done on loading udebs from an "extra" source, using the
udebs from firmware-nonfree.
In order to force firmware loading, the firmware udebs may need to rmmod and
modprobe the relevant driver, since many drivers can't handle firmware
loading at any time other than module loading, and udev may try to load
drivers early.
This isn't particularly pretty, but neither is non-free firmware. It will,
however, work.
If your CD-ROM needs non-free firmware to run, you need to netboot or boot
from floppy or hard drive or USB stick. If your network card needs non-free
firmware to run, you need to boot from CD or floppy or hard drive or USB
stick. If your hard drive needs non-free firmware to run, you need to
netboot or boot from CD or floppy or USB stick. If your USB stick needs
non-free firmware to run, you need to netboot or boot from CD or floppy or
hard drive. If your floppy drive needs non-free firmware to run (really
only possible if it's connected to, say, a SCSI card which needs such
firmware), then you need to netboot or boot from CD or hard drive or USB
stick.
If your SCSI card needs non-free firmware to run and your hard drive and CD
and network card and floppy are all connected to it, you have a problem,
and should be told to build your own custom installer CD. Or buy a better
SCSI card.
The two most common drivers which feature firmware loading are tg3 and e100.
In both cases, the drivers do not require the firmware loading for the vast
majority of actual cards on the market, so the above bootstrap problems will
not come up for most tg3 or e100 users.
(4) The above three can be done simultaneously. And started immediately.
After they're done, work on moving the remaining 11 non-free distributable
files moved to non-free and converting their drivers to firmware loading.
(5) Long-term attempts to contact companies and get them to relicense their
firmware so it's distributable. Having actually removed the
nondistributable stuff might make it easier to get them to listen.
Please feel free to forward this stuff to more appropriate lists. I really
don't know what's most appropriate, so I'm responding where I read it.
--
Nathanael Nerode <neroden@fastmail.fm>
Bush admitted to violating FISA and said he was proud of it.
So why isn't he in prison yet?...
Reply to: