Hi,
I thought I'd file two separate bug reports with details of test
installations (with main-only images, and with firmware-enabled images),
on two test machines (one with Radeon, one with Nvidia), but after
having collected information from a single test installation, I fear
this is going to be a waste of time for no real added value. Therefore,
I'll just reply inline with my current gut feelings.
This is an overlong mail, apologies for that, but both topics are
somewhat connected, so…
Release team, you can search for @RT to jump at partial conclusions.
Cyril Brulebois <kibi@debian.org> (2021-06-14):
> This is an umbrella bug report that I intend to use to track at least
> two separate issues (that are really the two sides of the same coin):
# main-only images
> - Official installation images (main-only) might lead to a successful
> installation but result in a black screen upon rebooting the
> installed system. Some of these issues are due to free drivers
> wanting or even requiring some firmware to work properly, on a
> per-device basis.
## isenkram-based proof-of-concept
I've investigated isenkram as a possible thing to document for end-users
to try whenever they have firmware-related issues. Let me say upfront
that this requires being able to log into the installed system: it
relies on having the “missing firmware” messages in dmesg, and automates
looking up the relevant firmware packages, enabling contrib/non-free,
and installing the firmware packages that were detected as needed.
A couple of comments:
- When I first looked into it, I must say I wasn't exactly impressed by
the state of the isenkram-autoinstall-firmware script in the
isenkram-cli package (#989884); it was easy enough to fix the
multiple issues in there to make it functional again in bullseye
though.
- The initial reaction to #989884 made me doubt whether we should be
relying on this component, at least for a moment: While I understand
the maintainer's point of view regarding what the package's main
goals are, it felt like our d-i needs were handwaved away somehow.
Practically though, the patches were merged in a timely manner, and
as long as we can rely on that for the duration of a stable release,
we might have a sufficiently good answer for people looking to
complete the Debian installation.
- The issues I fixed in isenkram-cli were around some fallback code
path, and the AppStream maintainer suggested there might have been
some issues in AppStream causing the primary code path to fail:
https://bugs.debian.org/989884#24
The relevant update was accepted into testing since then, but I
haven't had a chance to check that hypothesis. It would be great to
know whether we actually have two working lookup implementation and
not just the fallback one. (This could mean more reliability during
a release cycle, in case one of them breaks — again.)
## unability to reproduce black screens with test hardware
While the two test laptops (that were sent to me for debugging purposes)
were picked to come with two possibly problematic video cards, I haven't
been able to produce the black screen I was trying to get: the Radeon
one manages to work with llvmpipe out of the box (switching to RENOIR
once firmware is installed + reboot has happened), the Nvidia one has an
Intel GPU anyway.
I've tested the nomodeset kernel command line option though, which gets
me a working graphical environment as well with different settings, and
I *think* the system was using fallback, generic drivers (think VESA,
FBDEV, and the like), but I couldn't 100% confirm this by checking X
logs (which is slightly ironic for a former X maintainer but oh well).
I also *think* using this option would help people dodge the black
screen they can get with some other hardware, so that they can follow
the isenkram-based procedure; I haven't been able to confirm that myself
yet. It might make sense to have a call for help/feedback here, along
those lines: “after installing Debian with a graphical environment, if
you get a black screen, does passing `nomodeset` on the kernel command
line help get the system boot to a graphical environment? [y/n]”. We
might ask for more details like the actual hardware they're testing
with, to see where the workaround is effective (and where it's needed).
I don't view this call for help/feedback as a blocker for a d-i release,
or for 11.0; we can always refine documentation if that hint is not
sufficient. And hopefully, people assisting end users will suggest using
firmware-enabled images anyway (read more about that below).
## proposed solution
If others agree, I suggest we add the following somewhere in the
installation guide (I don't think we should have a specific warning in
the installer; or more specifically it would be way too late to have it
be shown localized at this stage; more on that in the second part of my
mail). Heavily summarized (to give readers of this bug report an idea,
not for direct user consumption!!!):
1. Sometimes, bad luck, your system requires some firmware packages,
which might not have been detected during the installation.
Furthermore, they are usually in the contrib or non-free sections
(insert big fat warning), and wouldn't be installed automatically.
From an installed system, the following instructions would look for
missing firmware files, find the relevant packages, and install
them, automatically enabling contrib and/or non-free as required:
sudo apt install isenkram-cli
sudo isenkram-autoinstall-firmware
Then reboot the system.
[Whether we document other package managers, how to open a terminal
emulator, etc. is left as an exercise for the documentation writers.
Ditto for the practical consequences, meaning an extra sources.list
file, which would be some implementation-specific details that could
change over time. At the moment though, that'd be:
/etc/apt/sources.list.d/isenkram-autoinstall-firmware.list;
it also uses deb.debian.org — hardcoded]
2. Sometimes, bad luck, you might get a black screen when booting the
installed system. Hopefully that can be worked around by adding the
`nomodeset` option on the Linux kernel command line [maybe we
already have some places where we document how to do that in GRUB].
If that trick works, you should be able to execute the instructions
mentioned above from the system booted in a “failsafe mode”.
@RT: I'd be happy to have some feedback from your regarding this
approach; telling people to enable contrib/non-free so that they
can install firmware packages is definitely *not* something I take
lightly, but I'd be happy to have some kind of review there to see
if that looks reasonable at all. I'm not sure there are many
options these days anyway…
Bonus point with this approach: we don't need to touch the installer, we
can merge translations to the installation guide as they flow in, and
the website should have an uptodate documentation as long as the
debian-www sync works fine (I don't recall exactly how that works, I
think we have various, per-release directories on the website; we might
need to get the source package accepted in the relevant suite though,
but we know how to deal with stable-proposed-updates, so…).
----------------------------------------------------------------------
# firmware-enabled images
> - Unofficial installation images (shipping firmware from non-free)
> might also lead to a successful installation but result in the same
> kind of issue due to the way firmware detection is happening.
## easy, but ugly fix
A fallback plan would be to just install all firmware packages. That
would be *nasty*, would lead to some fun for at least one of them which
requires accepting some EULA; from my IRC ramblings, those might be
problematic and might need blacklisting or special handling:
- firmware-ipw2x00 (EULA, IIRC; didn't recheck)
- firmware-ivtv (EULA, IIRC; didn't recheck)
- firmware-microbit-micropython* (conflicts with the doc package)
- firmware-b43-installer (only a downloader)
- firmware-b43legacy-installer (only a downloader)
If we were to go the “(almost) all in” route, it might make sense to
either blacklist (some of) those, or to whitelist the known-ok ones,
which would require some monitoring of new additions over time (which
dillon, behind d-i.debian.org, could help us with).
@RT: This is my worst case scenario approach, letting us ship an
installer (for firmware-related images) that have fewer chances of
letting users get a black screen.
## more targeted approach: modalias information
As a reminder, the missing firmware detection logic in the installer
uses dmesg to detect specifically-formatted messages mentioning firmware
files; this doesn't work for a variety of components, at least:
- video drivers: we use fbdev and don't try to get all the fancy DRM
things up and running → we cannot know radeon, nouveau, or whatever
are going to require this or that firmware file in the installed
system;
- sound driver: firmware-sof-signed might be required to get sound up
and running on some Intel boards.
Ben suggested augmenting AppStream metadata (see DEP-11) of
firmware-nonfree binaries with modalias information, so that it would
be feasible to establish some hardware → firmware mapping. It's not been
included in the archive yet, see MR#19 for src:firmware-nonfree:
https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/19
Specific snippets can be checked at:
https://people.debian.org/~benh/firmware-metainfo/
I have verified that crossing those modalias entries with info extracted
from udev would let us spot that this or that specific firmware package
would be interesting (even if I don't think we can ever be sure how
*required* a given firmware is), instead of installing them all.
In particular, I have verified that a netboot-gtk build of the
installer, booted under FBDEV as usual, does report the relevant AMD or
Nvidia card via `udevadm info -e`, and that it makes it possible to
decide installing firmware-amd-graphics (AMD) or firmware-misc-nonfree
(Nvidia).
This doesn't help us (yet) regarding Intel SOF, since that package is
(ultimately) built from src:firmware-sof, which doesn't have this logic
yet. But I don't think we should delay the release for that (it could be
listed in errata, and/or fixed in 11.1, whenever the package is ready),
and if the machine at least boots fine, including the graphics stack,
the isenkram trick above should let users install the remaining firmware
packages on their own.
There might be other “widely spread” firmware that might benefit from
the same logic, but I won't be spending too much time thinking about
this right now; such patches can come later, and those updated package
firmwares should be pickled up automatically afterwards.
Regarding the implementation, my current approach would be:
- have firmware-nonfree's MR merged and uploaded;
- make sure the relevant information bubbles up to the dep11/Components
files (YAML format, even if the excerpts above were XML);
- have debian-cd extract the relevant firmware package←→modalias
mapping from YAML into one or N flat files to be consumed by d-i, for
firmware-enabled images primarily (or exclusively);
- patch d-i to consume that or those files to queue up as many firmware
packages as required.
Note: it's likely that at least some firmware packages are going to be
installed rather early in the installation process; network drivers
(wired and/or wireless ones) are likely going to be probed early, and
have generated the relevant messages in dmesg.
I have still to familiarize myself with the whole (existing) “automatic
installation of firmware packages when running on a firmware-enabled
image” logic though, to see where the patches would be best suited.
Still a gut feeling at this stage: a finish-install script, running
before the final update-initramfs one, might be the best place to do
that final check; it would build the list of interesting firmware
packages, and either skip those which are installed already, and install
the others; or just apt install the whole list, and let apt be a no-op
for all those already installed.
@RT: This would be my preferred approach, and based on cdbuilder coming
back up, plus my recent verifications in a d-i context, I think
this should be manageable in time for the announced release date.
Ideally we would be able to release an RC3 a week from now, letting
us fix any boo-boo in time before mid-August.
# final thought
Circling back to the main-only images, having that hardware→firmware
lookup would let us have a chance to mention that “this and that and
that firmware package might be needed on your system”, adding a pointer
to the “isenkram procedure” of the installation guide. If that'd be
desired, the relevant mapping file would need to end up not only in
firmware-enabled images, and we would need new strings, new
translations, and that's almost certainly only achievable during the
next release cycle.
In any cases, I'm happy to start with something specific to the
firmware-enabled images anyway, so that the possible fallouts are
contained within a rather known and well-defined area. We should have
enough time to implement that for 11.0, but we don't have *plenty*.
Thanks for your attention! I've had plenty on my mind, and some
braindump was in order. Sorry it's been *that* long a read.
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Attachment:
signature.asc
Description: PGP signature