TL;DR: No progress on the social side (doc) yet, but lots of progress on
the technical side (making firmware-enabled images work better)!
Kernel team, feel free to skip to @KT for a little question.
I'd like to thank Ben again for the modalias thing, this made it very
easy to deploy firmware packages!
Cyril Brulebois <kibi@debian.org> (2021-07-24):
> # main-only images
>
> ## isenkram-based proof-of-concept
[…]
> - The issues I fixed in isenkram-cli were around some fallback code
> path, and the AppStream maintainer suggested there might have been
> some issues in AppStream causing the primary code path to fail:
> https://bugs.debian.org/989884#24
> The relevant update was accepted into testing since then, but I
> haven't had a chance to check that hypothesis. It would be great to
> know whether we actually have two working lookup implementation and
> not just the fallback one. (This could mean more reliability during
> a release cycle, in case one of them breaks — again.)
I've tried downgrading isenkram-cli to the version before my fixes
regarding the fallback code path, and it didn't find anything, so either
the primary code path or appstream is still broken. I've made a note to
possibly investigate this further, but this definitely doesn't block the
release.
> ## proposed solution
>
> If others agree, I suggest we add the following somewhere in the
> installation guide (I don't think we should have a specific warning in
> the installer; or more specifically it would be way too late to have it
> be shown localized at this stage; more on that in the second part of my
> mail). Heavily summarized (to give readers of this bug report an idea,
> not for direct user consumption!!!): […]
Given Paul's ACK, I'll turn this into a bug report against the
installation guide, with a slightly better draft.
> ----------------------------------------------------------------------
>
> # firmware-enabled images
>
> ## easy, but ugly fix
Simon's solution makes the fallback plan even more straightforward. I
don't think we'll need it though. Read on!
> ## more targeted approach: modalias information
>
> Ben suggested augmenting AppStream metadata (see DEP-11) of
> firmware-nonfree binaries with modalias information, so that it would
> be feasible to establish some hardware → firmware mapping. It's not been
> included in the archive yet, see MR#19 for src:firmware-nonfree:
> https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/19
This was uploaded to unstable, I'm currently waiting for the updated
metadata to show up in the archive.
Meanwhile, I've written a new tool for debian-cd, which Steve has merged
and refined further to match other choices made in that area (something
like: due to size constraints, we don't ship all firmware packages), so
I'm confident either of us should be able to refine this further once we
actually have metadata in place. That happened in debian-cd master, and
was uploaded as debian-cd 3.1.35:
https://salsa.debian.org/images-team/debian-cd/-/blob/master/tools/generate_firmware_patterns
Basically, we extract the relevant bits from the DEP-11 YAML file, and
generate patterns files that we ship on the firmware-enabled images:
- /firmware has firmware-*.deb
- /firmware/dep-11 has firmare-*.patterns
Meanwhile, I've hacked d-i to include those files, built manually while
the rest was being worked on, so that I could test the runtime on my
test laptops.
> This doesn't help us (yet) regarding Intel SOF, since that package is
> (ultimately) built from src:firmware-sof, which doesn't have this
> logic yet. But I don't think we should delay the release for that (it
> could be listed in errata, and/or fixed in 11.1, whenever the package
> is ready), and if the machine at least boots fine, including the
> graphics stack, the isenkram trick above should let users install the
> remaining firmware packages on their own.
The isenkram-based fallback (once we have the actual doc) is still a
valid way to ensure this package can get installed from within the
installed system, but I've also included the relevant IDs as a special
case on the debian-cd side, so that we don't have to worry about that
one (it seems to show up quite often on laptops, so…):
https://salsa.debian.org/images-team/debian-cd/-/blob/master/tools/generate_firmware_patterns#L135-160
> There might be other “widely spread” firmware that might benefit from
> the same logic, but I won't be spending too much time thinking about
> this right now; such patches can come later, and those updated package
> firmwares should be pickled up automatically afterwards.
>
> Regarding the implementation, my current approach would be:
> - have firmware-nonfree's MR merged and uploaded;
DONE.
> - make sure the relevant information bubbles up to the dep11/Components
> files (YAML format, even if the excerpts above were XML);
TODO.
> - have debian-cd extract the relevant firmware package←→modalias
> mapping from YAML into one or N flat files to be consumed by d-i, for
> firmware-enabled images primarily (or exclusively);
DONE, to be checked once the previous item is complete, and some fixups
might be required until a full image build behaves like my heavily
hacked netboot-gtk images (meaning purely from debian-installer.git, no
debian-cd involved).
> - patch d-i to consume that or those files to queue up as many firmware
> packages as required.
DONE.
I've patched hw-detect to use that information in addition to the usual
grepping of dmesg to find missing firmware, and reused the logic that
was there already to enable non-free and contrib (in that order, which I
didn't think it was reasonable to fix at this stage of the freeze[1]),
and install firmware packages inside /target (the installed system, as
opposed to the installer's context).
1. https://salsa.debian.org/installer-team/apt-setup/-/commit/ba70d27d150c5fd869653082ee9180410346436b
This seems to work great on both test laptops so far, and after
installation, isenkram-autoinstall-firmware basically does nothing,
since d-i took care of approximately everything.
I have still a slight discrepancy for some BT firmware files that aren't
seen in dmesg but that are apparently documented in modinfo, which
isenkram spots, leading to an extra firmware-iwlwifi on one of my test
laptops. I haven't tested full desktop installations, so I'm not sure
whether that's actually needed, but that seems pretty minor at this
point. If users are actually wondering about that, they can still follow
the isenkram-based fallback (once we have the actual doc). Therefore,
I'm considering this very minor, keeping a note about it, but I won't
look into it before 11.0.
I've also fixed and tested #973733 since one of those laptops was
playing tricks regarding actual module names, making the reloading of
modules after firmware installation… insufficient. Plus fixed a few
other things so that logs are somewhat cleaner:
https://salsa.debian.org/installer-team/hw-detect/-/commits/master
@KT:
I haven't uploaded it since I'm seeing iwlwifi requesting what appears
to be a debug-only firmware (iwl-debug-yoyo.bin), that's even requested
via an aptly-named firmware_request_nowarn() function in linux.git; I'd
like to check with the kernel maintainers whether it's expected for this
message to show up in dmesg still (given the name and comments in
linux.git, I'm not sure it should). In any cases, it's slightly sad to
pause and prompt users from firmware files we don't ship anywhere
(AFAICT), that they don't actually need.
Therefore, I'm tempted to blacklist this firmware file in hw-detect
(i.e. pretend it was never requested).
> Note: it's likely that at least some firmware packages are going to be
> installed rather early in the installation process; network drivers
> (wired and/or wireless ones) are likely going to be probed early, and
> have generated the relevant messages in dmesg.
That's exactly what happens, we have:
- check-missing-firmware: dealing with hot-deploying firmware files and
reloading modules.
- install-firmware: dealing with the installation of firmware packages
into /target (carrying over those seen by check-missing-firmware,
plus the new bunch thanks to modalias information).
> I have still to familiarize myself with the whole (existing)
> “automatic installation of firmware packages when running on a
> firmware-enabled image” logic though, to see where the patches would
> be best suited.
All done.
As a bonus, I've (re)discovered that netcfg doesn't propagate WPA
settings to the installed system when NM isn't installed there.
Considering this not-a-new-problem©®™, and that shouldn't affect end
users too much given they're likely going to install a desktop
environment (others might know about wpa_supplicant); I'd like to
investigate this at some later point, and hopefully get a fix merged
into 11.1, and 10.n if I'm successful.
> ----------------------------------------------------------------------
>
> # final thought
>
> Circling back to the main-only images, having that hardware→firmware
> lookup would let us have a chance to mention that “this and that and
> that firmware package might be needed on your system”, adding a pointer
> to the “isenkram procedure” of the installation guide. If that'd be
> desired, the relevant mapping file would need to end up not only in
> firmware-enabled images, and we would need new strings, new
> translations, and that's almost certainly only achievable during the
> next release cycle.
>
> In any cases, I'm happy to start with something specific to the
> firmware-enabled images anyway, so that the possible fallouts are
> contained within a rather known and well-defined area. We should have
> enough time to implement that for 11.0, but we don't have *plenty*.
We went exactly that route with Steve: changing things under /firmware
only, so that there can't (or really shouldn't) be any fallouts on
main-only images. The rest can wait until bookworm (if we even want to
implement such a thing).
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Attachment:
signature.asc
Description: PGP signature