Hi,
Cyril Brulebois <kibi@debian.org> (2023-01-24):
> I'm also not sure what our plan for e.g. ARM images (concatenateable
> images) would be; I don't think we'll pull any firmware packages during a
> d-i build, so they wouldn't end up under [2], but maybe it'd be feasible
> to just concatenate some file/image containing non-free-firmware packages
> (along with the associated metadata if relevant) as published by debian-cd
> or something? Or would those concatenateable images need to load firmware
> from external media anyway?
>
> 2. https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/
>
> In any case, we could probably try and accommodate architecture-specific
> issues… by wrapping required calls inside proper conditions, so that
> special needs (on release archs or non-release archs) don't negatively
> impact regular users.
>
>
> In the meanwhile, I'll disable both mountmedia calls in the upcoming
> upload:
> - to reduce delays when official installation images are just
> self-sufficient (that's why we had that GR in the first place!);
> - to gather feedback/complaints from users for which those calls are
> actually important, so that we can better assess the needs, and the
> possible shortcomings in the historical implementation.
I've thought about this a little more, with the following in mind:
netboot in general and netboot SD card images in particular.
# netboot
Netboot images are built by debian-installer, and d-i builds will
certainly not start pulling things from outside main; that being said,
it's been well known that CPIO archives can be concatenated (see
https://wiki.debian.org/DebianInstaller/NetbootFirmware), so it's
possible to take the firmware directory produced by a debian-cd build
(as found on netinst images), and append a CPIO archive of that
directory to a netboot or netboot-gtk initrd. Tested by using direct
kernel loading in virt-manager, stitching things together this way:
# pick firmware/ as prepared by debian-cd (e.g. tmp/sid/CD1):
pax -L -x sv4cpio -w firmware | pigz -9c >firmware.cpio.gz
# append to d-i's initrd.gz:
cat initrd.gz firmware.cpio.gz > initrd+firmware.gz
Then boot with:
- Kernel path: netboot/gtk/debian-installer/amd64/linux
- Initrd path: netboot/gtk/debian-installer/amd64/initrd+firmware.gz
- Kernel args: vga=791 (to ensure the framebuffer gets enabled)
Beware, the initial initrd.gz is 83M, the firmware.cpio.gz is 186M! That
means I needed to bump my very usual VM's RAM rom 1G to 1.5G, maybe even
2G (depending on which d-i features get used). Things should be less crazy
once we limit debian-cd builds to picking firmware packages from one
section (non-free-firmware only, instead of contrib + non-free + n-f-f),
but that's still an important size bump.
All that ensures a /firmware is found by hw-detect, and firmware can be
detected/deployed/installed as needed, just like in the netinst case.
It shouldn't be a problem to provide a pre-built firmware.cpio.gz
archive with the relevant metadata (produced by debian-cd) in the dep11
subdirectory, for use by hw-detect. We should probably start publishing
them in a less hidden location than the existing:
https://cdimage.debian.org/cdimage/unofficial/non-free/firmware/bookworm/20230123/
Maybe adding a “non-free-firmware” directory directly under this
location?
https://cdimage.debian.org/cdimage/release/current/
After all, each arch-specific directory will have non-free-firmware bits
included, so having an extra directory in the release directory shouldn't
be an issue?
And we might clean things up in:
https://wiki.debian.org/DebianInstaller/NetbootFirmware
Maybe strongly recommend starting from the upcoming new firmware.cpio.gz
file, and only keep the relevant bits (this and that firmware package,
and relevant metadata), to keep the overhead low?
# SD card images (might also be applicable to the upcoming ChromeOS images)
Looking at how those images are built, it seems we have a predefined,
per-arch image size (150M or 200M), and gen-hd-image that's called in two
modes: firmware or partition. The former takes care of the board-specific
part (e.g. firmware.a64-olinuxino.img.gz) while the latter takes care of
the board-independent part (partition.img.gz). Both can be concatenated
with the following command as documented in the README:
https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/SD-card-images/README.concatenateable_images
zcat firmware.<board_name>.img.gz partition.img.gz > complete_image.img
And that works, generating an image with a single fat16, fat32, or ext2
partition with contents merged together, because of the pre-defined image
size.
We could think about bumping the image size, but given the size of the
firmware directory (as in containing the firmware-* packages and metadata,
not the board-specific firmware files), this would seem like a waste of
space for all those not needing firmware files (probably requiring twice
the current size to make sure everything fits)…
I'm not sure how often boards installed via SD card images require
firmware packages, and how many of them, maybe it would be feasible to
include instructions or even a script that only extracts this and that
firmware package (and metadata) from the upcoming new firmware.cpio.gz, so
that it can be appended after partition.img.gz, provided there's enough
space within the defined image size?
But I think it would require at least detecting the underlying partition
type, determining the right numbers, and calling either mcopy or genext2fs
with the right parameters…
https://sources.debian.org/src/debian-installer/20210731%2Bdeb11u7/build/util/gen-hd-image/#L348-L367
All of this assuming that the end results can be appended as the third
part of the <board firmware> + <partition> + <firmware> combination!
(Anticipated caveat: we would likely need to tweak filenames for some FAT
variants, I haven't restricted myself to the good old 8.3 format.)
# Summary
For regular netboot images, existing firmware addition should keep
working, even if we should produce a slightly different firmware CPIO
archive (adding metadata), publish it into a more discoverable place, and
update instructions accordingly. For those *not* adding firmware stuff, it
might make sense to keep mountmedia calls; an easy way to find out when
those might make sense would be to check if a (/cdrom)/firmware directory
exists at all (it would on netinst, wouldn't on netboot, would on netboot
augmented with firmware stuff).
For netboot SD card images, we are space-constrained, so *maybe* adding a
few selected firmware packages (and metadata for automated use by
hw-detected) could work, but I suppose the general case might end up users
don't add anything. Especially since that's not just about concatenating
compressed initrds together, we would need some more elaborate glue…
All this reinforces my impression regarding adding (/cdrom)/firmware
detection. Since we're talking about an installation image with a single
partition, maybe those mountmedia calls work more reliably when there's
external storage plugged in anyway? Again, I don't know much about how
mountmedia works, or about problems/limitations users might have
encountered.
I might prepare the upcoming D-I release without mountmedia at all though,
that's probably the best way to get feedback about it.
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Attachment:
signature.asc
Description: PGP signature