Hi,
It's been a while since my first proof of concept (demonstrated at the
2014 mini-DebConf in Paris), but I think I've managed to reach a point
where I'm rather content and ready to get bits and pieces merged where
they belong. That doesn't mean I have everything figured out, that's
why I'm reaching out to both the installer & images team!
The primary question we need to discuss is what we are aiming for. I'll
start with a (not so) little flashback to explain what I've worked on,
why, and what is likely (not) going to be our limited target.
This is a long read… I've highlighted “to do”-like items with an arrow
(=>).
debian-installer
================
First things first, we needed a debian-installer build that knew about a
backported linux kernel, so I hacked my way in debian-installer.git; it
wasn't too hard since we can count on apt to resolve dependencies, and
after all, we only needed the kernel-image udeb and its modules.
The latest branch is stretch-backports-v2, which combines both a number
of commits that are aimed at the master branch, and a couple others that
want to only be used in a stretch-backports branch:
https://salsa.debian.org/installer-team/debian-installer/commits/stretch-backports-v2
net-retriever
-------------
As usual, I've tested the resulting installer with the netboot(-gtk)
image and of course it failed to load additional udebs. These netboot*
images are aimed at booting from the network, and they also need to
fetch additional udebs from a mirror. Since net-retriever only knows
about a single source of udebs, it needed to be told about multiple
sources (stretch and stretch-backports) and also how to merge entries.
I've detailed my choices in this commit in particular:
https://salsa.debian.org/installer-team/net-retriever/commit/09535b0613a85e02d511b92b60683b20405b6d42
For more details, see the whole stretch-backports-v2 branch for
net-retriever:
https://salsa.debian.org/installer-team/net-retriever/commits/stretch-backports-v2
Now, it doesn't matter too much, since the stretch-backports repository
will evolve over time, and udebs for the specific kernel used during the
d-i build will disappear. I think this is too much of a volatile (no
pun intended) target to support. It's still a very nice way to test d-i
without having to build an ISO with debian-cd, so I'd like to merge
these changes in net-retriever anyway.
base-installer
--------------
Second, it makes quite some sense to not only run d-i with a newer
kernel, but to also install it, so that users can boot from their brand
new hardware that wasn't supported by d-i with stable components only.
The kernel selection and installation is implemented in base-installer
(see library.sh), and that happens to run before apt-setup. Since I
didn't want to butcher sources.list manually to install the kernel
directly at this stage, I've decided to implement installing the latest
kernel from backports in a finish-install script (also shipped in the
base-installer package, for consistency), running at the very end of the
installation process.
This only happens when backports support was enabled (this is detected
through the presence of /etc/udebs-backports-source).
At first, using the linux-image-$arch metapackage looked easy enough but
of course that works for amd64, but not for i386 (one would need to pick
686 or 686-pae). I've still pushed a branch with this implementation as
it can be useful as is:
https://salsa.debian.org/installer-team/base-installer/commits/stretch-backports-v0
We could probably modify library.sh to reuse it, but I'd like to have
minimal changes (for reasons explained in the very last section). We
could probably just iterate over all installed linux-image-'*' packages,
pick the one from the src:linux package, and upgrade it from the
backports repository.
=> I'll look into that later on.
apt-setup
---------
Also only when /etc/udebs-backports-source is present, the backports
service gets automatically enabled, which makes the later installation
of linux-image packages possible.
Single commit for this:
https://salsa.debian.org/installer-team/apt-setup/commit/6133bdcd5a7903105ad967ab087f2f12f9d1c59d
debian-cd
=========
As mentioned above, tricks were needed in net-retriever for the netboot*
images, to make it possible to load linux kernel modules udebs. Those
tricks aren't needed when generating an ISO image. As a reminder, on the
d-i side, a cdrom_isolinux image is generated, and the debian-cd tool
needs to be configured to use that image instead of the official image
available on the configured mirror.
I didn't make this dynamic yet, but I've modified my local configuration
to set BACKPORTS=backports-list, to include a number of packages from
the stretch-backports suite. This file was generating by listing all
udebs produced by the linux source package (as done in the d-i build
system):
USE_UDEBS_BACKPORTS_FROM=stretch-backports
...
source=linux
...
binaries=$(grep-dctrl -s Package -F Source $source $APTDIR/state/lists/*${USE_UDEBS_BACKPORTS_FROM}*Packages | awk '{print $2}')
=> This needs to become dynamic, probably through a helper tool that
can be called automatically if backports are enabled in debian-cd.
=> It might be nice to have some kind of consistency check, at least
to make sure the ABI is the same for the linux kernel produced by
the d-i build, and for the modules available in the backports
suite. [Note: this might be true for weekly builds too, see recent
complaints/bug reports.] Bonus points if the source version is
checked too, for extra caution.
As a consequence, all those were included in the ISO, in the following
file:
debian/dists/stretch/main/debian-installer/binary-amd64/Packages.gz
This means no extra tweak is needed there, all udebs are available in a
single place: easy!
Of course we still need the modified apt-setup and base-installer
binaries. I've made them available as local packages (LOCALDEBS) and
they were automatically used.
The finish-install script ran as expected, but I ended up with a 4.16
kernel from backports in the installed system, as the linux-latest
binaries still point to it rather than to 4.17 (presumably because
linux/stretch-backports is still Needs-Build on mipsel at the moment).
It was downloaded from the network mirror.
=> It would be nice if we could include the linux-image-* packages from
backports (and their dependencies) directly on the installation image,
so as to be independent of what's happening in the stretch-backports
suite on online mirrors. Users would have reproducible results over
time, instead of a jumpy target. I don't know debian-cd enough to
determine how feasible it would be. I'm also not sure what would happen
if we had “old” linux-image packages on the ISO, and “newer” ones on
mirrors.
Open questions
==============
Now that I've explained (in length…) how it works, I think it's high
time we define what our target is.
Due to the volatility of a backports suite, I would tend to not even try
to support netboot. Aiming for a least a netinst image would look good
to me. If we can manage that, we can probably also produce a CD1 for
those who want to have more packages than just what's on a netinst (I've
had at least one such request). I'm not going to debate how many
desktops we should have CD1 for, that's really not my call. I'll just
mention that size constraints might be tricky since we'll have both the
linux-image packages for the base suite and those from the backports
suite. The whole DVD/BR thing is probably entirely overkill.
=> Choice to make: netinst and maybe CD1(s)? Something else?
It's probably a good idea to consider building matching unofficial
image(s) with firmwares embedded. I'm not an expert regarding debian-cd
so help is much welcome. Tweaks might be needed to get firmwares
installed and/or upgraded from the backports repository, be it to embed
them on the image, or to make them available to the installed system.
=> Choice: support unofficial firmware-enabled image(s)?
=> Question: how to ensure firmwares from backports are available in d-i
and in the installed system?
Finally, what architectures are we supporting? I can't speak for
debian-cd, but at least for the d-i part, we'll likely fork master into
stretch-backports and merge there. So we should keep track of any arch-
specific changes and debian-installer is likely to be buildable/built on
all architectures for no extra cost. If there are other bits that users
need to rely on, like flash-kernel and other components, I think they
should just consider installing testing instead. We can't reasonably be
expected to backport all the things.
=> Question: should we restrict architectures we build images for?
Last question: how often do we produce such an image? I don't think it's
reasonable to do that every time a new major version of linux is made
available in the backports repository. Doing that once around the middle
of the release cycle would look good to me. This would kind of mimic the
“and a half” thing we had in the past. Once we get the process right, we
might think about doing so another time or two, but we already have limited
humanpower to deal with stable point releases and alphas for testing
(at least oldstable is gone now)…
=> Question: what to advertise/communicate on? One-shot only is probably
a good rule of thumb?
What to do with modified components
===================================
My current approach would be to get net-retriever, apt-setup, and
base-installer (once this last one is ready of course) updated in
stable. This would greatly help building, testing, and running
debian-installer without having to special-case everything in various
places. Of course this is subject to a green light from the release
team, but I might be trusted to get things right (and/or to get the
pieces working again if anything breaks).
We would be able to rely on their having built-in support for backports
(remember, it's only enabled when a specific file is present). We could
then just upload src:debian-installer to stretch-backports (after dak's
been patched to allow that), and point debian-cd to the resulting
images.
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Attachment:
signature.asc
Description: PGP signature