RFC: Thoughts on building modules
One thing that strikes me as excellent about Debian is the build
system. The autobuilders and tools make it very likely that package
builds are reproducible and a variety of tools like debhelper make it
easier to do the "right thing" in many circumstances than doing
something wrong. I've come away from this experience convinced that
it is very important to have good tools for common tasks.
Sadly, I don't think building module packages for modules not in the
kernel lives up to the high standards of the rest of the Debian build
process.,especially for module packages to work with stock kernels
and be uploaded to the Debian archive. These concerns are mostly
technical and are to a great extent independent of
how many kernel images we have, although obviously if there are a
lot of kernel images then autimation is even more important.
Manoj's kernel package provides an excellent tool for building custom
kernels for end users. It also provides an adequate tool for end
users building modules for their kernels. The modules are shipped
generally as tarballs that are unpacked into /usr/src/modules. Then,
kernel-package generates .debs for each of these modules when
make-kpkg modules_image is run. The make-kpkg modules target is
intended for maintainers and is somewhat useful. However, it has
several shortcomings for official modules package. Because it is run
out of make-kpkg it is intended to build multiple modules for a single
kernel version. It turns out what maintainers probably want is to
build multiple versions of the same module for different kernel
versions at once. Also, module maintainers probably want to build on
multiple architectures. With make-kpkg this means having an account
on multiple architectures. It would be really nice to be able to take
advantage of the buildds to build modules for other architectures just
as the build non-module packages.
So, having established that we have some work to do, let's step back
and think about what is going on when you try and build modules. Each
architecture has a set of kernel-images that are current. Users
running a stock kernel are expected to install one of these images.
You as a module maintainer want to make it so that users can install a
version of your module for the stock kernel image they have
selected. So, for each architecture you want to find out what the
current set of stock kernel images are. Normally, there will be at
least one old kernel version supported and one or more new kernel
versions; there may be multiple flavors/subarchitectures of each
kernel version. There is currently no good way to find out what
kernel images are available. You could grep through the packages
file, but that doesn't exclude images awaiting removal or images for
special purposes that are not stock images.
Assuming you do manage to find out what stock images exist, you need
to build your module for them. There are roughly two approaches you
could take: using kernel headers or using kernel source. Apparently
some/many modules require a kernel source tree; a kernel headers tree
is insufficient. I don't actually know how common this is; I believe
the module I maintain could successfully build against a kernel
headers package if I tried. If you are using make-kpkg to build your
modules then you must have a source tree. For i386 apparently the
preferred way (according to kernel-image-i386 maintainer) is to use
kernel headers. That may work fine for i386, but other architectures
(sparc and m68k were checked) only provide one kernel headers package,
not one kernel headers per flavor. This is problematic because at
least according to the i386 kernel-image maintainer, we cannot
guarantee that modversions symbols will be the same between flavors,
so we cannot guarantee that modules built against one set of kernel
headers will work with other kernel flavors.
So, let's assume you want source for your kernel. On i386 you're all
set; you basically just download the kernel source and build against
it. On other platforms, it is more problematic. For Sparc, there are
patches applied to the kernel in the kernel-image-2.2-sparc package
for example. PPC, m68k and ARM appear to actually generate
kernel-patch packages for their architectures although I have seen
them get out of sync with the kernel images available in the archive.
Normally these patches don't seem to affect interfaces my modules care
about, so I can be sloppy and just download the kernel source
package. However, if we are talking about reproducible builds,
these patches matter, or at least an assertion that they will never
affect module builds.
There's also the issue of dependencies. When building most packages
one person is driving the need for rebuilds by changing the source
package. Sometimes some library or policy will change and you'll
have to rebuild even though there are not source changes you would
like to make. However this is fairly rare, and tends not to be
architecture specific. However, in the case of module packages,
several people can and often will create a situation that causes
rebuilding. You need to rebuild if the module maintainer changes
the source of the module package. You also need to rebuild if any
architecture changes the current set of stock kernel images.
So, now that I've ranted a bit about what we need to do, let's look at
ways of doing it. Most important constraint seems to be automated;
there are too many different modules to build to get it right by
hand. Another fairly important constraint is that we want to take
advantage of buildds; it seems silly to require module maintainers to
have package-installation access to a machine of every architecture.
So, in the forseeable future, if we are going to take advantage of
buildds we need to have a source package for them to build.
I see three options. We can have the modules built out of the main
source package that also produces the architecture: all package
containing the tarball for end users to use with kernel-package.
Alternatively, we can have a separate modules source package for
building only the module; this has the advantage of separating the
module components from other components of the system, which is nice
for things like PCMCIA and Openafs that have significant userspace
components. We could also try and get module building integrated into
stock kernel source packages; this would have the advantage of
rebuilding modules whenever architectures change their stock kernel
images, but has the disadvantage of not letting the module maintainer
easily rebuild modules when the module source change. I suspect
getting architecture kernel maintainers to agree to the third option
would be hard. I've also considered combined options like having the
module be built both by a module-specific package and by a
kernel-image specific package, being owned by whichever built it most
recently, but I suspect solutions in this space would interact very
badly with Katy/testing/ftpmaster. I suspect that the source package
will be something close to the second option, but I'd like to find an
option that doesn't require me to rebuild all i386 modules just
because alpha added a new stock kernel image.
We then move along to how to find out what the stock kernel images
are. We could build them into the source package which has the
dubious advantage the the module maintainer can build for a different
set of stock packages than the architecture actually builds for, along
with the major disadvantage of having to keep the information up to
date. Normally we try to avoid making maintainers keep up with
porting issues like what kernel versions are current on a given
architecture. The other option is to build the appropriate
information/dependencies into a architecture-specific package for each
architecture. The format should be standardized. For example, the
kernel-modinfo-sparc package might contain information on the current
stock kernel images for sparc that my module-specific package could
use to build modules for all the kernels.
The big issue I don't know how to deal with is kernel headers vs
kernel source. Really I think before I can go forward in ways to plan
to solve this problem, I need to understand that issue better.
Hopefully replies to this message along with an outstanding question I
have in another thread will help clarify.
Thanks for your consideration/comments,
--Sam
Reply to: