Bug#727708: init system thoughts
I see that the LWN commentariat already has my decision selected in
advance, so I might as well write up some more detailed thoughts before
any other words are put into my mouth!
Choice of default
-----------------
Firstly, I've tried to keep my employer affiliation out of this as much
as possible, and I have come under no pressure from that quarter. I'll
reiterate my previous comments on debian-devel:
http://lists.debian.org/debian-devel/2013/10/msg00747.html
http://lists.debian.org/debian-devel/2013/10/msg00777.html
http://lists.debian.org/debian-devel/2013/10/msg00818.html
It's apparently no surprise that my preference for a default lies with
Upstart. To the extent that I have a pre-existing bias it has more to
do with the fact that I have substantially more real-world deployment
experience with Upstart in an environment structurally very similar to
Debian (Ubuntu, obviously), which has led me to believe that it would be
a fine choice for deployment in Debian which could be put in place with
a minimum of disruption, and which would close as few doors as possible
to experimentation and innovation.
That said, I will add that I have been impressed with the technical
aspects of systemd that I have seen while working on adding support for
it to my packages by way of research for this debate. Its documentation
is comprehensive, a great deal of work has clearly gone into the range
of options available in unit files, and it has admirable facilities
available for system administrators. I am quite happy for it to be my
second choice (which is by no means irrelevant given our voting
protocol); I think it would be a more than worthwhile step forward from
sysvinit, just in a somewhat different direction that does not appeal
quite as much to me.
Reservations with systemd
-------------------------
My main concerns with systemd relate to its broad scope regarding units
it provides for system initialisation tasks currently performed by other
packages, and the potential for that to interfere with past and future
work elsewhere in Debian. I can understand why the systemd authors felt
it valuable to expand its scope in this way, to provide a more
consistent experience across the board. Nevertheless, it seems to
significantly complicate the job of integrating it with some excellent
features that already exist in Debian and which are superior to those
currently in most other distributions. I realise that the Debian
systemd maintainers have generally expressed willingness to deal with
this kind of integration problem where appropriate, but I find the
impedance mismatch to be much higher than it is with Upstart which
leaves these tasks up to the distribution.
The two examples which I've run across so far, both ones I was inclined
to look at since I'm familiar with the territories, are:
#716812 (binfmt-support)
(I'm the author and maintainer of binfmt-support.)
The discussion on this (archived earlier in #727708) does now seem
to have been resolved reasonably amicably so far; I've turned
binfmt-support into an independent upstream package hosted on
nongnu.org, given it a systemd unit file while I was there, and will
work on getting that into more distributions.
However, I maintain that this is really something that has no
particular reason to be integrated with the init system, that it has
much more interesting interactions with packaging than it does with
other system events, and that it is busywork to convert things away
from a system that works measurably better in Debian and derivatives
than (TTBOMK) any other GNU/Linux distribution.
#608457 (console-setup)
(I've contributed to console-setup in some small ways, although I'm
not its primary maintainer by any stretch of the imagination.)
console-setup is a fantastic piece of innovation in Debian; instead
of shipping two entirely independent sets of keymap files for
virtual consoles and for X, the former of which is mostly unloved,
let's generate virtual console mappings dynamically from XKB! It's
naturally not an easy task since the console is less capable in
various important ways, but it's demonstrably possible to do a
pretty accurate job. Like binfmt-support, it probably ought to be
turned into an independent upstream package to make it easier for
other distributions to adopt, but in the meantime it's tremendously
useful for Debian.
This seems a bit conceptually tricky to integrate with systemd,
because localectl(1) exposes the console-data style of keymap naming
in its interface, and with console-setup you only use the X style.
The Debian systemd maintainers haven't yet offered a suggestion in
the bug, but perhaps they'll come up with something appropriate
(maybe just disabling set-keymap/list-keymaps, or converting both
ways and accepting the inevitable round-trip errors). But my point
is that this is a class of problems that simply doesn't arise with
an init system that has a smaller scope.
Basically, systemd would be more compelling to me if it tried to do
less. I don't expect to persuade systemd advocates of this, as I think
it amounts to different basic views of the world, but the basic approach
here is probably the single biggest factor influencing my position.
The criticisms of Upstart's event model in the systemd position
statement simply do not make sense to me. Events model how things
actually happen in reality; dependencies are artificial constructions on
top of them, and making them work requires the plethora of different
directives in systemd (e.g. Wants, which is sort of a non-depending
dependency) to avoid blocking the boot process on a single failing
service. Yes, there are some bugs possible in the Upstart design, but I
don't agree that those are world-ending fundamental issues and I think
they're generally incrementally fixable. The systemd design appears to
me to expose far more complexity to the user writing unit files, and I
think it's important that their mental model be as straightforward as
possible.
(Now, I've been working with Upstart's model for years, and it's now a
pretty fundamental way of how I think of system operation; so if people
who are new to *both* models rather than partisans of one side or the
other consistently tell me that the systemd model is easier to grasp,
then I'll listen.)
There is of course also the non-Linux porting issue, which Ian, Russ,
and Steve have discussed at more length elsewhere, and I won't repeat it
at length. I will just add in response to Russ that there is a great
difference between one project whose lead maintainer has explicitly said
that he will reject patches for porting to non-Linux architectures
because they fundamentally do not make sense with its architecture, and
another project one of whose developers has been working on such a port
with some degree of success so far.
Like Ian and Steve, I don't have much in the way of personal attachment
to the non-Linux architectures in Debian, but I feel that it is very
important not to close off options and to keep the spaces for
experimentation with different kernels that we've built; you can't
meaningfully experiment with a different kernel without building an OS
on top of it, and Debian has provided a useful framework for this for
many years now. There is some chance that we might be able to magic up
a truly compelling scheme whereby we can support different init systems
on different kernels - I would certainly encourage porters to try - but
it seems like a risky and difficult plan.
Reservations with Upstart
-------------------------
I am no fan of the CLA; this will not especially surprise readers from
Canonical although I'm not sure I've said it straight out in a Debian
context before. I think that it has proven to have far more downsides
than upsides for the company and that it is an unnecessary obstacle to
building a development community, and I've made that argument
internally. This is a minus point for me. That said, from Debian's
point of view I don't see significant practical differences from a
BSD-style licence, and especially given the declared willingness to take
non-CLA patches in the Debian packaging in a reasonable way I don't see
it as a blocker. (Also, given that Upstart intentionally has a smaller
scope, a smaller number of developers is less of a problem.)
The ptrace arrangements used for "expect fork" and "expect daemon" have
been rather flaky in practice, especially when Upstart jobs are written
by people not experts in doing so, and they are an obstacle to
portability. I think we should strongly discourage use of these in
Debian in favour of some other readiness protocol. My personal
aesthetic preference is for the "expect stop" scheme or something close
to it, but I really don't care that deeply and the sd_notify scheme or
similar would work too. Indeed, I might well be inclined to support
disabling the ptrace-requiring features entirely in Debian, and working
to disable them in Ubuntu in time as well (although that would require a
transition plan).
I think Russ has sold me on systemd's journal being a win, at least in
terms of how it enables much better status output for services, and it's
a shame that something equivalent isn't present in Upstart. My
practical experience of late has been that the per-job log files you get
by default are good enough for most purposes where I need to debug
service operations, but it certainly isn't all packaged up as neatly.
Deployment plans
----------------
I am strongly in agreement with Russ' position on the matter of multiple
init systems in the archive. While the world would be a lot simpler if
we only had one to support, I don't see that as achievable in the short
term without (to me) unacceptable losses, and this is mainly a matter of
refraining from deleting code. I don't think we have to be doctrinaire
about mandating lots of effort for every service maintainer, but at
least asking them not to break things that people are still using (and
that we'll need for upgrades at least for a while) seems reasonable.
I would love to see the "multiple" position in the debate wiki
(https://wiki.debian.org/Debate/initsystem/multiple) fleshed out into
more of a practical deployment proposal; I unfortunately haven't had
time to dive into it in detail but at present it feels rather handwavy
to me. I think that this should be a major part of what we try to
thrash out over the next few weeks (at least to establish feasibility,
given our mandate not to do detailed design work within the TC as such),
and that there should eventually be ballot options related to this
orthogonal to the choice of default.
--
Colin Watson [cjwatson@debian.org]
Reply to: