Bug#727708: init system other points, and conclusion
This message contains some supplemental information to go with my primary
writeup, and some profound thanks for the people involved in this
investigation.
I apologize for the huge volume of mail, and I know it's going to take a
while to digest. I appreciate people's willingness to read all these
messages in detail. This is a very complex decision-making process. I'm
pretty mentally exhausted by the effort of trying to synthesize all these
pieces, so my apologies in advance for the inevitable errors and omissions
(some of which affected my original writeup).
1. Thanks
Throughout this evaluation process, my interactions with upstart and
systemd upstream developers and Debian packagers have been uniformly
excellent. Bug reports filed against both systemd and upstart have
received excellent and timely response, and all involved have been quite
willing to explain things I've misunderstood, correct my false starts, and
discuss technical and practical aspects of their designs.
I was particularly impressed by the clear effort that the systemd and
upstart maintainers in Debian have put into fully integrating their init
systems in such a way that makes them easy to test and use with existing
Debian packages. This includes but is not limited to update-rc.d support,
invoke-rc.d support, status synchronization with sysvinit, past Policy
discussion, and attention to upgrade paths and init-switching use cases.
I also want to particularly thank the OpenRC upstream development team for
their involvement in this process and their contributions to the
discussion. I personally don't think that package is a god match for
Debian's needs on Linux, but that's through no fault of the people
involved, and I think they would be an excellent upstream if that package
looked like a good fit for the needs of any of Debian's non-Linux ports.
I also want to thank Petter Reinholdtsen, Roger Leigh, and everyone else
who has worked on the sysvinit package over the years, the insserv
conversion to dependency-based boot, and the inclusion of LSB support. If
it weren't for their hard work over the years, we would be in a far worse
position than we are today. It's often hard to see people discussing the
inadequacies of something into which you put years of hard work. I want
to call attention to their long-term contributions to the distribution and
the number of Debian systems that have booted through their efforts
through the years.
I am carefully keeping this discussion to the Technical Committee bug
report so that all of my comments stay in context with their rebuttals,
but this one section I think is unrelated to the rest of the discussion
and should be more widely posted, so I will also be posting the above to
my blog and Planet Debian.
2. upstart vs. systemd: The Equivalencies
In doing this evaluation, I looked at quite a few different things. In a
great number of cases, I found both upstart and systemd to be at an
equally high level of quality. There are too many of those to list here,
but I wanted to mention a few points that had been the topic of wider
discussion.
* The documentation of both systems was uniformly excellent. Both systems
had complete reference manuals plus guidance to packagers for how to
integrate with them. systemd also had excellent guidance for upstream
developers on how to add systemd support relevant to upstream packages.
* Both packages preserve traditional logging behavior, continue to
properly utilize syslog, and retain logs in the places where I expected
them. This is obviously not surprising for upstart, but I wanted to
mention it explicitly since those unfamiliar with the systemd journal
may be afraid that it would migrate logs into a separate, unexpected
store. This is not the case; systemd logs are available in the normal
places and can continue to be managed through syslog configuration. The
journal is supplementary and enables some additional features discussed
elsewhere.
* Both packages move to configuration, from code, many of the annoyances
and fiddly pieces involved in starting daemons. Both support
non-forking daemon models and proper PID tracking in their own ways.
Both directly support such routine needs as setting user and group, OOM
score adjustment, resource limits, and so forth.
* Both packages provide MAC integration, which is something that I think
will become more important for Debian in the future. (systemd provides
some additional features around Smack, but at least given what I know
right now, I don't think this is a significant differentiation.)
There was another point that I am calling equivalent since I didn't
evaluate it either way. It's possible that this would convert into an
advantage for one side or the other after further investigation:
* Both systemd and upstart provide a way to run the system as a user
process to manage user jobs in a similar fashion to how they manage
system jobs. I believe this is already used extensively in Ubuntu to
manage user desktop sessions. I don't where the systemd equivalent has
been used.
Finally, my interaction with both upstreams has been excellent, as I
mentioned above in the Thanks section. I believe both maintenance teams
would make excellent partners and upstreams for Debian going forward.
3. upstart: The Minor Advantages
The following points did not make the cut into my main analysis. I
consider them minor advantages that don't carry the same weight as the
things that I commented on. However, I did want to mention some that had
been the topic of discussion.
* The upstart and libnih code bases are beautiful pieces of work. I was
quite impressed by the code quality and documentation level, and more
impressed by the extensive test suite. upstart and libnih follow the
gold standard, commonly advocated but rarely met, of having around half
of the code in the package be test cases. These are clearly code bases
that have seen a great deal of love, care, and consistent development
standards. The systemd code base also struck me as solid and at the
standard that we would expect for a critical package, but the testing
model is not as comprehensive or as integrated with the code, and it
didn't impress me the same way that the upstart code did.
* Both upstart and systemd are used by other major distribution projects,
but upstart is used by Ubuntu, with which Debian has a special
relationship. Debian collaborates more extensively with Ubuntu than
with any other project, including largely sharing packaging between the
projects and benefiting greatly from each other's testing and
integrations. With upstart, Debian would have the immediate use of many
upstart configurations that are either already in the archive or already
available in Ubuntu, and which already fit Debian packaging standards
and tools. I consider this a minor advantage for upstart.
* upstart provides a more straightforward escape to shell scripting for
some difficult initialization situations, whereas the systemd syntax for
doing so is rather awkward. This cuts both ways, so you'll find that
this point makes an appearance in the section below as well. But I
think that, in some situations, this is a readability and ease of
integration advantage for upstart.
4. systemd: The Minor Advantages
Similarly, during this evaluation, I found several things that I preferred
in systemd, but which didn't make the significance cut in my final
evaluation. Here are some of them for the record.
* systemd does not require a CLA, whereas upstart does. The Canonical
Contributor Licensing Agreement has been much-discussed in these
threads, and some people find it quite intrusive and unacceptable. The
upstart package maintainers have committed to carrying non-CLA-covered
contributions as local patches, which does a lot to ameliorate this
concern, but I still consider this a minor advantage for systemd.
* systemd provides really nice command-line tools for understanding the
state of the system and the relationships between the unit files. I
don't believe upstart has an equivalent of systemctl list-dependencies,
for example. (systemctl status got special mention in my main
evaluation.)
* Both upstart and systemd, due to their more declarative nature, allow
for daemon configurations that are more portable between different
systems running the same init system. systemd has taken this farther
into comprehensive and useful advice to upstream daemon authors for how
to incorporate systemd support into a package, and also provides
step-by-step instructions for how to install systemd unit files during
package installation. While I don't believe that full portability of
unit files will be achievable, I still think Debian would benefit from
this when integrating systemd, as upstream unit files will often be
usable as-is or with some minor patching.
* systemd puts a lot of work into providing all the configuration
directives required to express a huge variety of use cases without
having to write complex startup commands or escape out to shell. This
is the flipside, in some ways, to upstart's easy escape to shell. I
particularly noted the wide variety of Condition* settings to control
whether a unit should start. This is valuable for distribution
packaging cases, but I think it's even more valuable for users of
systemd-controlled systems in setting up local overrides and conditions
for when they want to run a particular service.
--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>
Reply to: