Bug#727708: loose ends for init system decision

[Russ Allbery <rra@debian.org>]

This message is about a transition plan for an init system replacement and
about how to handle portability to our non-Linux ports.  I'm keeping the
same subject as Ian's message on the same basic topics and attaching it to
the same thread, but this is more of a separate writeup than a reply.
I'll reply to Ian's message separately.

I've expressed my opinion separately about which init system we should
choose.  This message is written from the assumption that we will choose
either upstart or systemd as the init system for the non-Linux ports.  I
believe either system would pose basically the same concerns.


1. Role of Non-Linux Ports in Debian

There has been a lot of discussion in various places, including the Debian
mailing lists, about the utility of the non-Linux ports and about how much
they should play a role in project decisions.  I think this deserves to be
addressed directly here.

One of the things that I love about Debian, and one of the reasons why I
am involved in this project as opposed to a different distribution, is
that it is a labor of love.  Debian is not driven by market forces; we
have users and we want the distribution to work for them, but we're not
competing for market share.  Debian does not make decisions based on
simple popularity, or on customer counts.  Rather, Debian is a project of
mutual cooperation among Debian's contributors.  We work on what we care
about, regardless of whether that makes economic sense.

It is not, in general, necessary to justify what you want to do in Debian.
It doesn't matter if it's going to be used by thousands of people or two
people.  If you can do your work to the standards that we expect to be
maintained across the archive, and without negative impact on Debian's
other contributors, you can work on whatever you love.  And, furthermore,
we all support each other in our passions.  Debian is built on a culture
of deference to other people's work, and reasonable accomodation to the
projects that other people want to work on.

Now, there is a fine balance here.  Deference to other people's work does
not mean a requirement to join their work.  Reasonable accomodation does
not mean that every Debian developer is required to test their software on
non-Linux ports.  The goal is that all of us should be empowered to work
on the things we are passionate about, which implicitly includes being
empowered to not work on the things that are not of interest to us.
Therefore, for some efforts, and specifically for the non-Linux port
efforts, the work is mostly born by the porters.  They're expected to
test, diagnose problems, and submit patches.  The deference and reasonable
accomodation that we expect of Debian contributors is to merge those
patches when they are reasonable, to not undermine the porting effort when
there is a reasonable approach that preserves it, and to be aware of the
implications of their packaging for those ports in the broad sense (such
as qualifying build-dependencies with [linux-any] where appropriate).

We do not expect Debian contributors to reject significant new upstream
versions or significant new integrations because they will affect the
non-Linux ports, or, for that matter, other projects in Debian.  We do
expect those changes to follow the standards that we expect of Debian as a
whole, and that porting efforts will be treated with deference and
reasonable accomodation.

I think this status applies to both our Hurd port and our kFreeBSD port.
Those ports have different challenges, and arguably different levels of
utility to general users, but as mentioned, I don't consider popularity to
be a useful metric here.  It doesn't make any difference to me if the port
is used by thousands of people or if it's only used by the people actively
working on it: the same principles of deference and reasonable
accomodation apply.

I believe strongly that this is something that defines Debian as a project
and is worth preserving.

It's also worth saying here that I don't believe any of the above is
particularly controversial within the project.  We have wide-ranging
arguments about it in the abstract, and quite a few disparaging comments
have been thrown around about the non-Linux ports in the init system
discussion, which makes me sad.  But when it comes to the day-to-day work
in the project, nearly all maintainers are quite good (and have been for
years) about merging required patches for the Hurd and pushing them
upstream, responding to bug reports on systems they don't use, and making
a reasonable effort to accomodate other people's projects.

Similarly, our non-Linux porters have been exemplary.  Neither the Hurd
nor the kFreeBSD ports have expected every Debian contributor to directly
support their platforms.  They track down problems, develop patches, and
submit tested patches to the BTS.  They have developed their own solutions
for packages that are not portable and worked out how to integrate them
with the archive.

Despite the often off-putting public arguments, and despite occasional
tensions and counterexamples, by and large we do a good job at this.  And
we should be proud of that.

This is, of course, directly applicable to the init system discussion
since neither systemd nor upstart are currently portable to either of our
non-Linux ports.  It will continue to be directly applicable to this
discussion even if upstart is ported to kFreeBSD until such time as it's
also ported to the Hurd (or the Hurd porting group decides they're no
longer interested in working on the port, but I don't expect that to
happen).


2. Impact of Multiple Init Systems

Obviously, the ideal situation for project-wide integration and support,
and for Debian's ability to make full use of the capabilities of new init
systems, is for all of Debian's ports to be running the same init system.
Attempting to support multiple init systems has several obvious drawbacks:

* Packages will either be limited by the functionality of the least
  capable init system or will behave differently (and have to be
  configured differently) on different ports.

* Nearly all Debian contributors are personally only going to be running
  the default init system on the amd64 or i386 ports.  It's not viable to
  ask them to test all packages on non-Linux ports.  This, plus the
  distribution of our user base, means that the configuration for the
  default init system is the one that will be tested, and configurations
  for any other init system will tend to bitrot.

* Related, Debian contributors will normally not be in a position to test
  patches for daemons for non-Linux ports, and will be entirely reliant on
  contributed patches from porters or users of those ports, which means
  the level of quality we can maintain for those configurations will be
  lower.

* If we're going to support init system switching, we will have to
  continue to externalize daemon configuration in files that are
  compatible with the sysvinit system.  Both systemd and upstart have
  better ways to handle local configuration than /etc/default files, but
  their methods are incompatible (since they involve overriding the
  init-system-specific configuration files in their own ways).  If we
  fully adopt one of those init systems, we can largely drop the somewhat
  clunky /etc/default machinery in favor of much simpler overrides or
  direct conffile modifications.

* Upstreams may, over time, drop support for init systems that are rarely
  used outside of Debian's non-Linux ports.  I don't think this is likely
  to happen soon, but I can anticipate a world in the future (particularly
  if upstart adds support for the systemd socket activation protocol)
  where some upstreams will strip support for fork and exit daemon startup
  methods from their daemons, or require use of socket activation and
  remove socket binding support.  In some cases, we already have tools to
  deal with this; handling daemons that don't support fork and exit is
  fairly easy, for example.  But in some cases we may not, and I think
  maintaining full code for socket binding and setup as a patch that
  upstream does not want is too much to ask from all Debian contributors.

However, there are also some counter-balancing points:

* All packages in Debian are already ported to sysvinit, which means that
  the init scripts and related machinery already exist.  Furthermore,
  maintaining Debian's normal support for partial upgrades and loose
  upgrade ordering between stable releases requires that we continue to
  support sysvinit scripts at least through the release of jessie.  In the
  short term, we cannot avoid supporting two init systems if we're going
  to provide robust upgrade support, which gives us short-term support of
  the non-Linux ports for "free."

* Shell init scripts are obnoxious to write and often have buggy corner
  cases, but once written, they mostly keep working in their buggy glory
  with some minor maintenance.  In other words, they don't do as good of a
  job, but regressions are relatively uncommon.  This means that it is
  possibly viable for porters to provide patches and continued maintenance
  of init scripts for at least the services that are considered most
  critical even if the primary maintainer cannot easily test.  Packages
  with extremely complex init scripts will require someone do special
  testing, but the number of such packages is relatively low.  This will
  fail if upstreams start dropping required support for sysvinit-style
  startup, but I don't think this will happen soon.

* Maintaining init scripts is not, in the grand scheme of Debian work, one
  of the harder things we do.  I think asking Debian contributors to
  maintain sysvinit scripts when Debian's default init system is something
  else falls outside the boundary of reasonable accomodation, but that's
  mostly because of the testing requirement.  I do not think that leaving
  init scripts in the package where they already exist and applying
  patches from porters falls outside those boundaries.


3. systemd and upstart As Multiple Systems

I said a great deal above about Debian's non-Linux ports.  It's worth
considering that the same statements potentially apply to multiple
next-generation init systems.

As we've seen from this debate, both upstart and systemd inspire
passionate loyalties and preferences.  Having looked at both of them, I
fully understand the feeling.  I have a strong personal preference for
systemd, but upstart is a beautiful piece of code that would be a delight
to work on.  It's clean, well-documented, consistent, and has an excellent
test suite.  Both projects are working hard at writing something they
believe in.

Given that, I don't believe a Technical Committee choice of a default init
system is going to make either the systemd or the upstart maintainers want
to stop maintaining their packages.  For upstart, there is also the
ongoing fact that Ubuntu uses upstart, and Ubuntu is one of our major
downstreams and a collaborative project.

I therefore think that, regardless of which init system we pick, we should
keep in mind the above principle of Debian's deference and reasonable
accomodation to other people's projects and apply that to systemd and
upstart as well as to the non-Linux ports.  Obviously, this also has the
same issues mentioned above: Debian contributors can only be expected to
test on the primary init system, other configurations will tend to bitrot
without active porter attention, and so forth.  But if people want to take
on the work, that deserves our respect.


4. Conclusions

I previously argued that much of the benefit of a new init system comes
from when we can stop maintaining init scripts.  I still believe that, but
after thinking more about the cultural and project issues at stake here,
as well as the immediate needs for a clean upgrade path, I ended up at
more of a compromise position than I expected.

I believe Debian should take this path forward:

1. We should select a new init system for jessie, either systemd or
   upstart.  Support for that init system should be release-critical, but
   only in the sense that the daemon works properly under that init
   system.  In other words, use of the sysvinit compatibility of either
   init system is acceptable support for jessie.

2. All packages providing init scripts must continue to support sysvinit
   scripts through the jessie release.  Such support will continue to be
   release-critical.  This is going to be painful for packages that want
   to do an early conversion, since it means testing two different init
   systems for this release cycle, but I think this is the right thing to
   do regardless for a clean upgrade path and Debian's normal robust
   committment to upgrades.  Here it has the additional advantage of
   giving the non-Linux ports some breathing space to strategize.

3. Related, up through the jessie release, packages must (where possible;
   it's possible there will be cases where this is simply impossible)
   support switching back and forth between the new default init system
   and sysvinit.  This means that configurations should not be moved out
   of /etc/default and that startup files for the new init system should
   read the same configuration that the existing sysvinit scripts use (or
   both should be modified compatibly).

4. Post-jessie, support for sysvinit will no longer be release-critical,
   and package maintainers will no longer be expected to ensure that it
   continues working.  However, for as long as Debian has accepted
   non-Linux ports using a different init system, package maintainers
   should continue to ship init scripts if they work and should apply
   patches and other reasonable fixes from porters for those init scripts.
   In other words, this should be treated the same as merging patches for
   the Hurd to remove hard-coded constant assumptions: if the change is
   reasonable and doesn't break Linux ports (and this should be fairly
   easy to handle for nearly all cases with init scripts), the package
   maintainer should merge it.

5. Support for the other init system that was not chosen should be handled
   in the same fashion, should a team choose to pursue it.  If we select
   systemd, package maintainers should still be willing to merge
   contributed upstart configuration, with the understanding that they
   can't test it and any support is on a best-effort basis only.
   Similarly, if we select upstart, package maintainers should be willing
   to merge systemd unit files and to enable upstream systemd support
   where requested and where it doesn't interfere with the operation of
   the daemon under upstart, with the understanding that the package
   maintainer is not committing to testing or directly supporting this
   configuration.

6. Debian's non-Linux ports should either use the same init system as
   Debian's Linux ports or agree on an init system that they're both going
   to use.  The porting work is going to be hard enough without the ports
   going in different directions on which secondary init system they want
   to use.  I prefer to leave it up to the porters to decide which init
   system to choose, but I do think OpenRC would be a strong contender.

7. After jessie, functionality between systems running the primary Linux
   init system and other init systems (including non-Linux ports) should
   be allowed to drift.  In other words, there will be cases where
   features will only be available with the primary init system.  Possible
   examples include security hardening, socket activation, automatic
   daemon restarts, and so forth.  Packagers are under no obligation to
   port those features to other init systems, but should welcome and merge
   patches that do so.  After jessie, packagers will no longer be required
   to preserve daemon configuration when the init system is switched, so
   use of such facilities as modification of upstart configuration files
   or systemd overrides may be used.

We should revisit this decision again after the jessie release in case the
situation has substantially changed.

This is all rather unsatisfying all around, I realize.  Debian is going to
miss out on some opportunities to pursue new init system features
aggressively and consistently across the archive, particularly for the
jessie release where we're maintaining full sysvinit compatibility, if we
take this posture to both upgrade compatibility and deference to porting
efforts.  And this is also going to increase the porting work required for
non-Linux ports substantially, since the init scripts will have to be
included in that and porting won't be as easily measured by buildd logs.

However, I think it's the best available approach that balances our ideals
as a project against the opportunities offered by a new init system.  This
approach does permit full use of new init system features for jessie
except for eliminating /etc/default files (which I doubt we'd successfully
do quickly anyway), and opens up the full spectrum of use cases after
jessie.  The cost is that packagers should merge contributed patches to
the init systems that they don't use.  I don't think this is too much to
ask, nor do I think it will have serious effects on package complexity
based on my own experience configuring a package to work under all three
init systems I considered.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>