Re: Bug#727708: init system thoughts
On Mon, Dec 30, 2013 at 10:37:42PM -0800, Steve Langasek wrote:
> On Mon, Dec 30, 2013 at 09:58:07PM -0800, Josh Triplett wrote:
> > > But in the real world, we have a lot of services that we just want to start
> > > in runlevel 2 and be able to trust that the network and disk are sorted.
> > > This is the classic guarantee that sysvinit gave us pre-udev, but it's
> > > fallen apart since then because a fast system can get all the way through
> > > rcS before the kernel has even managed to enumerate all the network devices.
>
> > udev didn't break this assumption; this assumption became unreasonable
> > once people started wanting to dynamically change networks. If you have
> > a service that can't cope with network interfaces showing up later on,
> > that same service won't cope with connecting to a new wireless network,
> > or plugging or unplugging an Ethernet cable.
>
> > I've run into this *exact* problem with upstart, and I found it entirely
> > untenable.
>
> You've also said that your experience with upstart was on ChromeOS, not
> Ubuntu. I'm happy that ChromeOS is using upstart, but it shares almost none
> of the more recent event policy that would be common to Ubuntu and Debian
> and is not a useful proxy for understanding how upstart would behave in
> Debian.
I've also used upstart on Ubuntu (on servers, in particular), though not
extensively (or perhaps I should say "as little as possible"). And I've
used upstart in Debian, which I'd expect to be an exemplar of how
upstart behaves in general. (I will certainly admit to not having used
it in Debian *recently*, but I've used it in Ubuntu recently. I found
it no less painful.) More importantly, I've studied the upstream
behavior of upstart quite extensively; I wouldn't by any means consider
myself an expert, but I've read the upstart documentation in detail
multiple times. If you have additional information about ways in which
Ubuntu has made upstart behave differently than the upstream upstart
behavior that would apply to this particular case, I'd be interested in
hearing about them.
Also, I'd consider it a deficiency in upstart if understanding upstart
in one distribution does not necessarily provide universal information
about how it behaves everywhere. systemd makes a point of addressing
that problem: if you understand how systemd works in one distribution,
you understand how it works, period, modulo any (hopefully temporary)
distribution packaging bugs.
> Not that I'm sure what you mean by "this exact problem", anyway. The fact
> that services that don't deal with network changes are problematic? Yes,
> ok, but I don't see how that has anything to do with using upstart or not,
> and is not what I was trying to say upstart solved. The more common case,
> where you need to start a service that listens on a socket, is that it is
> running on a server and you need to make sure the static network - which,
> once configured, will not change for the lifetime of the server - is up
> before trying to start the service.
That's the exact case I was talking about, modulo the non-existence of a
concept like "the static network". I've worked with upstart jobs for
services that don't understand how to deal with a changing network, and
found it a sufficiently lost cause that it was easier to fix the service
(or rather, to select a better implementation of the service). The
events associated with network devices did not seem particularly clear,
nor did the execution model of upstart with respect to one upstart job
and multiple relevant events.
> Again, we're all agreed that the server
> shouldn't be coded that way, but that doesn't change the fact that there are
> a lot of these daemons out there in packages with init scripts, and we want
> them to be as reliable as possible.
The right way to make them as reliable as possible is to treat them as
buggy and fix them. Anything else is a hack to work around buggy
daemons. I'd argue that any facility in either upstart or systemd (or
sysvinit for that matter) that waits around for the network should be
exclusively for use by the system administrator as a dependency of local
services, and that Debian should not ship any daemon depending on such a
facility; that's the kind of thing I'd like to see making its way into
Policy. Since such a service would be buggy today with sysvinit, I
don't think such a Policy change would depend on any particular init
system.
> > In any case, systemd does indeed "support" this case; simply make your
> > service depend on network-online.target, which will block for a
> > reasonable amount of time to see if a network interface comes online,
> > and eventually time out if that doesn't occur. This will actually work
> > even if your primary network is a wireless network managed by
> > NetworkManager, and even if that network doesn't actually work until the
> > user has logged in, assuming your service is not actually in the
> > dependency path of the user logging in.
>
> And what makes this work in the case where you *aren't* using
> NetworkManager? I see no integration with ifupdown in the systemd package.
Regarding ifupdown, see above regarding buggy tools that don't
understand dynamic network configuration. Nonetheless, the relevant
integration with ifupdown lives in the NetworkManager package, which
knows how to parse ifupdown's configuration, and will properly register
the network as "up" when ifupdown's "static" network has been brought
up.
I don't think anyone has reimplemented that functionality in another
package, but if someone wanted to write a duplicate service that
listened for ifupdown events and blocked network-online.target, that'd
be straightforward enough: just drop a script into /etc/network/if-up.d
that creates a flag file in /run when it considers the network "up", and
a service blocking network-online.target that waits for that flag file.
(Or, if you want to go a step further, make it a persistent daemon that
properly provides the DBus interface that other services use to
determine if the network is "up".) Such a service would conflict with
(and break) the equivalent functionality in NetworkManager (not least of
which by unnecessarily blocking network-online.target if a network that
ifupdown doesn't know about is up), so it shouldn't be part of the
systemd package, but perhaps an appropriate .service file and hook could
live in ifupdown.
If I thought that the existence of a network-online.target
implementation for ifupdown would change anyone's vote, I'd be happy to
write it and submit it as a patch against ifupdown, following the
trivial implementation above; I assume, however, that it wouldn't. In
any case, while it might be nice for ifupdown to integrate better with
systemd, I don't see any fundamental deficiency in systemd's dependency
model here that makes upstart more capable of handling this case, as you
suggested in your original mail on this topic.
- Josh Triplett
Reply to: