[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: init system daemon readiness protocol

Vincent Bernat:
> All of them are relying on the fact that the monitored process won't fork.
> They are therefore not able to handle readiness and dependencies.

Also untrue. Handling dependencies has nothing to do with forking, and it's an error to think that anyone handles readiness. Readiness is not something that any service manager does reliably. There's just not the consistent mechanism for it, yet. The RabbitMQ server on one of my machines, to pick one example, can take bloody ages to start up when it so happens that it has a lot of persistent messages to scan back in from disc. And it doesn't even bind its listening sockets, let alone communicate, until it's done. The end result is that no AMQP client program can connect, and just (say) stall waiting for the login response. No service manager fixes this. RabbitMQ server doesn't fork and exit, and it doesn't yet speak any notification protocol. The service managers think that it's up and ready, so they go ahead and start all of the AMQP client services, that are all quite properly marked as depending from it and that are all quite properly not started until after the server is. The clients then repeatedly fail to connect().

(This is not the only problem with RabbitMQ and service managers. There are at least two startup races with the erlang runtime and the erlang port mapper that I know of. It would be nice if, for starters, the server could at least be passed a listening socket file descriptor. Yes, there's "rabbitmqctl". But that's the point, of course. Major softwares still work on the "Use only our package-specific XYZctl massive russian-doll-like scripts for service management." basis.)

Russ Allbery, Ian Jackson, and others all discussed this, and made these same points, back in December 2013 and January 2014. I highly recommend reading the debian-ctte mailing list discussion. You'll see discussion of the lack of standardization of readiness protocols, the absence of any widely adopted readiness protocol, and (yes) both M. Jackson and M. Allbery in favour of discouraging the idea of forking in order to "daemonize" something that is already a daemon. (As M. Allbery knows, I've been encouraging people not to uselessly fork for ... ahem! ... some years. I recently revisited my FGA with an observation from my on-going project to make 155 nosh service bundles for the BSDs. A lot of BSD programs, in recent years, have quietly gained "don't fork" command line options of one sort or another.)

On the subject of readiness protocols: To see another already-established mechanism, perhaps avoid reinventing the wheel, and once again see the error of thinking that all this was only just thought of and entirely missing a whole history of tools development, see Laurent Bercot's fifodir mechanism (http://skarnet.org/software/s6/fifodir.html).

On the subject of forks: It's interesting to observe that an EVFILT_PROC kevent() with NOTE_TRACK works reliably on FreeBSD/PC-BSD for tracking across forking daemons. This is not to say that forking is suddenly alright. But it gives the lie to any claim that daemontools-family service managers lose track when daemons fork. The nosh service-manager is quite happy with the old you-have-to-turn-an-undocumented-debug-option-on-to-stop-it forking Vixie cron on FreeBSD, for example, and is perfectly aware of and tracking the forked child process after the parent exits. You'll notice that I didn't put a "fghack" in nosh. It's not needed on FreeBSD, and should be equally well unneeded on Debian kFreeBSD. I must remember to give M. Bercot, M. Pape, M. Guenter, and M. Sampson some good news.

And as I said, dependencies have nothing to do with forking. Dependencies are a matter of having some way of specifying dependencies in service configurations, and topologically sorting the low-level start/stop tasks. Felix von Leitner's minit was doing dependencies back in 2000.

Reply to: