Bug#982950: ssh.service starts sshd before network is online: please switch to After=network-online.target instead of just After=network.target
Control: severity -1 important
On Wed, Feb 17, 2021 at 09:36:15AM +0100, Thomas Goirand wrote:
> Package: openssh-server
> Version: 1:8.4p1-4
> Severity: grave
No. It may yet need to be sorted out before release, but this is a rare
situation and I'm not having it being release-critical right now,
especially because it's not clear that it's openssh-server's problem.
> On a Sid/Testing system, currently we have in /lib/systemd/system/ssh.service:
>
> After=network.target auditd.service
>
> While this isn't a problem in most installation, it didn't work under our setup,
> because we use "bgp-to-the-host" networking. In this setup, we need FRR (the
> BGP routing daemon which is a fork of Quagga, if you didn't know) to provide
> network connectivity to the server. Our configuration is something like this:
>
> # cat /etc/frr/frr.conf
> [...]
> !
> int lo
> ip address 10.56.17.7/32
> !
> [...]
>
> This means that, until FRR is fully up and running, with the BGP session
> established, the server IP (10.x.x.x/32 bound to the loopback interface) isn't
> set yet on the server, and the ssh daemon cannot bind on the IP (as it's not
> there yet).
Are you using ListenAddress in sshd_config? Normally sshd doesn't
require network interfaces to be online - it just binds to 0.0.0.0 or
[::] and that's enough for it to be bound to interfaces as they come up.
If lo has to be up for this to work (which is surprising to me, but
maybe), then I think there's a decent argument for frr to be part of
network.target on such systems.
> Our fix was pretty simple:
>
> # cat /etc/systemd/system/ssh.service.d/override.conf
> [Unit]
> After=network-online.target auditd.service
>
> But IMO, this is very wrong to mandate doing this, and not having ssh
> connectivity after a reboot, is kind of a grave problem.
>
> So, could you hard-wire this in the openssh-server package directly, so Debian
> users can avoid such an override? Indeed After=network.target doesn't tell you
> that network is ready. After=network-online.target does, and that's IMO what
> the ssh daemon should be using.
I don't agree that network-online.target is appropriate in general, from
its documentation:
network-online.target
Units that strictly require a configured network connection
should pull in network-online.target (via a Wants= type
dependency) and order themselves after it. This target unit
is intended to pull in a service that delays further
execution until the network is sufficiently set up. What
precisely this requires is left to the implementation of
the network managing service.
Note the distinction between this unit and network.target.
This unit is an active unit (i.e. pulled in by the consumer
rather than the provider of this functionality) and pulls
in a service which possibly adds substantial delays to
further execution. In contrast, network.target is a passive
unit (i.e. pulled in by the provider of the functionality,
rather than the consumer) that usually does not delay
execution much. Usually, network.target is part of the boot
of most systems, while network-online.target is not, except
when at least one unit requires it. Also see Running
Services After the Network is up[1] for more information.
sshd does not in general require a configured network connection just to
start up, and making it do so would be a pretty significant change to
the unit dependency graph on most systems; it would make "is not [part
of the boot process]" above typically untrue, for one thing.
See also
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/, which
among other things (in general the tone of that page is that a
well-written service should not use After=network-online.target) says:
"If you write a server: listen on [::], [::1], 0.0.0.0 and 127.0.0.1
only. These pseudo-addresses are unconditionally available."
That's what sshd does in its default configuration. If it doesn't work,
the systemd documentation suggests that something else is not fulfilling
its end of a contract somewhere.
--
Colin Watson (he/him) [cjwatson@debian.org]
Reply to: