After software update systemd runs into timeouts
I run a server with Debian 9 and haven't resisted the conversion to
systemd strong enough, therefore I now have a couple of problems with
stability and availability.
One of them is, that often after software updates systemd-logind.service
fails to start and ssh logins take a very long time. I suspect this has
to do with shared libs being replaced while being used by running
processes. This usually causes no problems for everything else except
systemd. When that problem occurs, using lsof I find and restart every
service using deleted files. Sometimes this helps but not always.
When I try to restart systemd-logind.service it fails with an error
message after a timeout of 1:30 minutes, but then doesn't give any
information about the problem (instead it prefers to show off how little
resources it used and cluttering up output with static info like URLs,
both of which don't belong in a status output):
# time systemctl restart systemd-logind.service
Job for systemd-logind.service failed because a timeout was exceeded.
See "systemctl status systemd-logind.service" and "journalctl -xe" for details.
real 1m30.270s
user 0m0.003s
sys 0m0.004s
# systemctl status systemd-logind.service
â?? systemd-logind.service - Login Service
Loaded: loaded (/lib/systemd/system/systemd-logind.service; static; vendor preset: enabled)
Active: activating (start) since Wed 2020-08-05 11:06:06 CEST; 51s ago
Docs: man:systemd-logind.service(8)
man:logind.conf(5)
http://www.freedesktop.org/wiki/Software/systemd/logind
http://www.freedesktop.org/wiki/Software/systemd/multiseat
Main PID: 29560 (systemd-logind)
Status: "Processing requests..."
Tasks: 1 (limit: 4915)
Memory: 1.0M
CPU: 10ms
CGroup: /system.slice/systemd-logind.service
â??â??29560 /lib/systemd/systemd-logind
Aug 05 11:06:06 bit systemd-logind[29560]: New session 3893 of user urs.
Aug 05 11:06:06 bit systemd-logind[29560]: New session 3889 of user urs.
...
The log messages in /var/log/syslog also don't give enough information
about what has failed, but it seems systemd tries to restart that service
in the background:
Aug 5 11:06:06 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:06:06 bit systemd[1]: Failed to start Login Service.
Aug 5 11:06:06 bit systemd[1]: systemd-logind.service: Unit entered failed state.
Aug 5 11:06:06 bit systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Aug 5 11:06:06 bit systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Aug 5 11:06:06 bit systemd[1]: Stopped Login Service.
Aug 5 11:06:06 bit systemd[1]: Starting Login Service...
# grep Start /var/log/syslog | tail -10
Aug 5 11:37:41 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:37:41 bit systemd[1]: Starting Login Service...
Aug 5 11:39:11 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:39:11 bit systemd[1]: Starting Login Service...
Aug 5 11:40:41 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:40:41 bit systemd[1]: Starting Login Service...
Aug 5 11:42:11 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:42:11 bit systemd[1]: Starting Login Service...
Aug 5 11:43:41 bit systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Aug 5 11:43:41 bit systemd[1]: Starting Login Service...
With jounalctl you also see the bloated logs which make them unpleasant to
search and it doesn't have more info than /var/log/syslog.
I had this problem 5 to 10 times until now, I have tried fixing it
everytime for an hour or so, and finally given up rebooting the machine.
Is there anything one can do to make systemd working properly again?
<rant>
In my 30 years of experience with Unix I have always loved the simplicity,
elegance, and stability. With Linux in general and specifically Debian
with apt, I liked how easily you can update software without disturbing
the running system -- until the advent of systemd. It makes everything
more complicate, non-Unix-ish, and less stable. In my experience, systemd
is the end to "Debian as the maintainable OS".
Now I have the choice of not updating, rebooting after most updates, or
having to wait a minute or so for each remote login. All unacceptable for
a server with long-running processes, hosting a couple of virtual
machines, and high uptimes.
</rant>
urs
Reply to: