Bug#1120743: apt.systemd.daily update fails because it runs before network interfaces are ready
Package: apt
Version: 3.0.3
(Likely affects multiple versions)
unattended-upgrades was not working (I let it try and fail for over a
month while there were known updates available, it never worked). The
problem was with /usr/lib/apt/apt.systemd.daily.
'apt.systemd.daily update' was invoked, by systemd, after system wake
from suspend, before the network was available. Even though it was
unsuccessful, it still touched the timestamp file indicating it had
already ran that day. Subsequent attempts to run would simply be
skipped, since apt.systemd.daily checks the timestamp file, and aborts
if the stamp file indicates it has already ran that day.
The update failing because the script was run before network interfaces
were up, after a suspend/wake, was apparently a known issue, as a
partial work around was already included in apt-daily.service.
apt-daily.service includes the line:
ExecStartPre=-/usr/lib/apt/apt-helper wait-online
But, 'apt-helper wait-online' does nothing when using ifupdown. You can
down the interface entirely, and 'apt-helper wait-online' will still,
immediately, return with a zero exit code, the apt.systemd.daily script
will attempt, unsuccessfully, to do the apt update, but the script will
set the flag, as if it was successful, so no further attempts will run
that same day.
Per this Ubuntu bug report where it appears 'apt-helper wait-online' was
introduced, there was no attempt to make it work for anything but
systemd networkd and network manager.
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1699850
Per that bug report, 'apt-helper wait-online' waits up to 60 minutes for
the network to become available. The following hacky line, added to
apt-daily.service, accomplishes the same, and works regardless of how
network interfaces are managed.
# try once per minute for 60 minutes to see if network ready, if yes,
continue
ExecStartPre=/bin/bash -c "for ((i=1; i<60; i++)); do nc -z -w 1
deb.debian.org 80 2>/dev/null && break || sleep 60; done"
This work around gets apt periodic's update to work again, so
unattended-upgrades works again; unattended-upgrades has picked up
updates each of the 3 days since putting this hacky work-around into
place; while failing to work each of the 40 days prior to putting the
work-around hack into place.
It seems there are multiple bugs:
1. apt periodic update should not touch the "stamp" file if there was an
error that prevented it from performing the update.
2. 'apt-helper wait-online' should work for all supported network
management tools, including ifupdown.
3. if systemd had sane behavior, 'apt-helper wait-online', and my
similar hack, would not be necessary, in the first place.
Reply to: