Bug#1025828: cloud.debian.org: cloud-init may run time synchronization dependent jobs before the clock is synchronized
Package: cloud.debian.org
Severity: normal
Certain functionality built in to cloud-init depends on a reasonably
accurate clock, such as apt repo metadata signature verification. In the
case where a system's hardware clock is far out of sync, chrony may not have
completed synchronization before apt tries to refresh apt caches in the
package_update_upgrade_install module or in a userdata shell script. An
example of the resulting failure follows:
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 169.254.169.123 3 6 1 1 -2986h[ -2986h] +/- 586us
Sun Aug 7 13:25:16 UTC 2022
Get:2 http://cdn-aws.deb.debian.org/debian bullseye InRelease [116 kB]
Get:3 http://cdn-aws.deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://cdn-aws.deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Reading package lists...
E: Release file for http://security.debian.org/debian-security/dists/bullseye-security/InRelease is not valid yet (invalid for another 124d 5h 58min 50s). Updates for this repository will not be applied.
E: Release file for http://cdn-aws.deb.debian.org/debian/dists/bullseye/InRelease is not valid yet (invalid for another 33d 20h 52min 34s). Updates for this repository will not be applied.
E: Release file for http://cdn-aws.deb.debian.org/debian/dists/bullseye-updates/InRelease is not valid yet (invalid for another 124d 6h 53min 11s). Updates for this repository will not be applied.
E: Release file for http://cdn-aws.deb.debian.org/debian/dists/bullseye-backports/InRelease is not valid yet (invalid for another 124d 6h 53min 11s). Updates for this repository will not be applied.
2022-08-07 13:25:17,663 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2022-08-07 13:25:17,664 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 20.4.1 finished at Sun, 07 Aug 2022 13:25:17 +0000. Datasource DataSourceEc2Local. Up 10.28 seconds
The userdata script was:
#!/bin/bash
chronyc sources
date
DEBIAN_FRONTEND=noninteractive apt-get update
A possible workaround to this is to run a command like `chronyc waitsync`
before apt. It turns out to be not entirely trivial to insert such a
command in the right place with userdata, though.
We should make some reasonable effort to ensure the clock is in sync before
executing the cloud-final.service, which is where all the package
installation work happens. It's not immediately clear exactly where we
should do this, whether it's in the chrony configs, cloud-init itself,
cloud-init's systemd configuration, or elsewhere. And then, should the fix
be in one of the packages or the images?
Reply to: