Bug#1025618: cloud-init and firewalld systemd unit files have ordering cycles
Package: cloud-init
Version: 20.4.1-2+deb11u1
Severity: important
X-Debbugs-Cc: xilun@australdx.fr
Dear Maintainer,
firewalld and cloud-init have ordering cycles between their systemd unit
files, leading to more or less broken boot results when both are installed
and active, because at each boot systemd decides to skip a
non-deterministically choosen service (not necessarily cloud-init or
firewalld) to break the cycle.
I'm not sure if any of firewalld or cloud-init is more at fault (maybe
in not respecting some systemd rules?) so I'm also opening a duplicate
of this bug for the other package.
This can have various but potentially serious consequences, depending on
what should be, but is not, started.
Examples of boot traces of this issue happening:
* example 1:
sysinit.target: Found ordering cycle on cloud-init.service/start
sysinit.target: Found dependency on networking.service/start
sysinit.target: Found dependency on network-pre.target/start
sysinit.target: Found dependency on firewalld.service/start
sysinit.target: Found dependency on basic.target/start
sysinit.target: Found dependency on sockets.target/start
sysinit.target: Found dependency on uuidd.socket/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start
* example 2:
sysinit.target: Found ordering cycle on cloud-init.service/start
sysinit.target: Found dependency on networking.service/start
sysinit.target: Found dependency on network-pre.target/start
sysinit.target: Found dependency on firewalld.service/start
sysinit.target: Found dependency on dbus.service/start
sysinit.target: Found dependency on sysinit.target/start
sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start
* example 3:
firewalld.service: Found ordering cycle on dbus.socket/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start
* example 4:
firewalld.service: Found ordering cycle on dbus.service/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.service/start deleted to break ordering cycle starting with firewalld.service/start
basic.target: Found ordering cycle on sysinit.target/start
basic.target: Found dependency on cloud-init.service/start
basic.target: Found dependency on networking.service/start
basic.target: Found dependency on network-pre.target/start
basic.target: Found dependency on firewalld.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job cloud-init.service/start deleted to break ordering cycle starting with basic.target/start
* example 5:
basic.target: Found ordering cycle on sockets.target/start
basic.target: Found dependency on uuidd.socket/start
basic.target: Found dependency on sysinit.target/start
basic.target: Found dependency on cloud-init.service/start
basic.target: Found dependency on networking.service/start
basic.target: Found dependency on network-pre.target/start
basic.target: Found dependency on firewalld.service/start
basic.target: Found dependency on dbus.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start
firewalld.service: Found ordering cycle on dbus.socket/start
firewalld.service: Found dependency on sysinit.target/start
firewalld.service: Found dependency on cloud-init.service/start
firewalld.service: Found dependency on networking.service/start
firewalld.service: Found dependency on network-pre.target/start
firewalld.service: Found dependency on firewalld.service/start
firewalld.service: Job dbus.socket/start deleted to break ordering cycle starting with firewalld.service/start
* example 6:
networking.service: Found ordering cycle on network-pre.target/start
networking.service: Found dependency on firewalld.service/start
networking.service: Found dependency on dbus.service/start
networking.service: Found dependency on basic.target/start
networking.service: Found dependency on sockets.target/start
networking.service: Found dependency on uuidd.socket/start
networking.service: Found dependency on sysinit.target/start
networking.service: Found dependency on cloud-init.service/start
networking.service: Found dependency on networking.service/start
networking.service: Job network-pre.target/start deleted to break ordering cycle starting with networking.service/start
At first I experienced the issue on a Debian Stable (Bullseye), then
I was able to reproduce the problem on an up-to-date Bookworm.
Note that "systemd-analyze verify" seems to be unable to find the issue,
however, the following repro (tried on Bookworm) shows a way to detect
the cycles statically (plus, during a reboot you can observe it
directly):
$ sudo apt install firewalld cloud-init
$ echo "datasource_list: [ Fallback ]" | sudo tee /etc/cloud/cloud.cfg.d/99_fallback.cfg
$ sudo reboot
$ wget https://raw.githubusercontent.com/jantman/misc-scripts/4560db773f463101273539e625c9b48e9f53f87f/dot_find_cycles.py
$ # ^ I found that script by reading https://github.com/systemd/systemd/issues/3829
$ sudo apt install 2to3 python3-pygraphviz python3-pydotplus python3-pydot python3-graphviz python3-networkx
$ 2to3 -w dot_find_cycles.py
$ chmod +x dot_find_cycles.py
$ sudo systemd-analyze dot --no-pager --order 2>/dev/null | python3 ./dot_find_cycles.py - | tee order-cycle_cloud-init-firewalld.txt
networking.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> networking.service
networking.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> networking.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> dbus.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> cloud-init-hotplugd.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> sockets.target -> dbus.socket -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> basic.target -> systemd-pcrphase-sysinit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
systemd-networkd.service -> network-pre.target -> firewalld.service -> polkit.service -> sysinit.target -> cloud-init.service -> systemd-networkd-wait-online.service -> systemd-networkd.service
Best regards,
Guillaume Knispel
-- System Information:
Debian Release: 11.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-19-amd64 (SMP w/64 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages cloud-init depends on:
ii fdisk 2.36.1-8+deb11u1
ii gdisk 1.0.6-1.1
ii ifupdown 0.8.36
ii locales 2.31-13+deb11u5
ii lsb-base 11.1.0
ii lsb-release 11.1.0
ii net-tools 1.60+git20181103.0eebece-1
ii procps 2:3.3.17-5
ii python3 3.9.2-3
ii python3-configobj 5.0.6-4
ii python3-jinja2 2.11.3-1
ii python3-jsonpatch 1.25-3
ii python3-jsonschema 3.2.0-3
ii python3-oauthlib 3.1.0-2
ii python3-requests 2.25.1+dfsg-2
ii python3-yaml 5.3.1-5
ii util-linux 2.36.1-8+deb11u1
Versions of packages cloud-init recommends:
ii cloud-guest-utils 0.31-2
ii eatmydata 105-9
ii sudo 1.9.5p2-3
Versions of packages cloud-init suggests:
pn btrfs-progs <none>
ii e2fsprogs 1.46.2-2
pn xfsprogs <none>
-- no debconf information
Reply to: