Hi,
i was debugging some issues i randomly had when deploying the cloud
image (VMWare, ansible created cidata iso image in an OVA file).
Specifically - sometimes after the system was up the dns
was broken.
- /etc/resolv.conf was a symlink to the resolvconf stub
- The resolvconf stub was empty (Except some comments that its the stub)
So DNS is broken.
After some debugging i found that randomly it seems the cloud image
trys to do DHCP on boot before the cloud-init run. When that happens
cloud init happily writes the /etc/network/interfaces.d/50-cloud-init
with a DNS Server in it - but fails to bring up the interface or feed
resolved. My guess is that ifupdown has a half way up interface or something.
Shortened logfile of the failing first boot - Another symptom is that the first
boot takes >1 Minute because DHCP needs to time out.
[ ... kernel setup ... ]
Apr 22 17:34:40 svrb-flo-test kernel: vmxnet3 0000:03:00.0 ens160: intr type 3, mode 0, 3 vectors allocated
Apr 22 17:34:40 svrb-flo-test kernel: vmxnet3 0000:03:00.0 ens160: NIC Link is Up 10000 Mbps
Apr 22 17:34:40 svrb-flo-test ifup[420]: ifup: waiting for lock on /run/network/ifstate.ens160
Apr 22 17:34:40 svrb-flo-test dhclient[498]: Internet Systems Consortium DHCP Client 4.4.1
Apr 22 17:34:40 svrb-flo-test sh[498]: Internet Systems Consortium DHCP Client 4.4.1
Apr 22 17:34:40 svrb-flo-test sh[498]: Copyright 2004-2018 Internet Systems Consortium.
Apr 22 17:34:40 svrb-flo-test sh[498]: All rights reserved.
Apr 22 17:34:40 svrb-flo-test sh[498]: For info, please visit https://www.isc.org/software/dhcp/
Apr 22 17:34:40 svrb-flo-test dhclient[498]: Copyright 2004-2018 Internet Systems Consortium.
Apr 22 17:34:40 svrb-flo-test dhclient[498]: All rights reserved.
Apr 22 17:34:40 svrb-flo-test dhclient[498]: For info, please visit https://www.isc.org/software/dhcp/
Apr 22 17:34:40 svrb-flo-test dhclient[498]:
Apr 22 17:34:40 svrb-flo-test dhclient[498]: Listening on LPF/ens160/00:50:56:ba:d5:54
Apr 22 17:34:40 svrb-flo-test sh[498]: Listening on LPF/ens160/00:50:56:ba:d5:54
Apr 22 17:34:40 svrb-flo-test sh[498]: Sending on LPF/ens160/00:50:56:ba:d5:54
Apr 22 17:34:40 svrb-flo-test sh[498]: Sending on Socket/fallback
[ ... ]
Apr 22 17:35:41 svrb-flo-test sh[498]: No DHCPOFFERS received.
Apr 22 17:35:41 svrb-flo-test sh[498]: No working leases in persistent database - sleeping.
Apr 22 17:35:41 svrb-flo-test dhclient[498]: No working leases in persistent database - sleeping.
Apr 22 17:35:41 svrb-flo-test sh[553]: ens160=ens160
Apr 22 17:35:41 svrb-flo-test systemd[1]: Finished Raise network interfaces.
Apr 22 17:35:41 svrb-flo-test systemd[1]: Reached target Network.
Apr 22 17:35:41 svrb-flo-test systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: Cloud-init v. 20.4.1 running 'init' at Fri, 22 Apr 2022 17:35:42 +0000. Up 67.67 seconds.
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | ens160 | True | 172.30.130.9 | 255.255.255.0 | global | 00:50:56:ba:d5:54 |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | ens160 | True | fe80::250:56ff:feba:d554/64 | . | link | 00:50:56:ba:d5:54 |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | lo | True | ::1/128 | . | host | . |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | 0 | 0.0.0.0 | 172.30.130.1 | 0.0.0.0 | ens160 | UG |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: | 1 | 172.30.130.0 | 0.0.0.0 | 255.255.255.0 | ens160 | U |
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Apr 22 17:35:42 svrb-flo-test cloud-init[574]: ci-info: +-------+-------------+---------+-----------+-------+
Also while debugging i found this:
root@svrb-flo-test:~# cat /etc/network/interfaces
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
# Cloud images dynamically generate config fragments for newly
# attached interfaces. See /etc/udev/rules.d/75-cloud-ifupdown.rules
# and /etc/network/cloud-ifupdown-helper. Dynamically generated
# configuration fragments are stored in /run:
source-directory /run/network/interfaces.d
This really looks odd. Duplicate include ...
After a reboot the /etc/resolv.conf gets filled and no DHCP is issued.
root@svrb-flo-test:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.
nameserver 172.31.100.15
(Is it only me that i find the description highly confusing - Its the resolvconf
stub file and it is intermixed with systemd-resolved stub infos)
My network user-data.yml section looks like this:
config:
- type: physical
name: ens160
subnets:
- type: static
address: 172.30.130.9/24
gateway: 172.30.130.1
dns_nameservers:
- 172.31.100.15
Flo
--
Florian Lohoff f@zz.de
Any sufficiently advanced technology is indistinguishable from magic.
Attachment:
signature.asc
Description: PGP signature