Re: cloud-init failing to run user-data, about 1 in 10 times
In one failing run, I see three tracebacks:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/ec2_utils.py", line
178, in _get_instance_metadata
response = caller(md_url)
File "/usr/lib/python3/dist-packages/cloudinit/url_helper.py", line
107, in read_file_or_url
exception_cb=exception_cb)
File "/usr/lib/python3/dist-packages/cloudinit/url_helper.py", line
301, in readurl
raise excps[-1]
cloudinit.url_helper.UrlError:
HTTPConnectionPool(host='169.254.169.254', port=80): Max retries
exceeded with url: /2009-04-04/dynamic/instance-identity (Caused by
NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection
object at 0x7f1f7d7b70b8>: Failed to establish a new connection:
[Errno 101] Network is unreachable',))
2018-12-05 12:53:25,219 - util.py[DEBUG]: Getting data from <class
'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1992, in subp
env=env, shell=shell)
File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py",
line 523, in find_source
if s.get_data():
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py",
line 383, in get_data
return super(DataSourceEc2Local, self).get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py",
line 142, in get_data
return_value = self._get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceEc2.py",
line 105, in _get_data
with EphemeralDHCPv4(self.fallback_interface):
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line
46, in __enter__
leases = maybe_perform_dhcp_discovery(self.iface)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line
99, in maybe_perform_dhcp_discovery
return dhcp_discovery(dhclient_path, nic, tdir)
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line
163, in dhcp_discovery
util.subp(cmd, capture=True)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2000, in subp
stderr="-" if decode else b"-")
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['/var/tmp/cloud-init/cloud-init-dhcp-cxj1x1bs/dhclient',
'-1', '-v', '-lf',
'/var/tmp/cloud-init/cloud-init-dhcp-cxj1x1bs/dhcp.leases', '-pf',
'/var/tmp/cloud-init/cloud-init-dhcp-cxj1x1bs/dhclient.pid', 'eth0',
'-sf', '/bin/true']
Exit code: -
Reason: [Errno 13] Permission denied
Stdout: -
Stderr: -
2018-12-05 12:53:25,249 - main.py[DEBUG]: No local datasource found
On Thu, Dec 6, 2018 at 10:43 AM Joshua Powers <josh.powers@canonical.com> wrote:
>
> On 12/4/18 12:48 PM, Jason Price wrote:
> > How can I triage this further?
> >
>
> Logs for cloud-init are stored in:
>
> /var/log/cloud-init.log
> /var/log/cloud-init-output.log
>
> View those on the failing instance, look for any trace or other error
> messages. There is also the 'cloud-init collect-logs' subcommand to
> collect and tar up data to send others help triage failures.
>
> If you find it is related to cloud-init, you can ask questions at
> #cloud-init on freenode.
>
Reply to: