Re: iwd + systemd-networkd + resolvconf wrinkles
On Thu 17 Mar 2022 at 12:12:28 (+0000), Thomas Pircher wrote:
> David Wright wrote:
> > As I said, I tried that.
>
> Ack. I must have glossed over that. Sorry. The rest of my mail stands,
> though.
>
> > > You can configure various settings for the DNS resolver in your
> > > systemd-networkd setting and in /etc/systemd/resolved.conf.
> >
> > Like what?
>
> Full description here:
> https://www.freedesktop.org/software/systemd/man/systemd.network.html#%5BDHCPv4%5D%20Section%20Options
> https://www.freedesktop.org/software/systemd/man/resolved.conf.html
Yes, I read those, but I can see nothing to profitably change.
> But what I find useful is to be able to select per interface if DNS
> should be used from the DHCP server, if there is a clash.
> I also ended up disabling DNSSEC on some machines due to a broken
> server.
I am assuming that I don't have that problem at home. As for
on-the-road, I'm not sure I'd be capable of diagnosing such problems.
> > > On bookworm you also have the resolvectl tool, which helps debugging DNS
> > > issues.
> >
> > And bullseye has that too. I don't really know how to use it.
>
> Cool. If you just type resolvectl, it will show you which information it
> got on each interface.
This is machine F, where /etc/resolv.conf is a file, containing 192.168.1.1 :
$ resolvectl
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
Link 2 (enp2s2)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 5 (wlp2s4)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
$ host www.google.com
www.google.com has address 142.250.138.105
www.google.com has address 142.250.138.103
www.google.com has address 142.250.138.106
www.google.com has address 142.250.138.99
www.google.com has address 142.250.138.104
www.google.com has address 142.250.138.147
www.google.com has IPv6 address 2607:f8b0:4000:80e::2004
$ host www.lionunicorn.co.uk
www.lionunicorn.co.uk has address 149.255.60.149
$
Those responses were instantaneous. (I don't think I should expect
resolvectl query to work here.)
And this is machine R, with systemd-resolved running:
$ ls -l /etc/resolv.conf
lrwxrwxrwx [ … ] /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ resolvectl
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (enp1s0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 4 (wlan0)
Current Scopes: DNS LLMNR/IPv4
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
$ host www.google.com
www.google.com has address 142.251.32.196
www.google.com has IPv6 address 2607:f8b0:4023:1002::63
www.google.com has IPv6 address 2607:f8b0:4023:1002::67
www.google.com has IPv6 address 2607:f8b0:4023:1002::93
www.google.com has IPv6 address 2607:f8b0:4023:1002::69
;; connection timed out; no servers could be reached
$ resolvectl query www.google.com
www.google.com: 2607:f8b0:4000:805::2004 -- link: wlan0
142.251.46.132 -- link: wlan0
-- Information acquired via protocol DNS in 33.6ms.
-- Data is authenticated: no
$ resolvectl query www.lionunicorn.co.uk
www.lionunicorn.co.uk: resolve call failed: Connection timed out
$
Here, host's substantive response was immediate, but I had to wait for
the prompt to return.
> You can also debug your slow queries by using "resolvectl query
> google.com". It will show you which interface the query goes out on and
> how long it took to get the response.
The attached file has the date, hour, hostname, systemd-resolved and
PID removed, and it pertains to the www.lionunicorn.co.uk query above.
Perhaps this would pinpoint a problem.
> > There seem to be timeouts involved in most cases, so time ping -c 1 foo
> > will typically take 15sec, and host lookups will take 10 or 20sec.
>
> That is far too long. A wild guess: you may have received a bunch of
> unresponsive DNS servers from your DHCP reply, and your machine is
> trying to use them in turn, until it finds a working server?
> DNSSEC problem? Or do you get IPv6 addresses for the DNS server, but
> they are not reachable?
>
> You can try debugging this with the resolvectl tool, to find out the
> list of the servers. Then query them with the dig tool from the
> bind9-dnsutils package:
>
> dig google.com @8.8.8.8
>
> Replace the IP address in @8.8.8.8 with the an IP from the output of
> resolvectl.
This response is immediate on R:
$ dig www.lionunicorn.co.uk @192.168.1.1
; <<>> DiG 9.16.22-Debian <<>> www.lionunicorn.co.uk @192.168.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30004
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.lionunicorn.co.uk. IN A
;; ANSWER SECTION:
www.lionunicorn.co.uk. 10800 IN A 149.255.60.149
;; Query time: 191 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Thu Mar 17 23:03:37 CDT 2022
;; MSG SIZE rcvd: 66
$
In case of interest, the query time (191, 195 msec) on machine R is
consistently slower than machine F (28, 44 msec). (R is a /much/
faster machine than F, being 10 years younger.)
> > # resolvectl query smtp.lionunicorn.co.uk answered in 57.6 secs.
> > # resolvectl query lionunicorn.co.uk failed with:
> > lionunicorn.co.uk: resolve call failed: Query timed out
>
> On my machine I get:
>
> # resolvectl query smtp.lionunicorn.co.uk
> smtp.lionunicorn.co.uk: 149.255.60.149 -- link: vlan3512
>
> -- Information acquired via protocol DNS in 31.0ms.
> -- Data is authenticated: no
>
> Try running the queries with dig, as described above.
Silly me: I used www this time, instead of smtp. But I just checked,
and it resolves the same address in the same time. It's all just
pointing at cloud204.unlimitedwebhosting.co.uk in reality.
> > The debug output is difficult to interpret, though I did notice that
> > it was reporting "cache misses" repeatedly on the same address (but
> > there must be some caching going on, because there was an occasional
> > hit being reported).
>
> It really sounds like some of the DNS servers are not reachable.
>
> > The idea of "debugging DNS issues" doesn't exactly thrill me. I'm
> > imagining a scenario where I'm sitting in an airport or motel room,
> > having managed to make a connection with iwd and negotiate their
> > captive portal or whatever, and then run into /this/ problem.
>
> Ack, fully understand. I do think there is something broken in your
> network setup or the server that gives you the list of DNS server is not
> configured correctly.
The same server is being used by machines F and R, and I could have
equally well set the two laptops up in the opposite configuration.
So something has to be wrong in the configuration of R. The nameserver
address of 127.0.0.53 in R's /etc/resolv.conf goes into one end of
systemd-resolved, and should come out at the other end as 192.168.1.1,
the sole nameserver, residing at the edge of my LAN. What goes on
inside systemd-resolved's stomach is a deep mystery.
> If you have found a way to fix the problem, or work around it, by using
> another tool, and this works for you, all the power to you. :-)
Yes, either resolvconf (for travel), or a fixed 192.168.1.1 suffices
(at home).
Cheers,
David.
44:01 : Setting log level to debug.
44:01 : Sent message type=method_return sender=n/a destination=:1.40 path=n/a interface=n/a member=n/a cookie=40 reply_cookie=2 signature=n/a error-name=n/a error-message=n/a
44:38 : Got message type=method_call sender=:1.41 destination=org.freedesktop.resolve1 path=/org/freedesktop/resolve1 interface=org.freedesktop.resolve1.Manager member=ResolveHostname cookie=2 reply_cookie=0 signature=isit error-name=n/a error-message=n/a
44:38 : idn2_lookup_u8: www.lionunicorn.co.uk → www.lionunicorn.co.uk
44:38 : Looking up RR for www.lionunicorn.co.uk IN A.
44:38 : Looking up RR for www.lionunicorn.co.uk IN AAAA.
44:38 : Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=41 reply_cookie=0 signature=s error-name=n/a error-message=n/a
44:38 : Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetNameOwner cookie=42 reply_cookie=0 signature=s error-name=n/a error-message=n/a
44:38 : Got message type=method_return sender=org.freedesktop.DBus destination=:1.1 path=n/a interface=n/a member=n/a cookie=17 reply_cookie=42 signature=s error-name=n/a error-message=n/a
44:38 : Cache miss for www.lionunicorn.co.uk IN AAAA
44:38 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
44:38 : Using feature level UDP+EDNS0 for transaction 38413.
44:38 : Using DNS server 192.168.1.1 for transaction 38413.
44:38 : Sending query packet with id 38413 of size 50.
44:38 : Positive cache hit for www.lionunicorn.co.uk IN A
44:38 : Transaction 27284 for <www.lionunicorn.co.uk IN A> on scope dns on wlan0/* now complete with <success> from cache (unsigned).
44:38 : Got message type=method_return sender=org.freedesktop.DBus destination=:1.1 path=n/a interface=n/a member=n/a cookie=16 reply_cookie=41 signature=n/a error-name=n/a error-message=n/a
44:38 : Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',arg0=':1.41' successfully installed.
44:43 : Timeout reached on transaction 38413.
44:43 : Retrying transaction 38413.
44:43 : Cache miss for www.lionunicorn.co.uk IN AAAA
44:43 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
44:43 : Using feature level UDP+EDNS0 for transaction 38413.
44:43 : Sending query packet with id 38413 of size 50.
44:48 : Timeout reached on transaction 38413.
44:48 : Retrying transaction 38413.
44:48 : Cache miss for www.lionunicorn.co.uk IN AAAA
44:48 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
44:48 : Using feature level UDP+EDNS0 for transaction 38413.
44:48 : Sending query packet with id 38413 of size 50.
44:54 : Timeout reached on transaction 38413.
44:54 : Retrying transaction 38413.
44:54 : Cache miss for www.lionunicorn.co.uk IN AAAA
44:54 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
44:54 : Using feature level UDP+EDNS0 for transaction 38413.
44:54 : Sending query packet with id 38413 of size 50.
44:59 : Timeout reached on transaction 38413.
44:59 : Retrying transaction 38413.
44:59 : Cache miss for www.lionunicorn.co.uk IN AAAA
44:59 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
44:59 : Using feature level UDP+EDNS0 for transaction 38413.
44:59 : Sending query packet with id 38413 of size 50.
45:04 : Timeout reached on transaction 38413.
45:04 : Retrying transaction 38413.
45:04 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:04 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:04 : Using feature level UDP+EDNS0 for transaction 38413.
45:04 : Sending query packet with id 38413 of size 50.
45:09 : Timeout reached on transaction 38413.
45:09 : Retrying transaction 38413.
45:09 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:09 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:09 : Using feature level UDP+EDNS0 for transaction 38413.
45:09 : Sending query packet with id 38413 of size 50.
45:15 : Timeout reached on transaction 38413.
45:15 : Retrying transaction 38413.
45:15 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:15 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:15 : Using feature level UDP+EDNS0 for transaction 38413.
45:15 : Sending query packet with id 38413 of size 50.
45:20 : Timeout reached on transaction 38413.
45:20 : Retrying transaction 38413.
45:20 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:20 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:20 : Using feature level UDP+EDNS0 for transaction 38413.
45:20 : Sending query packet with id 38413 of size 50.
45:25 : Timeout reached on transaction 38413.
45:25 : Retrying transaction 38413.
45:25 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:25 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:25 : Using feature level UDP+EDNS0 for transaction 38413.
45:25 : Sending query packet with id 38413 of size 50.
45:30 : Timeout reached on transaction 38413.
45:30 : Retrying transaction 38413.
45:30 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:30 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:30 : Using feature level UDP+EDNS0 for transaction 38413.
45:30 : Sending query packet with id 38413 of size 50.
45:36 : Timeout reached on transaction 38413.
45:36 : Retrying transaction 38413.
45:36 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:36 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:36 : Using feature level UDP+EDNS0 for transaction 38413.
45:36 : Sending query packet with id 38413 of size 50.
45:41 : Timeout reached on transaction 38413.
45:41 : Retrying transaction 38413.
45:41 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:41 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:41 : Using feature level UDP+EDNS0 for transaction 38413.
45:41 : Sending query packet with id 38413 of size 50.
45:46 : Timeout reached on transaction 38413.
45:46 : Retrying transaction 38413.
45:46 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:46 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:46 : Using feature level UDP+EDNS0 for transaction 38413.
45:46 : Sending query packet with id 38413 of size 50.
45:51 : Timeout reached on transaction 38413.
45:51 : Retrying transaction 38413.
45:51 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:51 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:51 : Using feature level UDP+EDNS0 for transaction 38413.
45:51 : Sending query packet with id 38413 of size 50.
45:56 : Got DNS stub UDP query packet for id 16367
45:56 : Looking up RR for wren IN AAAA.
45:56 : Sending response packet with id 16367 on interface 1/AF_INET of size 33.
45:56 : Processing query...
45:57 : Timeout reached on transaction 38413.
45:57 : Retrying transaction 38413.
45:57 : Cache miss for www.lionunicorn.co.uk IN AAAA
45:57 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
45:57 : Using feature level UDP+EDNS0 for transaction 38413.
45:57 : Sending query packet with id 38413 of size 50.
46:02 : Timeout reached on transaction 38413.
46:02 : Retrying transaction 38413.
46:02 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:02 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:02 : Using feature level UDP+EDNS0 for transaction 38413.
46:02 : Sending query packet with id 38413 of size 50.
46:07 : Timeout reached on transaction 38413.
46:07 : Retrying transaction 38413.
46:07 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:07 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:07 : Using feature level UDP+EDNS0 for transaction 38413.
46:07 : Sending query packet with id 38413 of size 50.
46:12 : Timeout reached on transaction 38413.
46:12 : Retrying transaction 38413.
46:12 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:12 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:12 : Using feature level UDP+EDNS0 for transaction 38413.
46:12 : Sending query packet with id 38413 of size 50.
46:18 : Timeout reached on transaction 38413.
46:18 : Retrying transaction 38413.
46:18 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:18 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:18 : Using feature level UDP+EDNS0 for transaction 38413.
46:18 : Sending query packet with id 38413 of size 50.
46:23 : Timeout reached on transaction 38413.
46:23 : Retrying transaction 38413.
46:23 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:23 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:23 : Using feature level UDP+EDNS0 for transaction 38413.
46:23 : Sending query packet with id 38413 of size 50.
46:28 : Timeout reached on transaction 38413.
46:28 : Retrying transaction 38413.
46:28 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:28 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:28 : Using feature level UDP+EDNS0 for transaction 38413.
46:28 : Sending query packet with id 38413 of size 50.
46:33 : Timeout reached on transaction 38413.
46:33 : Retrying transaction 38413.
46:33 : Cache miss for www.lionunicorn.co.uk IN AAAA
46:33 : Transaction 38413 for <www.lionunicorn.co.uk IN AAAA> scope dns on wlan0/*.
46:33 : Using feature level UDP+EDNS0 for transaction 38413.
46:33 : Sending query packet with id 38413 of size 50.
46:38 : Got message type=signal sender=org.freedesktop.DBus destination=n/a path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameOwnerChanged cookie=18 reply_cookie=0 signature=sss error-name=n/a error-message=n/a
46:38 : Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RemoveMatch cookie=43 reply_cookie=0 signature=s error-name=n/a error-message=n/a
46:38 : Freeing transaction 38413.
46:38 : Freeing transaction 27284.
46:38 : Sent message type=error sender=n/a destination=:1.41 path=n/a interface=n/a member=n/a cookie=44 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.Timeout error-message=Query timed out
46:38 : Got message type=error sender=org.freedesktop.DBus destination=:1.1 path=n/a interface=n/a member=n/a cookie=19 reply_cookie=44 signature=s error-name=org.freedesktop.DBus.Error.ServiceUnknown error-message=The name :1.41 was not provided by any .service files
Reply to: