Bug#633793: Resolution

[Scott Duckworth <sduckwo@clemson.edu>]

I am passing along this information in case anybody else runs into this
problem.

We found the cause of the problem with our DNS servers.  We have a Cisco
ACE load balancer sitting in front of a few DNS servers, which are
presented as one DNS server.  UDP fast aging was enabled on the ACE load
balancer, and turning it off fixed the problem.  Here's why:

getaddrinfo() queries for AAAA or A records as two separate queries from
the same source port, and the DNS server responds with two separate
answers.  UDP fast aging on the ACE removes the source address rewriting
rule on a UDP "connection" after seeing the first response packet.  So
the 2nd DNS server response packet didn't get its source IP address (the
real DNS server) rewritten to the virtual load balanced IP address, and
there must be something in newer libc/libresolv libraries that checks if
the source IP address of the response is the same as who the query was
sent to.

The fix was to disable on the Cisco ACE load balancer:

  loadbalance vip udp-fast-age

and then add a UDP expiration timer of 10 seconds or so for DNS queries.