Re: OpenSSH: cause of random kex_exchange_identification errors?
On 2022-02-05 18:39:27 -0300, Henrique de Moraes Holschuh wrote:
> On Wed, 02 Feb 2022, Vincent Lefevre wrote:
> > When I want to connect with SSH (ssh/scp) to some machine, I sometimes
> > get errors, either
> >
> > kex_exchange_identification: Connection closed by remote host
> >
> > or
> >
> > kex_exchange_identification: read: Connection reset by peer
>
> That's a very early stage of the initial connection, and your SSH client
> just noticed the remote server (or some middle box like a firewall, or
> oversubscribed NAT gateway) dropped the TCP connection for whatever
> reason.
Yes, this is what I observed, e.g. when I reproduced the error
with telnet.
> It will be related to the TCP connection itself, and nothing else. IP
> protocol family/address/port, or something in the TCP path misbehaving.
>
> The most common reason is that the remote server disliked your IP
> address and/or port due to /etc/hosts.allow/deny, firewalling, or
> something in sshd_config.
I could reproduce the issue from multiple IP addresses (both from
the local network and from external networks), and the errors are
completely random. Immediately after a failure, this can succeed.
There's a fail2ban, but the IP can get banned only after
authentication failures, and in this case, the error is not the same:
the connection is not accepted (something like "connection refused",
not a kex_exchange_identification error, where the connection is
first accepted).
> Ensure you check both IPv4 and IPv6.
Only IPv4 is supported there (the host does not have an IPv6 address,
so that there can't be any mistake).
> > The admin of the machine could see nothing particular in the logs.
> > He eventually modified the MaxStartups value, but this did not
> > solve the issue (but AFAIK, if this were the cause, there would
> > have been something about it in the logs). The machine has enough
> > available memory.
> >
> > Any idea about the possible cause of these random errors?
>
> The debugging needs to be done either in the server side, or on *both*
> sides.
>
> If you're using socket-forwarding stuff in the client side, check that.
> This is exceedingly rare nowadays, so I doubt it. Stuff like SOCKS4 or
> SOCKS5 TCP proxies.
Nothing like that.
> Find what is actually listening on the TCP socket server-side, it might
> not be sshd (interposers like systemd socket activation, xinetd/inetd,
> etc). The logs/access control you need to look at server-side might not
> be SSHD's in that case.
There's a xinetd running, but its config files show that it does not
handle sshd, and there's a /usr/sbin/sshd running anyway (with lots
of children corresponding to all the current connections).
> If it is sshd, ensure it is actually logging all you need, and carefully
> study the logs.
It appears that the failure occurs too soon. The first thing that
sshd normally logs is a "Accepted publickey" line, but the connection
is closed before authentication. Apparently the admin could only see
my successful connections in the logs.
> If nothing helps, packet-dump both sides (client and server) and find
> out what sent the TCP RST, as that might give you clues for the "why".
> A middlebox might be doing it...
>
> But get the remote admin to re-check server-side /etc/hosts.allow and
> deny, sshd_config, etc. with an eye on "what might my SSH client be
> using when it failed, and what it might have been using when it worked"
> before all that debugging work. Don't forget to consider IPv6, or all
> possibly outgoing ranges of IPv4 NAT, if any. It might pay off :-)
/etc/hosts.allow and /etc/hosts.deny just contain comments.
But if the goal were to reject the connection before authentication,
then the connection should not be accepted in the first place.
Currently the errors have stopped (or are just too rare to reproduce).
If they occur again, more debugging could be done.
--
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
Reply to: